Programming in the cloud is almost here. I have to qualify that with almost, as I'm still having some issues setting up my own cloud server, and I always have issues paying real money to subscribe to cloud services. Why pay when you can host your cloud yourself. Ubuntu 14.10 has a number of very interesting option for 'Metal as a Service' (MaaS) which allows you to virtualize a group of machines to make a cloud and not worry about which machine is hosting what. Now all I have to do is find a bunch of commodity machines, some network attached storage (NAS) load the MaaS software and flip the switch and hope I don't take down the electric grid here in Fircrest. Perhaps I can find some company that dumping it's four year old laptops and get a deal. I've had some experience with virtuallizing machines, but gave up on that for the time being as I had other things to deal with. And now my 6-core machine is a gaming platform that really rips. So commodity machines, or bare motherboards on spacers might be a better answer than a single multi-core machine. Sure you have the network latency, but you can also swap out machines if they break with minimal effort, and you can add cores (assuming each board has a cheap single core CPU), and the energy requirements are minimal as there's no graphics card for any of them.
But first I wanted to try the cloud programming platform 'Cloud9' on Art which I set up in my previous blogs, but has no specified purpose other than a Linux platform to do some development on. So it has all the prerequisites for my own implementation of Cloud9. The Cloud 9 people are obviously hoping I'll just use their services, where I can logon and get a programming IDE in my browser. They also offer the software for free if you've got the intestinal fortitude to install and configure it yourself. I've got the intestinal fortitude, I'm just a little short on patience, so once I got it up and running, I had some issues with figuring out how to use it, how to set up a workspace, etc. But that's no difference than my experience with Eclipse, as it seems pretty opinionated about how everything should be set up, and I have my own ideas that I deem optimal. I would like to get Cloud 9 functioning, as it would open up development on any machine with a decent browser and a screen big enough to read with my glasses. No more having to set up all the prerequisites on any machine I want to do development on, and keeping them in sync and wondering if I have Mongo or whatever installed.
So this is my next project in my home network system, finding some company that's having a fire sale on out of date (for windows) laptops, and setting up MaaS with Art being the controlling node. That should allow me plenty of space in which to experiment with setting up and eliminating environments until I can get one that works at the level I expect. At the same time I'm going to continue to work with Cloud 9 to see if I can get it to function in a way that I expect, and have my all purpose browser-based IDE and stop worrying about which of my computer has this software or that software. This weekend I'll do more work on Cloud 9 and see if I can figure it out, and give some step-by-step instruction on setting it up in my next blog.
Friday, May 16, 2014
Thursday, May 1, 2014
Installing a development server part III
To recap the previous three posts, I installed Ubuntu 14.04 from the ground up, configured all my servers to talk to one another, and installed Jenkins, Nexus, and Jira. The only other application I want to install now is Confluence, but first I'm going to make a side track and create some reverse proxies for Nexus, Jira and eventually Confluence.
In Ubuntu server, Apache comes pre-installed with virtual server software. That means that if I specify hosts to my domain name they are processed by separate configuration. That way I can create a nexus.server.com and point it to the nexus server running on port 8081, and a jira.server.com and point it to jira running on port 8082. All call are served by Apache httpd, so will be coming from the default port 80. It's mostly a naming convenience, I could just poke a hole in my firewall at ports 8081 and 8082, but I like having a minimal amount of holes in my firewall. In addition, since Apache httpd will be the front end, it has a good reputation for being secure and responsive.
Unfortunately, making a reverse proxy can often be a problematic adventure. If webapps all played by the rules, it would not be a problem. But webapps occasionally do redirects, and if the redirect points to the internal server, any external browsers aren't going to process them. The outside world doesn't know anything about codemonkey, however I have several domain names registered which I can use.
The first thing, once you have a domain name registered, is to set up the DNS zone. All of my domain name are registered with godaddy.com, and mostly they are registered with the godaddy nameserver (NS). From there I can edit the DNS zone file and add CNAME that point to '@' which means they all point to the same server (my home server). So I created a CNAME for www, nexus, and jira, which all point to the same server, and Apache sorts out who the requests should go to. If you're having trouble following this, I can't blame you DNS is one of the older systems on the internet, and it's configuration is rather arcane. There are several books on DNS, but I can't recommend any. Exposing these services is completely optional, I just do it for experience, and I do need nexus outside my house if I ever do any Java programming on the road. The other two are just nice to have, Confluence will be my www, which is the default host, so that's what will be associated with my domain. Jira and Confluence go hand-in-hand, so might as well expose both of them. But exposing them, also means exposing yourself to risk, although it is minimal. You probably expose yourself more if you use Internet Explorer.
With godaddy all configured, now I have to configure Apache. In Ubuntu, the configuration file is split up in a very logical manner. They have directories 'sites-available' and 'sites-enabled'. You create the specifics about a virtual server in the 'sites-available' directory and then when they are ready, you link them into the sites-enabled directory. That way you can take your site down for maintenance by removing them from the sites-enabled directory, work on them in sites-available and then link them back when you're finished. To start off, there's a default configuration that you can copy into the sites-available directory:
cd /etc/apache2
sudo cp sites-available/000-default.conf sites-available/nexus.conf
sudo cp sites-available/000-default.conf sites-available/jira.conf
In Ubuntu server, Apache comes pre-installed with virtual server software. That means that if I specify hosts to my domain name they are processed by separate configuration. That way I can create a nexus.server.com and point it to the nexus server running on port 8081, and a jira.server.com and point it to jira running on port 8082. All call are served by Apache httpd, so will be coming from the default port 80. It's mostly a naming convenience, I could just poke a hole in my firewall at ports 8081 and 8082, but I like having a minimal amount of holes in my firewall. In addition, since Apache httpd will be the front end, it has a good reputation for being secure and responsive.
Unfortunately, making a reverse proxy can often be a problematic adventure. If webapps all played by the rules, it would not be a problem. But webapps occasionally do redirects, and if the redirect points to the internal server, any external browsers aren't going to process them. The outside world doesn't know anything about codemonkey, however I have several domain names registered which I can use.
The first thing, once you have a domain name registered, is to set up the DNS zone. All of my domain name are registered with godaddy.com, and mostly they are registered with the godaddy nameserver (NS). From there I can edit the DNS zone file and add CNAME that point to '@' which means they all point to the same server (my home server). So I created a CNAME for www, nexus, and jira, which all point to the same server, and Apache sorts out who the requests should go to. If you're having trouble following this, I can't blame you DNS is one of the older systems on the internet, and it's configuration is rather arcane. There are several books on DNS, but I can't recommend any. Exposing these services is completely optional, I just do it for experience, and I do need nexus outside my house if I ever do any Java programming on the road. The other two are just nice to have, Confluence will be my www, which is the default host, so that's what will be associated with my domain. Jira and Confluence go hand-in-hand, so might as well expose both of them. But exposing them, also means exposing yourself to risk, although it is minimal. You probably expose yourself more if you use Internet Explorer.
With godaddy all configured, now I have to configure Apache. In Ubuntu, the configuration file is split up in a very logical manner. They have directories 'sites-available' and 'sites-enabled'. You create the specifics about a virtual server in the 'sites-available' directory and then when they are ready, you link them into the sites-enabled directory. That way you can take your site down for maintenance by removing them from the sites-enabled directory, work on them in sites-available and then link them back when you're finished. To start off, there's a default configuration that you can copy into the sites-available directory:
cd /etc/apache2
sudo cp sites-available/000-default.conf sites-available/nexus.conf
sudo cp sites-available/000-default.conf sites-available/jira.conf
sudo cp sites-available/000-default.conf sites-available/confluence.conf
Confluence will be the default server, but I keep 000-default around for safe keeping. It's named with the 000 so that it will be the first configuration picked up for when hosts don't match. It points to /var/www/html, so you might put in a simple html file that always redirects to www. Always make sure your html is simple so that hackers don't have anything to take advantage of. Now I will edit each of the conf files to do the reverse proxying. First Jira, here's the jira.conf:
<VirtualHost *:80>
ServerName jira.myserver.com
ServerAdmin randysr@kamradtfamily.net
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# JIRA Proxy Configuration:
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyRequests Off
ProxyPreserveHost On
ProxyPass / http://codemonkey:8082/
ProxyPassReverse / http://codemonkey:8082/
</VirtualHost>
Now we enable the site and restart apache:
sudo a2ensite jira.conf
sudo service apache2 reload
The only issue is that if I use http://jira.myserver.com I get the defaut site, but if I type http://jira.myserver.com/ I get Jira. It needs that trailing slash. But ok, that shouldn't be an issue. Finally I need to log on to Jira as admin, and set the base URL so it references the new external name.
So far, so good, next is nexus. Nexus has an issue: it has the name nexus in the path. I'm sure there a way to get rid of that, but for now, I'll keep it just to make things easy. Otherwise the redirects don't work. In order to allow the name nexus in the path I need to add it to the two slashes in the jira.conf. So now the config file looks like this:
<VirtualHost *:80>
ServerName nexus.myserver.com
ServerAdmin randysr@kamradtfamily.net
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# Nexus Proxy Configuration:
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyRequests Off
ProxyPreserveHost On
ProxyPass /nexus http://codemonkey:8081/nexus
ProxyPassReverse /nexus http://codemonkey:8081/nexus
</VirtualHost>
Again, not a major problem, I just need to use the path http://nexus.myserver.com/nexus. Now I can set up my mirror in my settings.xml to point to myserver, and I can use Maven to code at my local coffee shop.
Finally I need to install and configure Confluence and set it up as my default server. It's pretty similar to Jira, the first step is to run the bin file. Run it as root to be able to set it up as a service. In this case you can use all default values as it uses ports 8090 and 8000, which are not currently used. Test out the install by browsing to http://localserver:8090. So since it doesn't use a path, I can now copy the apache config file from jira.conf to confluence.conf and just change the ServerName and ports. I can restart apache and now run the configuration from the external URL. First thing we need to do is to create a database for confluence just like we did for jira. You can refer to my previous blog for the details, but I just create a user called confluence and a database called confluencedb. Back at the confluence app, it will ask for the parameters you just created. The default database is confluence, and I changed it to confluencedb so I could create a user named confluence. I'm pretty new to postgres, and I'm not sure how it would deal with a user and database having the same name. Plus it makes it consistent with the naming I used for jira. Next it wants to know if I want an empty site or a demo site. I hate demo sites, they are often hard to remove once you get going, and hang around like an appendix ready to be exploited once someone find a loophole. So I start with an empty site, and I'll do a little research before I really start making spaces/pages. Next it asks if I want to import the users from jira, and I can't think of why not. That all works without a hitch. Now comes the fun of configuration spaces and pages in Confluence. But I've done enough work today. I'll finish up configuration later. I may or may not blog about the specifics, but I'll probably have something to say about using all of these apps, and how well they work together along with all the other development stuff I have. Since I installed all these to assist my programming, I'll probably get back to some programming, which I'll talk about on my programming blog http://rlkamradt.wordpress.com So see you there!
Installing a development server part II
I'm back from my break and ready to install and configure some software. I'll start with Jenkins first as it's probably the easiest, and doesn't require a lot of configuration. The actual install is just adding their binary repository to the trusted repository in Ubuntu, and running the normal installer. It's four lines of code:
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo vi /etc/apt/sources.list ## add to the end: deb http://pkg.jenkins-ci.org/debian binary/
sudo apt-get update
sudo apt-get install jenkins
You'll see a bunch of scrolling after the last two lines. Once it's done, Jenkins is installed as a service. You can go to the site to make sure it's running: http://yourserver:8080/ If you had Tomcat or something else running on 8080, I'm not sure what the installer would do, I'll leave that as an exercise for the student. You don't have to worry about securing it, unless you plan on exposing port 8080, reverse proxying it from Apache httpd, or you don't trust your cats, since it can only be seen on the internal network. I doubt I'll expose it to the world, as I don't care to check if my builds are failing while I'm out having coffee somewhere. Application #1 installed.
Next I'm going to install Nexus. Nexus runs without much installation. Here's the instructions from their site:
sudo cp nexus-2.8.0-05-bundle.tar.gz /usr/local
cd /usr/local
sudo tar xvzf nexus-2.8.0-05-bundle.tar.gz
ln -s nexus-2.8.0-05 nexus
If you noticed, the last line will probably fail, as they don't say simon says (sudo) This is assuming that you want it installed in /usr/local so that it can be run as a service. The /usr/local might be arbitrary, and you might be able to install it in /opt or some other directory, but I tend to follow the instructions to the T, as that's probably what they've tested with, and you never know where they may have hardcoded /usr/local into the code or some script. The next thing they tell you to do to run it as a service is:
./bin/nexus start
But that's going to fail. First they forgot to mention the whole 'cd nexus' thing that needs to happen. Then there's a little thing call user permission that will get in the way. There's a couple things you can do: run it as yourself, or create a 'nexus' user (perhaps without a shell account so no one can actually log on as nexus), and then chown -R the various directories to the user running nexus. I'm going to make it simple and run it as myself. I won't add it to the /etc/init.d directory so it won't start on boot, and I'll have to make sure I go back and restart it if the computer is rebooted for whatever reason. Not a big concern for me, but a real company whose developers depend on nexus may want to figure out a way to start it automatically on boot. Now you can try it out by browsing to http://servername:8081/nexus. Just in case you're wondering the admin username/password is admin/admin123. That's something you might want to change, as you may want to reverse proxy it to expose your repository so you can code away from home. You might be able to get away with only reverse proxying the pertinent subdirectories, but I would still change the admin password, you never know how a clever hacker might be able to get into the system once you've exposed a part of it. Nexus runs pretty well out of the box without a lot of configuration, but you may want to add additional repositories like java.net or jboss or your favorite Maven repositories. Since the './bin/nexus start' command runs it as a daemon, you don't have to worry about logging off your server, it will still run in the background. Application #2 installed.
Next is Jira. This has a fairly easy installation, set the jira-*.bin file to executable and run it:
randalkamradt@Codemonkey:~$ chmod +x *.bin
randalkamradt@Codemonkey:~$ sudo ./atlassian-jira-6.2.3-x64.bin
By doing the chmod on *.bin, I can change both Jira and Confluence. Just make sure you don't have any other *.bin files laying around your user directory, they'll get the execute flag set as well, which could lead to some hilarious consequences. Don't worry about running the install as root, that's just so it can install itself as a service like a good application. After a bunch of scrolling you get some prompts, you can take the default on most of them, except one. The default is a custom install, which you want because the default port is 8080, and that would cause a bloody battle between Jira and Jenkins, and I wouldn't want to watch such a spectacle. It requires two ports, an HTTP port and a control port. I set the HTTP port to 8082 and the control port to 8007 (remember 8081 is taken by nexus). Once it's finish you can browse to http://servername:8082, BUT only if you have a few minutes to set it up. On first start up it will run you through the configuration process. You will need to make sure your mail and postgres servers are running. If you did the installation like I went through in the last post, mail should already be running, and you can check it with 'telnet localhost 25' which will connect your terminal to port 25 where the SMTP server is running. If you get a response that starts with 220, you're probably good. Now you need to setup postgres with at least a postgres user and a database. Here's the instruction from Atlassian:
user:~$ sudo su - postgres
postgres:~$ createuser -P jira
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n
CREATE ROLE
postgres:~$ logout
user:~$ sudo su - jira
jira:~$ createdb jiradb
CREATE DATABASE
Opps, a few more little mistakes. The createuser postgres command need to take the --interactive flag to give all the prompts listed. Also, the Jira install created the jira system user (not to be confused with the jira database user we just created), and set its home directory to /home/jira, but didn't create that directory. To keep the system from complaining, I created the directory and set the owner to the jira system user. Otherwise this worked ok. Now we can browser to http://servername/8082 and begin the configuration process.
The first thing it ask is your language. I set mine to pirate (just kidding, they haven't translated it to pirate yet). Then you need to supply the information about the database. You just set up the Postgres database so I suggest you use that one with localhost as the server:5432 (my favorite random port), database is jiradb, user is jira, schema is public, password is whatever you used when you set up the user. There's a handy test connection button, press it and it will tell you everything's ok. If it's not, I suggest you keep your day job as long as your day job isn't setting up databases.
Next it will ask you for the 'title', I suggest Bug Zoo, or Insect Faire. It will ask you if it's public or private mode, that is can anyone add users or only the admin. If you plan on exposing this to the Internet, I would suggest private mode, or spammers will somehow find a way to send spam through your system. Even in private mode, the admin person can 'invite' a user, where the user will get an email and be able to click a link to sign up. Finally you'll be asked for a base URL. This maybe important if you're planning on reverse proxying, but unless you have your reverse proxy setup, you should leave this as the internal URL and hope you can configure it later to the external URL. On the next screen you will be asked for a license. I you already have a license, you can enter it, otherwise you can check one of the boxes and it will walk you through getting an eval license. The next screen you will set up an admin user, fill it in with the usual stuff. One thing to keep in mind, if you create two accounts for yourself, one for administration and one for general use, you'll need separate email addresses. The next screen will prompt if you want to set up your email system. Might as well do it now, if you've already tested your SMTP server it should be easy. You select a name which is just an arbitrary name (I choose monkeymail) the from email, which they pull from the admin email, a prefix which they add to all emails from the Jira system so you can filter them to the trash folder, A server type, you probably want SMTP and provider, you probably want custom, although you might just hook it straight up to a gmail or Yahoo mail server. I chose custom, since I went through the trouble of setting up an SMTP server for situations like this. The hostname is localhost, and the rest of the parameters should be default. Once again, there's a test button which you should click to make sure everything's hunky-dorey. Getting past this page should take you too the System Dashboard. There are a lot of other things to get setup, but I'll leave that to you. The only thing I would advise is to go to User Management and use the email invitation to invite as many users as you know will be using the system, as when you install Confluence next, it can import all the Jira users.
I'll take another break here and next I will install Confluence, work on some of the reverse proxies, and do some more configuration of the various components.
The fun stuff is yet to come.
wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
sudo vi /etc/apt/sources.list ## add to the end: deb http://pkg.jenkins-ci.org/debian binary/
sudo apt-get update
sudo apt-get install jenkins
You'll see a bunch of scrolling after the last two lines. Once it's done, Jenkins is installed as a service. You can go to the site to make sure it's running: http://yourserver:8080/ If you had Tomcat or something else running on 8080, I'm not sure what the installer would do, I'll leave that as an exercise for the student. You don't have to worry about securing it, unless you plan on exposing port 8080, reverse proxying it from Apache httpd, or you don't trust your cats, since it can only be seen on the internal network. I doubt I'll expose it to the world, as I don't care to check if my builds are failing while I'm out having coffee somewhere. Application #1 installed.
Next I'm going to install Nexus. Nexus runs without much installation. Here's the instructions from their site:
sudo cp nexus-2.8.0-05-bundle.tar.gz /usr/local
cd /usr/local
sudo tar xvzf nexus-2.8.0-05-bundle.tar.gz
ln -s nexus-2.8.0-05 nexus
If you noticed, the last line will probably fail, as they don't say simon says (sudo) This is assuming that you want it installed in /usr/local so that it can be run as a service. The /usr/local might be arbitrary, and you might be able to install it in /opt or some other directory, but I tend to follow the instructions to the T, as that's probably what they've tested with, and you never know where they may have hardcoded /usr/local into the code or some script. The next thing they tell you to do to run it as a service is:
./bin/nexus start
But that's going to fail. First they forgot to mention the whole 'cd nexus' thing that needs to happen. Then there's a little thing call user permission that will get in the way. There's a couple things you can do: run it as yourself, or create a 'nexus' user (perhaps without a shell account so no one can actually log on as nexus), and then chown -R the various directories to the user running nexus. I'm going to make it simple and run it as myself. I won't add it to the /etc/init.d directory so it won't start on boot, and I'll have to make sure I go back and restart it if the computer is rebooted for whatever reason. Not a big concern for me, but a real company whose developers depend on nexus may want to figure out a way to start it automatically on boot. Now you can try it out by browsing to http://servername:8081/nexus. Just in case you're wondering the admin username/password is admin/admin123. That's something you might want to change, as you may want to reverse proxy it to expose your repository so you can code away from home. You might be able to get away with only reverse proxying the pertinent subdirectories, but I would still change the admin password, you never know how a clever hacker might be able to get into the system once you've exposed a part of it. Nexus runs pretty well out of the box without a lot of configuration, but you may want to add additional repositories like java.net or jboss or your favorite Maven repositories. Since the './bin/nexus start' command runs it as a daemon, you don't have to worry about logging off your server, it will still run in the background. Application #2 installed.
Next is Jira. This has a fairly easy installation, set the jira-*.bin file to executable and run it:
randalkamradt@Codemonkey:~$ chmod +x *.bin
randalkamradt@Codemonkey:~$ sudo ./atlassian-jira-6.2.3-x64.bin
By doing the chmod on *.bin, I can change both Jira and Confluence. Just make sure you don't have any other *.bin files laying around your user directory, they'll get the execute flag set as well, which could lead to some hilarious consequences. Don't worry about running the install as root, that's just so it can install itself as a service like a good application. After a bunch of scrolling you get some prompts, you can take the default on most of them, except one. The default is a custom install, which you want because the default port is 8080, and that would cause a bloody battle between Jira and Jenkins, and I wouldn't want to watch such a spectacle. It requires two ports, an HTTP port and a control port. I set the HTTP port to 8082 and the control port to 8007 (remember 8081 is taken by nexus). Once it's finish you can browse to http://servername:8082, BUT only if you have a few minutes to set it up. On first start up it will run you through the configuration process. You will need to make sure your mail and postgres servers are running. If you did the installation like I went through in the last post, mail should already be running, and you can check it with 'telnet localhost 25' which will connect your terminal to port 25 where the SMTP server is running. If you get a response that starts with 220, you're probably good. Now you need to setup postgres with at least a postgres user and a database. Here's the instruction from Atlassian:
user:~$ sudo su - postgres
postgres:~$ createuser -P jira
Enter password for new role:
Enter it again:
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n
CREATE ROLE
postgres:~$ logout
user:~$ sudo su - jira
jira:~$ createdb jiradb
CREATE DATABASE
Opps, a few more little mistakes. The createuser postgres command need to take the --interactive flag to give all the prompts listed. Also, the Jira install created the jira system user (not to be confused with the jira database user we just created), and set its home directory to /home/jira, but didn't create that directory. To keep the system from complaining, I created the directory and set the owner to the jira system user. Otherwise this worked ok. Now we can browser to http://servername/8082 and begin the configuration process.
The first thing it ask is your language. I set mine to pirate (just kidding, they haven't translated it to pirate yet). Then you need to supply the information about the database. You just set up the Postgres database so I suggest you use that one with localhost as the server:5432 (my favorite random port), database is jiradb, user is jira, schema is public, password is whatever you used when you set up the user. There's a handy test connection button, press it and it will tell you everything's ok. If it's not, I suggest you keep your day job as long as your day job isn't setting up databases.
Next it will ask you for the 'title', I suggest Bug Zoo, or Insect Faire. It will ask you if it's public or private mode, that is can anyone add users or only the admin. If you plan on exposing this to the Internet, I would suggest private mode, or spammers will somehow find a way to send spam through your system. Even in private mode, the admin person can 'invite' a user, where the user will get an email and be able to click a link to sign up. Finally you'll be asked for a base URL. This maybe important if you're planning on reverse proxying, but unless you have your reverse proxy setup, you should leave this as the internal URL and hope you can configure it later to the external URL. On the next screen you will be asked for a license. I you already have a license, you can enter it, otherwise you can check one of the boxes and it will walk you through getting an eval license. The next screen you will set up an admin user, fill it in with the usual stuff. One thing to keep in mind, if you create two accounts for yourself, one for administration and one for general use, you'll need separate email addresses. The next screen will prompt if you want to set up your email system. Might as well do it now, if you've already tested your SMTP server it should be easy. You select a name which is just an arbitrary name (I choose monkeymail) the from email, which they pull from the admin email, a prefix which they add to all emails from the Jira system so you can filter them to the trash folder, A server type, you probably want SMTP and provider, you probably want custom, although you might just hook it straight up to a gmail or Yahoo mail server. I chose custom, since I went through the trouble of setting up an SMTP server for situations like this. The hostname is localhost, and the rest of the parameters should be default. Once again, there's a test button which you should click to make sure everything's hunky-dorey. Getting past this page should take you too the System Dashboard. There are a lot of other things to get setup, but I'll leave that to you. The only thing I would advise is to go to User Management and use the email invitation to invite as many users as you know will be using the system, as when you install Confluence next, it can import all the Jira users.
I'll take another break here and next I will install Confluence, work on some of the reverse proxies, and do some more configuration of the various components.
The fun stuff is yet to come.
Installing a development server
I got myself in a mess by using Ubuntu's aptitude package to remove a package I accidentally install (I installed full emacs instead of the no X version, so it install X which I don't want). Unfortunately when it removed packages that no longer had dependencies because emacs was no longer there, it removed a LOT of needed packages. Including reboot, which the system said it needed after all the packages were removed. I figured that I had only spent a few hours in setting up the server (codemonkey) that I would just do a reinstall of everything. So I'm going to document step-by-step the process here.
Of course first thing is I get my trusty Ubuntu 14.04 CD, stick it in the CD drive and hit the reboot button. It comes up and asks me several times if I speak english (and what kind of english, I suppose if I said I spoke British english it would use British spellings like colour instead of color). It asks if it wants it to detect the type of keyboard I have, which I chose the default no, as that's gotten me in trouble before (it asks if I have an umlaut, and I though it was asking if I have a double quote...duh). Next it ask which of the three network adaptors I have is the primary, and I pick the first one (I have to set up the other two manually later). It uses the DHCP system to auto setup the network, so it can download packages from the Internet (you didn't think the whole system would fit on a CD did you?) next it asks for the system name, and some figured out that it was Codemonkey, so I accept that. Next it asks for the first user name, as it needs at least one user, that is marked as sudo capable. I entered my name, user name, and usual password. It asks me if I really want to use my weak-ass password, to which I respond yes, as I prefer not to be judged by my computer. Then it asks if I live in Los Angeles so it can set my timezone, and I lie to it and answer yet (although I HAVE lived in Los Angeles, I have fortunately since escaped).
The next step is to partition my hard drive. Since I already have a system, albeit broken, it gets very concerned and now I have to take non-default answers, and it ask me again and again, do you really want to write over this fine system? Are you really really sure? Ok, this is your last chance! Is that your final answer? YES, YES, YES, YES. So finally I musters enough confidence to format my hard drive with the default set-up...swap...boot...LVM. Then it asks if I use an HTTP proxy to access the Internet (as if anyone really does that anymore). It asks how I want to handle upgrades, and my typical response is to automatically install security patches. It's not really automatic, as sometimes it requires a reboot of my system, so I have to logon occasionally and see the 'reboot required' message pops up and reboot as needed (It's not as bad as Windows which needs a reboot if you look at it wrong).
Next it presents me with some typical server packages that I might want to install. I pick OpenSSH, LAMP, Postgres, and Mail. I can't imagine not using ssh, as it is my main way of communicating with the server from my Mac in my easy chair, I need LAMP for the Apache server, Postgres because Jira and Confluence need it, as Oracle makes it difficult to use MySQL which I imagine they're trying to kill so they can get back to the business of selling software instead of giving it away. And I need mail so that Hudson can send me mail everytime I break something. It asks some configuration questions for those packages, so I enter the MySQL root password, the domain name for my mail server and basic setup for my mail server, and the common name for the self-signed certificate that I will be using, which I just enter as 'codemonkey'. If I create a real SSL certificate (which now costs 50 dollars a year from godaddy), I'll have to pick one of the external host names that all point back to codemonkey. It's a real pain that you have to have a separate official certificate for every hostname, I wish I could just have a single certificate for the domain that would cover all the hosts. You may have seen warning from sites, if you just enter the domain name instead of 'www' dot domain name and it says the the certificate was issued to www.blahblahblah because they only got a certificate for the www host and not the domain. Fortunately only the browser cares if the name on the certificate is right, all other programs blithely accept the certificate even if the common name is hackers.inc, and just use it to create the secure connection. Since I probably won't shell out 50 more bucks a year to godaddy, anyone that tries to access my website will have to get past the browsers security warning about trusting someone called 'codemonkey' and add my certificate to their list of trusted certificates. Now I'll get off my soap box and back to the install.
It want's to know if I want to add the boot program 'Grub' to my master boot record, which I answer the default yes. I like to know that my master boot record on my hard drive points to grub so I can make sure I'm not compromised by a rootkit virus which are very hard to detect and eliminate. Finally it ejects my CD so I can remove it and not accidentally re-install my system again, and continue to reboot into my new system. When the system reboots, I logon, and find that it needs a couple dozen updates, about 10 of which are security updates. So I run the command:
sudo apt-get upgrade
And with that my system is setup as a basic server. Now to configure ssh so that all my (Unix/Linux) computers can talk to each other. I do have one Unix system, my Mac. Surprising to many people that Macs are the antithesis of command line computer systems, one can drop down into a command shell and viola! Unix. Ha ha, fooled ya with my fancy UI. Actually it's part of what makes Macs so resilient and secure. From now on, all my new computers will be Macs, Windows can go away and lick its vulnerabilities.
To make your systems talk to each other, they have to exchange keys. It's like giving a spare house key to a trusted relative, so they can turn off the water you left on when you go off on vacation. At this point I can do the rest of the install from my Mac. But first, I should start some downloads (which I overconfidently deleted from my system after install them). I need four pieces of software to make my system complete: Jira, Confluence, Jenkins, and Nexus. Jira and Confluence are 10 dollars each (when I finally get my real licenses, I still have a week on my eval license). Jenkins and Nexus are free, although they have 'pro' versions incase you're a moron or middle management (middle management being a subset of the moron class (excepting my own manager in case he's reading this)), and need help installing or configuring the systems. Here's the URL's, you'll have to navigate to the download pages yourself, as they often change. (Jira me to down load the OSX version since I'm on my Mac, and I had to reassure it that I really needed the linux 64-bit installer)
https://www.atlassian.com/software/jira
https://www.atlassian.com/en/software/confluence
http://pkg.jenkins-ci.org/debian/
http://www.sonatype.org/nexus
Jenkins isn't actually a download, it's instructions on how to hack your system to be able to automatically download from their repository.
So now I need to do the key exchange. Technically I could have simply saved the .ssh directory on my server that I reinstalled, and replaced it after the reinstall. I've never actually done that, so I can't guarantee that it would work though. I open a command prompt on my Mac (if you don't know how to do this, perhaps you shouldn't be running a server). I have already set up my router to have DHCP reservation for my servers by MAC address, so the names available instead of using IP addresses. If you don't have such a system in place you may have to use IP address. Good luck with that. so here's the conversation:
I guess my Mac got suspicious about codemonkey not having an RSA key yet (as I'm about to set it up). If this is your first install, you won't get it, I only get it because I reinstalled the system. If you don't get this, you can skip to the next paragraph. Here's where we have to do some hacking. 'cd' to the '.ssh' directory (don't worry if you don't see it, it's hidden). Edit the authorized_keys file, find the line that starts with 'ssh-rsa' and ends with your servername, and delete it. That was the old key and we need to generate a new one. Next, edit the known_hosts file. You may find a lot of hosts here if you've got a githup account for instance, or other online accounts that require some kind of ssh access. You're going to find the line that starts with your servername (or server IP address or both separated by a comma) If the keys look like gibberish, that's good, they are basically very long passwords that make them very hard to guess. If you're using emacs like me and find the '~' backup files annoying, you should delete them now to avoid further annoyance. You will have to do this with any other servers that have exchanged keys with the old server as well. In case your key in known_hosts doesn't have the server name in it, you can look at the message above and find the line 'Offending RSA key in /Users/randalkamradt/.ssh/known_hosts:7' and use the number at the end in place of the servername.
Now when you ssh to the host you should get a new message about it not being recognised:
Of course first thing is I get my trusty Ubuntu 14.04 CD, stick it in the CD drive and hit the reboot button. It comes up and asks me several times if I speak english (and what kind of english, I suppose if I said I spoke British english it would use British spellings like colour instead of color). It asks if it wants it to detect the type of keyboard I have, which I chose the default no, as that's gotten me in trouble before (it asks if I have an umlaut, and I though it was asking if I have a double quote...duh). Next it ask which of the three network adaptors I have is the primary, and I pick the first one (I have to set up the other two manually later). It uses the DHCP system to auto setup the network, so it can download packages from the Internet (you didn't think the whole system would fit on a CD did you?) next it asks for the system name, and some figured out that it was Codemonkey, so I accept that. Next it asks for the first user name, as it needs at least one user, that is marked as sudo capable. I entered my name, user name, and usual password. It asks me if I really want to use my weak-ass password, to which I respond yes, as I prefer not to be judged by my computer. Then it asks if I live in Los Angeles so it can set my timezone, and I lie to it and answer yet (although I HAVE lived in Los Angeles, I have fortunately since escaped).
The next step is to partition my hard drive. Since I already have a system, albeit broken, it gets very concerned and now I have to take non-default answers, and it ask me again and again, do you really want to write over this fine system? Are you really really sure? Ok, this is your last chance! Is that your final answer? YES, YES, YES, YES. So finally I musters enough confidence to format my hard drive with the default set-up...swap...boot...LVM. Then it asks if I use an HTTP proxy to access the Internet (as if anyone really does that anymore). It asks how I want to handle upgrades, and my typical response is to automatically install security patches. It's not really automatic, as sometimes it requires a reboot of my system, so I have to logon occasionally and see the 'reboot required' message pops up and reboot as needed (It's not as bad as Windows which needs a reboot if you look at it wrong).
Next it presents me with some typical server packages that I might want to install. I pick OpenSSH, LAMP, Postgres, and Mail. I can't imagine not using ssh, as it is my main way of communicating with the server from my Mac in my easy chair, I need LAMP for the Apache server, Postgres because Jira and Confluence need it, as Oracle makes it difficult to use MySQL which I imagine they're trying to kill so they can get back to the business of selling software instead of giving it away. And I need mail so that Hudson can send me mail everytime I break something. It asks some configuration questions for those packages, so I enter the MySQL root password, the domain name for my mail server and basic setup for my mail server, and the common name for the self-signed certificate that I will be using, which I just enter as 'codemonkey'. If I create a real SSL certificate (which now costs 50 dollars a year from godaddy), I'll have to pick one of the external host names that all point back to codemonkey. It's a real pain that you have to have a separate official certificate for every hostname, I wish I could just have a single certificate for the domain that would cover all the hosts. You may have seen warning from sites, if you just enter the domain name instead of 'www' dot domain name and it says the the certificate was issued to www.blahblahblah because they only got a certificate for the www host and not the domain. Fortunately only the browser cares if the name on the certificate is right, all other programs blithely accept the certificate even if the common name is hackers.inc, and just use it to create the secure connection. Since I probably won't shell out 50 more bucks a year to godaddy, anyone that tries to access my website will have to get past the browsers security warning about trusting someone called 'codemonkey' and add my certificate to their list of trusted certificates. Now I'll get off my soap box and back to the install.
It want's to know if I want to add the boot program 'Grub' to my master boot record, which I answer the default yes. I like to know that my master boot record on my hard drive points to grub so I can make sure I'm not compromised by a rootkit virus which are very hard to detect and eliminate. Finally it ejects my CD so I can remove it and not accidentally re-install my system again, and continue to reboot into my new system. When the system reboots, I logon, and find that it needs a couple dozen updates, about 10 of which are security updates. So I run the command:
sudo apt-get upgrade
And with that my system is setup as a basic server. Now to configure ssh so that all my (Unix/Linux) computers can talk to each other. I do have one Unix system, my Mac. Surprising to many people that Macs are the antithesis of command line computer systems, one can drop down into a command shell and viola! Unix. Ha ha, fooled ya with my fancy UI. Actually it's part of what makes Macs so resilient and secure. From now on, all my new computers will be Macs, Windows can go away and lick its vulnerabilities.
To make your systems talk to each other, they have to exchange keys. It's like giving a spare house key to a trusted relative, so they can turn off the water you left on when you go off on vacation. At this point I can do the rest of the install from my Mac. But first, I should start some downloads (which I overconfidently deleted from my system after install them). I need four pieces of software to make my system complete: Jira, Confluence, Jenkins, and Nexus. Jira and Confluence are 10 dollars each (when I finally get my real licenses, I still have a week on my eval license). Jenkins and Nexus are free, although they have 'pro' versions incase you're a moron or middle management (middle management being a subset of the moron class (excepting my own manager in case he's reading this)), and need help installing or configuring the systems. Here's the URL's, you'll have to navigate to the download pages yourself, as they often change. (Jira me to down load the OSX version since I'm on my Mac, and I had to reassure it that I really needed the linux 64-bit installer)
https://www.atlassian.com/software/jira
https://www.atlassian.com/en/software/confluence
http://pkg.jenkins-ci.org/debian/
http://www.sonatype.org/nexus
Jenkins isn't actually a download, it's instructions on how to hack your system to be able to automatically download from their repository.
So now I need to do the key exchange. Technically I could have simply saved the .ssh directory on my server that I reinstalled, and replaced it after the reinstall. I've never actually done that, so I can't guarantee that it would work though. I open a command prompt on my Mac (if you don't know how to do this, perhaps you shouldn't be running a server). I have already set up my router to have DHCP reservation for my servers by MAC address, so the names available instead of using IP addresses. If you don't have such a system in place you may have to use IP address. Good luck with that. so here's the conversation:
megamac:~ randalkamradt$ ssh codemonkey
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Please contact your system administrator.
Add correct host key in /Users/randalkamradt/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/randalkamradt/.ssh/known_hosts:7
RSA host key for codemonkey has changed and you have requested strict checking.
Host key verification failed.
Now when you ssh to the host you should get a new message about it not being recognised:
megamac:.ssh randalkamradt$ ssh codemonkey
The authenticity of host 'codemonkey (192.168.1.132)' can't be established.
RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx;xx:.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'codemonkey,192.168.1.132' (RSA) to the list of known hosts.
But now codemonkey is recognized by megamac (the name for my macmini) and you should be able to logon. Because I use the same username on all systems, I don't have to specify a username, otherwise I'd have to use randalkamradt@codemonkey. If I want to exchange keys with another user on codemonkey, I'd have to prepend the username@ to the servername. Now I'm in codemonkey, I have to create a public and private key. This is for PSK authentication, where we create a public and private key, issue the public key to anyone and everyone, and they can encrypt anything they send with the public key, and in can only be decrypted with our private key (which we should keep safe and never give away) Here's how to do that:
randalkamradt@Codemonkey:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/randalkamradt/.ssh/id_rsa):
Created directory '/home/randalkamradt/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/randalkamradt/.ssh/id_rsa.
Your public key has been saved in /home/randalkamradt/.ssh/id_rsa.pub.
I don't use a passphrase, and that leave me vulnerable in case someone breaks into my house and steals all my private keys. But they'd have to get past my attack cat, Katie, first. I suppose I could also accidentally send people my private key, but I don't foresee that happening. Not having a passphrase means that programs on the two computers can communicate without me entering the passphrase or storing my password somewhere.
Next comes the key-exchange. You have to do this on both sides. Here's codemonkey exchanging with megamac:
randalkamradt@Codemonkey:~$ ssh-copy-id megamac
The authenticity of host 'megamac (192.168.1.102)' can't be established.
RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'megamac'"
and check to make sure that only the key(s) you wanted were added.
That should be the last time I have to enter megamac's password to access him from codemonkey. Notice that I had to add megamac to the list of known hosts. Now I have to do the reverse, giving codemonkey megamac's public key. The Mac version of Unix doesn't have the ssh-copy-id command, or I haven't found the right command, but I can do it the old fashion way: copy my public key from ~/.ssh/id_rsa.pub, ssh to codemonkey, and append it to ~/.ssh/authorized_keys. Now all my computers are speaking to each other, it's time to install the software.
All my downloads are complete, but now I have to transfer them to codemonkey. For this I can use scp:
megamac:~ randalkamradt$ scp atlassian-confluence-5.5-x64.bin codemonkey:~
atlassian-confluence-5.5-x64.bin 100% 237MB 7.9MB/s 00:30
megamac:~ randalkamradt$ scp atlassian-jira-6.2.3-x64.bin codemonkey:~
atlassian-jira-6.2.3-x64.bin 100% 204MB 8.9MB/s 00:23
megamac:~ randalkamradt$ scp nexus-2.8.0-05-bundle.tar.gz codemonkey:~
nexus-2.8.0-05-bundle.tar.gz 100% 43MB 8.7MB/s 00:05
And that should transfer them to codemonkey in my user directory. I'll finish up the work of installing the software at another time, as I need a break for now. But what we've done so far is substantial, we've installed a server with two databases, an Apache http server (go ahead and try http://servername and you'll see the 'it works' page) a mail server, and a SSH server. If you want these to be accessible to the Internet you'll need to punch holes in your router firewall (assuming you have a router) at ports 22, 25, 80, and 443 (if you want https). Most of these services are pretty secure, so opening up these ports is pretty safe, although I might replace the Apache 'it works' page with a more minimal 'it works' page, and don't punch holes for the databases, as they are not yet secure, and you probably don't want to expose them anyway, they're for internal use only.
To be continued...
Subscribe to:
Comments (Atom)