Category Archives: System administration

Http response time monitoring with Munin

There’s nothing better than graphs, and for a web site there are few better things to graph than the response times for your web pages. While there are plenty of external services out there that will probe your web site and graph the results, it’s a good idea to do this on your own too.

Munin is a monitoring tool that can provide graphs in plentiful for your servers. Out of the box, or at least out of its Ubuntu box, it monitors a variety of system metrics and applications, but there is no bundled support for response-time monitoring.

Luckily, it’s really easy to extend Munin with new plugins, so I decided to write my own plugin for monitoring response times, which you can download from my bitbucket repository. It will produce graphs like the one below:

http_response_time_ example graph

Here the plugin has been configured to monitor three URLs in the same graph. Unlike normal Munin probes these URLs are external to the actual server running the plugin, but you could just as well monitor localhost URLs too.

To get the plugin up and running you first need to install Munin, if you haven’t already. For a one-server setup under Ubuntu, with master and client both running on the same machine, you do:

sudo apt-get install munin munin-node

This will start the munin-node service in the background, and also add the master-node cron jobs to /etc/cron.d/munin. To install the plugin itself you add the Python script to a folder of your choice, make it executable, and then symlink it from the Munin plugin folder, like so:

sudo cp ./http_response_time_ /usr/local/bin/
sudo chmod 755 /usr/local/bin/http_response_time_
sudo ln -s /usr/local/bin/http_response_time_ /etc/munin/plugins/http_response_time_example

If you want to monitor more than one set of URLs, and thus have more than one graph, you can accomplish that by creating one symlink for each graph that you need. The names of the symlinks are used as section titles when configuring the plugin in /etc/munin/plugin-conf.d/munin-node. For the graph shown above, the configuration would look something like this:

[http_response_time_example]
env.url1_url http://www.djangoproject.com/
env.url1_name django
env.url1_label djangoproject.com
env.url2_url http://rubyonrails.org/
env.url2_name rails
env.url2_label rubyonrails.org
env.url3_url http://php.net/
env.url3_name php
env.url3_label php.net

The plugin requires that the following environment variables are specified for each URL to be monitored:

  • urlX_url — the URL that should be monitored
  • urlX_name — Munin field name for the probe
  • urlX_label — legend description of the URL

The ‘X’ in the variable names above should be replaced with an incremental index for each URL, e.g. url1 and url2. In addition, the following environment variables are also supported:

  • graph_title — the title of the graph (default is “Response time”)
  • graph_category — the category Munin should show the graph in
  • request_timeout — the socket request timeout (same for all URLs)
  • urlX_warning — warning level for the probe (for Nagios)
  • urlX_critical — critical level for the probe (for Nagios)

Note that Munin uses its own timeout when fetching plugin data. The default value is 10 seconds, which also is the default value for the URL request timeout. Because of this it might be appropriate to increase the Munin fetch timeout so that it equals the number of URLs being monitored times the request timeout, to make sure all probes have time to run.

Once you have configured the plugin to your satisfaction you need to restart the Munin node to make it discover the new plugin:

sudo /etc/init.d/munin-node restart

Happy graphing!

Installing Postfix and Dovecot on Ubuntu 8.10

Continuing on the path of getting my VPS up and running, the time has come to install the actual mail services. For this I’ll go with the standard Ubuntu choices of Postfix as MTA and basic MDA, and Dovecot as IMAP server.

Step 1 — install Postfix
Postfix is easily installed from the main Ubuntu repository by:

sudo apt-get install postfix

This launches a dialog where you need to choose what kind of mail server you intend to operate, and which domain your server will maintain the mail for. I chose Internet Site and entered my domain name (e.g. example.com). The package installer automatically sets a number of default settings, which you can override by launching another interactive dialog:

sudo dpkg-reconfigure postfix

However, dialogs are pretty annoying so I’ll edit the settings manually instead through the convenient postconf utility, and then tell Postfix to reload its config:

sudo postconf -e 'smtpd_banner = $myhostname ESMTP $mail_name'
sudo postconf -e 'myhostname = mail.example.com'
sudo postconf -e 'mydestination = example.com, mail.example.com, hostname.example.com, localhost.example.com, localhost'
sudo postconf -e 'home_mailbox = Maildir/'
sudo postconf -e 'mynetworks ='
sudo postconf -e 'mynetworks_style = host'
sudo /etc/init.d/postfix reload

You obviously need to replace all occurrences of example.com with your actual domain name. These settings control which domains this server is the final destination for, which networks to relay mail for (only this host), and what mailbox format to use. I like Maildir since it stores each mail in a separate file, which is both robust and convenient. See the official Postfix documentation for more configuration options.

The /etc/aliases file controls which email aliases Postfix should use when delivering mail locally on the machine. By default, it will contain a mapping for postmaster to root, but we should also add a mapping for root to the user who should read root’s mail (e.g. johndoe), and make that active for Postfix through the newaliases command:

sudo sh -c 'echo "root: johndoe" >> /etc/aliases'
sudo /usr/bin/newaliases

Step 2 — create an SSL certificate
To use IMAP over SSL we need to create an SSL certificate to use with Dovecot. I’ll use a self-signed certificate since this is just my personal server with few users. First we generate a private key for the certificate and make it readable only by root, and then we create the certificate itself:

sudo openssl genrsa -out /etc/ssl/private/dovecot.key 2048
sudo chmod 400 /etc/ssl/private/dovecot.key
sudo openssl req -new -x509 -key /etc/ssl/private/dovecot.key -out /etc/ssl/certs/dovecot.pem -days 1095

Make sure to provide the actual domain name of your mail server, e.g. mail.example.com, when asked about the “Common Name”. Otherwise email clients will complain every time they connect to the server. Since this is a self-signed certificate not backed by a Certification Authority clients will complain the first time anyway, but if you save the certificate subsequent connects will go through silently. You can read more about openssl here.

Step 3 — install Dovecot
Time for Dovecot. It can act both as an IMAP server and a POP3 server, but I will only use its IMAP capability. Who uses POP3 nowadays anyway? Install the package and open the main config file:

sudo apt-get install dovecot-imapd
sudo vi /etc/dovecot/dovecot.conf

Change the following options to enable IMAP over SSL, tell Dovecot to use the Maildir mailbox format, and point it to the SSL key and certificate:

protocols = imaps
mail_location = maildir:~/Maildir
ssl_cert_file = /etc/ssl/certs/dovecot.pem
ssl_key_file = /etc/ssl/private/dovecot.key

That’s all! Finally, make sure that port 993 is open in the firewall and restart Dovecot with the new config:

sudo /etc/init.d/dovecot restart

Installing jabberd2 and MySQL on Ubuntu 8.10

I mentioned in my last post that I had got a new VPS to host my “my mail and some other things” on. One of those other things is a Jabber / XMPP server for instant messaging.

There are many XMPP servers out there, and at least three of them are available in Ubuntu’s software repository: jabberd14, jabberd2, and ejabberd. There is also the popular Openfire server, which isn’t available in the Ubuntu repository but a breeze to install nevertheless through its web-based configuration UI.

My VPS does not come with a whole lot of memory, so I needed a Jabber server with a small memory footprint. Although you can trim the Java-based Openfire down to fairly low levels of memory usage, it’s at a disadvantage compared to the C/C++ based jabberd14 and jabberd2. The same goes for the Erlang-based ejabberd, so it came down to jabber14 or jabberd2. Considering that jabberd14 seems pretty dead with no updates since 2007, I chose jabberd2. This is how I installed it:

Step 1 — add universe to sources.list
The universe repository component needs to be enabled in /etc/apt/sources.list, see step 4 in my last post for how to do this.

Step 2 — install MySQL
jabberd2 can be used with several different storage and authentication backends. I prefer the default Ubuntu choices of MySQL for both storage and authentication, as I use MySQL for different things too and I like simplicity. If you don’t have it installed already, get it by:

sudo apt-get install mysql-server

Still hunting for a small memory footprint, I also switched the default MySQL config to the stock config for systems with little memory:

sudo cp /etc/mysql/my.cnf /etc/mysql/my.cnf.backup
sudo cp /usr/share/doc/mysql-server-5.0/examples/my-small.cnf /etc/mysql/my.cnf
sudo /etc/init.d/mysql restart

Somewhat disappointingly, MySQL still hogs more than 100 MB of precious memory, which is a bit funny since the config claims it’s meant to be used for systems with less than 64 MB of memory. ;-) However, if you don’t plan to use InnoDB tables, the magic-wand solution is to add the skip-innodb directive to the [mysqld] section of /etc/mysql/my.cnf. This alone brought the memory usage down to just 15 MB after startup for me.

Step 3 — install jabberd2
Get jabberd2 from universe by:

sudo apt-get install jabberd2

For some reason it seems to start up by default even though it’s not properly set up yet, so let’s shut it down while we’re configuring it:

sudo /etc/init.d/jabberd2 stop

Then create the necessary database and tables in MySQL:

gzip -d /usr/share/doc/jabberd2/db-setup.mysql.gz
mysql -u root -p
mysql> \. /usr/share/doc/jabberd2/db-setup.mysql
mysql> GRANT select,insert,delete,update ON jabberd2.* to jabberd2@localhost IDENTIFIED by 'secret';
mysql> quit

Replace ‘secret’ in the grant statement above with the password you want to use for the jabberd2 user. You’ll need this password in the steps below.

Step 4 — configure jabberd2
Essentially, configuring jabberd2 for MySQL boils down to three basic steps, decently covered in the official installation guide:

Additionally, it makes sense to also change the password of the jabberd2 backbone router component in router-users.xml and each of the component configuration files.

Although there are other configuration options available too, the above steps should be enough for a basic installation. Start jabberd2 up again through:

sudo /etc/init.d/jabberd2 restart

At this point, you should be able to connect to the server and register with an IM client of your choice, e.g. Miranda. Don’t forget to open up the Jabber ports in your firewall: port 5222 for normal connections, 5223 for SSL connections, and 5269 for server-to-server connections.

Basic sshd and iptables security in Ubuntu

I recently got a new VPS for hosting my mail and some other things. The server was installed with a minimal Ubuntu 8.10 distribution, which basically meant that nothing except init, syslogd, and sshd was running after boot. Before doing anything else with it, here’s what I did to lock it down a bit security-wise:

Step 1 — Add user account
It’s good practice to not do stuff logged in as the root user, so the first step is to add a user account from which we can sudo. Log in as root only this time and add the user (e.g. johndoe) and sudo access as follows:

useradd -m johndoe -s /bin/bash
passwd johndoe
visudo

Running visudo will edit the /etc/sudoers file with the default editor of your environment. Add this line to allow johndoe full sudo permissions:

johndoe ALL=(ALL) ALL

Step 2 — Install iptables
Still as root, we’ll set up the iptables firewall to make sure only specifically allowed inbound network traffic is allowed to the server:

apt-get install iptables

Configuring iptables can seem pretty complex at first, but here’s a decent tutorial. Firewall rules can be added directly from the command line:

iptables -F
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p icmp --icmp-type any -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 993 -j ACCEPT
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited
iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited

Essentially, the above rules allow all outbound traffic, block all inbound traffic by default, and specifically allow ssh, smtp, http, and imaps traffic, which is what I need to begin with. To make sure the rules are persistent after e.g. a server reboot we add them as a script hook to the network interface:

iptables-save > /etc/network/iptables
printf '#!/bin/sh\niptables-restore < /etc/network/iptables\n' > /etc/network/if-pre-up.d/iptables
chmod 754 /etc/network/if-pre-up.d/iptables

Step 3 — Update sshd config
Now that iptables and a user account with sudo rights are set up, it’s time to leave the root shell, log in as the new user, and then edit the sshd config to lock down future ssh access a bit:

sudo vi /etc/ssh/sshd_config

Add/change the following settings:

PermitRootLogin no
AllowUsers johndoe

This disallows root from logging in through ssh (console login is still allowed though) and restricts ssh access to only be allowed for the johndoe user. Make the changes active by reloading the config:

sudo /etc/init.d/ssh reload

Step 4 — Update sources.list
This being a very minimal install, the stock /etc/apt/sources.list file only included the main repository component. To install the denyhosts package (see next step) the universe component is needed, so we need to add it to sources.list:

deb http://archive.ubuntu.com/ubuntu intrepid main restricted
deb-src http://archive.ubuntu.com/ubuntu intrepid main restricted
deb http://archive.ubuntu.com/ubuntu/ intrepid-updates main restricted
deb-src http://archive.ubuntu.com/ubuntu/ intrepid-updates main restricted
deb http://archive.ubuntu.com/ubuntu/ intrepid universe
deb-src http://archive.ubuntu.com/ubuntu/ intrepid universe
deb http://archive.ubuntu.com/ubuntu/ intrepid-updates universe
deb-src http://archive.ubuntu.com/ubuntu/ intrepid-updates universe

Once done, update the package lists from the newly added sources and upgrade all currently installed packages to the latest versions:

sudo apt-get update
sudo apt-get upgrade

Step 5 — Install DenyHosts
Even with iptables and the sshd configuration changes we made, we still allow some users to log in through ssh, which makes us vulnerable to remote brute-force attempts to gain access through these accounts. One good way to do away with this threat is to only allow public-key authentication, or restrict access to only a list of specified IPs through iptables or /etc/hosts.deny, but if this is not practical for whatever reason the DenyHosts package comes to the rescue.

DenyHosts monitors the sshd authentication log to detect evil login attempts and adds suspicious IPs automatically to the /etc/hosts.deny file. It’s available in Ubuntu from the universe repository component (see previous step), and is easily installed like this:

sudo apt-get install denyhosts

This will automatically start a python daemon in the background, which also is persistent on reboot through a symlink in /etc/rc3.d/. The default settings are pretty decent, but should you want to review or change them you can do so in /etc/denyhosts.conf.

Ubuntu, cmemcache, and python-memcached (once again)

I wrote earlier about my not quite satisfactory attempt to install cmemcache on Ubuntu. After stumbling upon this exhaustive guide on how to set up a full-blown Django environment under Ubuntu I thought I’d try the apt-get route for libmemcache too. All you need to do to get libmemcache and the necessary header files installed is:

sudo apt-get install libmemcache-dev libmemcache0

After that, the steps for installing cmemcache itself are the same as before. The good news is that libmemcache and cmemcache both get installed without fuss using this approach, without any need for manual patching. The bad news is that the test.py suite still fails with same error message as before.

I cannot say that I fully understand the significance of this test failing. It might be harmless, but I sure like my tests to pass successfully so cmemcache will remain on my naughty list for now. Left with only python-memcached as a choice, I installed it instead, which was a breeze compared to cmemcache:

wget ftp://ftp.tummy.com/pub/python-memcached/python-memcached-1.43.tar.gz
tar xvfz python-memcached-1.43.tar.gz
cd python-memcached-1.43/
sudo apt-get install python-setuptools
sudo python setup.py install

python-setuptools is used by the setup.py script, so it needs to be installed before running the setup. Once done, you can test python-memcached almost identically to cmemcache earlier (memcached needs to run in the background on localhost for this to work):

>>> import memcache
>>> c = memcache.Client(['127.0.0.1:11211'])
>>> c.set('testkey', 'testval')
True
>>> c.get('testkey')
'testval'
>>>

Installing cmemcache on Ubuntu 8.10 (almost)

Despite my previous post about the confusing recommendation in the Django documentation to use cmemcache as Python client library for memcached I decided to give it a try.

cmemcache is based on the C library libmemcache, but for some reason the author includes a patch for libmemcache that should be applied before compiling. The only problem is that this patch doesn’t work. After patching, compiling, and installing libmemcache and then installing cmemcache, any attempt to import the cmemcache module in your Python code comes back with this error:

ImportError: /usr/local/lib/libmemcache.so.0: 
undefined symbol: mcm_buf_len

I’m not the first one experiencing this issue, as can be seen here and here. One of the comments in the second link mentions another patch, which supposedly should work better than the original one. Despite having an icky feeling about it, I went ahead and installed libmemcache with the fishy patch as follows:

wget http://people.freebsd.org/~seanc/libmemcache/libmemcache-1.4.0.rc2.tar.bz2
tar xvfj libmemcache-1.4.0.rc2.tar.bz2
cd libmemcache-1.4.0.rc2/
wget --no-check-certificate https://svn.pardus.org.tr/pardus/devel/programming/libs/libmemcache/files/libmemcache.patch
patch -p1 < libmemcache.patch
sudo apt-get install automake1.9 
./configure && make
sudo make install

The patch modifies the configure.ac file, which make first complained about, so I had to install automake1.9 before running make again. Once libmemcache is installed, cmemcache can be installed:

wget http://gijsbert.org/downloads/cmemcache/cmemcache-0.95.tar.bz2
tar xvfj cmemcache-0.95.tar.bz2
cd cmemcache-0.95/
sudo apt-get install python-dev
sudo python setup.py install

I had to install the python-dev package first because cmemcache depends on Python.h provided by that package. Running the test.py test suite after this actually works better than with the official patch earlier, but one test fails even so:

Traceback (most recent call last):
  File "test.py", line 254, in test_memcache
    self._test_base(cmemcache, 
                    cmemcache.StringClient(self.servers), ok=1)
  File "test.py", line 132, in _test_base
    self.failUnlessEqual(len(stats), 1)
AssertionError: 0 != 1

That being said, you can test cmemcache manually at this point and it actually seems to work a little bit (memcached should be running on localhost):

>>> import cmemcache
>>> c = cmemcache.StringClient(['127.0.0.1:11211'])
>>> c.set('testkey', 'testval')
1
>>> c.get('testkey')
'testval'
>>>

All things considered, this was a less than convincing experience with cmemcache, and the thought of running an unofficial and undocumented patch for an aging library in a production environment is not altogether appealing.

Update: See also my subsequent attempt of installing libmemcache through apt-get. Not really more successful, but easier at least.

Installing memcached from source on Ubuntu 8.10

In my previous post I wrote that installing memcached went fairly smoothly. I chose to install directly from source rather than from a package repository to make sure I had the most recent version available, because Ubuntu 8.10 ships with version 1.2.2 of memcached, whereas 1.2.6 is the latest. These are the steps that I took.

In case you are starting off from a clean install, like me, you might need to install the C/C++ compiler and Make in order to get going:

sudo apt-get install g++ make

Installing libevent
memcached is dependent on libevent, which is an event notification library, so we should install it first:

wget http://www.monkey.org/~provos/libevent-1.4.9-stable.tar.gz
tar xvfz libevent-1.4.9-stable.tar.gz
cd libevent-1.4.9-stable/
./configure && make
sudo make install

By default, libevent gets installed into /usr/local/lib/, which may or may not be a known library path for the dynamic linker in your system. To make sure, add it in like this:

sudo sh -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/libevent.conf'
sudo ldconfig -v

The verbose option -v to ldconfig is just there so you can watch the output and conclude that ldconfig sees /usr/local/lib/ and the libevent files.

Installing memcached
Installing memcached is very similar to the above steps:

wget http://danga.com/memcached/dist/memcached-1.2.6.tar.gz
tar xvfz memcached-1.2.6.tar.gz
cd memcached-1.2.6/
./configure && make
sudo make install

By default, memcached gets installed into /usr/local/bin/, and to test run it you can do:

/usr/local/bin/memcached -vv

Again, the double verbose option is there just so you can see that something is happening. For actual production use, you will want to use other options too, for example -d for daemon mode and -m for adjusting the memory limit.