Tag Archives: memcached

Installing cmemcache on Ubuntu 8.10 (almost)

Despite my previous post about the confusing recommendation in the Django documentation to use cmemcache as Python client library for memcached I decided to give it a try.

cmemcache is based on the C library libmemcache, but for some reason the author includes a patch for libmemcache that should be applied before compiling. The only problem is that this patch doesn’t work. After patching, compiling, and installing libmemcache and then installing cmemcache, any attempt to import the cmemcache module in your Python code comes back with this error:

ImportError: /usr/local/lib/libmemcache.so.0: 
undefined symbol: mcm_buf_len

I’m not the first one experiencing this issue, as can be seen here and here. One of the comments in the second link mentions another patch, which supposedly should work better than the original one. Despite having an icky feeling about it, I went ahead and installed libmemcache with the fishy patch as follows:

wget http://people.freebsd.org/~seanc/libmemcache/libmemcache-1.4.0.rc2.tar.bz2
tar xvfj libmemcache-1.4.0.rc2.tar.bz2
cd libmemcache-1.4.0.rc2/
wget --no-check-certificate https://svn.pardus.org.tr/pardus/devel/programming/libs/libmemcache/files/libmemcache.patch
patch -p1 < libmemcache.patch
sudo apt-get install automake1.9 
./configure && make
sudo make install

The patch modifies the configure.ac file, which make first complained about, so I had to install automake1.9 before running make again. Once libmemcache is installed, cmemcache can be installed:

wget http://gijsbert.org/downloads/cmemcache/cmemcache-0.95.tar.bz2
tar xvfj cmemcache-0.95.tar.bz2
cd cmemcache-0.95/
sudo apt-get install python-dev
sudo python setup.py install

I had to install the python-dev package first because cmemcache depends on Python.h provided by that package. Running the test.py test suite after this actually works better than with the official patch earlier, but one test fails even so:

Traceback (most recent call last):
  File "test.py", line 254, in test_memcache
                    cmemcache.StringClient(self.servers), ok=1)
  File "test.py", line 132, in _test_base
    self.failUnlessEqual(len(stats), 1)
AssertionError: 0 != 1

That being said, you can test cmemcache manually at this point and it actually seems to work a little bit (memcached should be running on localhost):

>>> import cmemcache
>>> c = cmemcache.StringClient([''])
>>> c.set('testkey', 'testval')
>>> c.get('testkey')

All things considered, this was a less than convincing experience with cmemcache, and the thought of running an unofficial and undocumented patch for an aging library in a production environment is not altogether appealing.

Update: See also my subsequent attempt of installing libmemcache through apt-get. Not really more successful, but easier at least.

Installing memcached from source on Ubuntu 8.10

In my previous post I wrote that installing memcached went fairly smoothly. I chose to install directly from source rather than from a package repository to make sure I had the most recent version available, because Ubuntu 8.10 ships with version 1.2.2 of memcached, whereas 1.2.6 is the latest. These are the steps that I took.

In case you are starting off from a clean install, like me, you might need to install the C/C++ compiler and Make in order to get going:

sudo apt-get install g++ make

Installing libevent
memcached is dependent on libevent, which is an event notification library, so we should install it first:

wget http://www.monkey.org/~provos/libevent-1.4.9-stable.tar.gz
tar xvfz libevent-1.4.9-stable.tar.gz
cd libevent-1.4.9-stable/
./configure && make
sudo make install

By default, libevent gets installed into /usr/local/lib/, which may or may not be a known library path for the dynamic linker in your system. To make sure, add it in like this:

sudo sh -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/libevent.conf'
sudo ldconfig -v

The verbose option -v to ldconfig is just there so you can watch the output and conclude that ldconfig sees /usr/local/lib/ and the libevent files.

Installing memcached
Installing memcached is very similar to the above steps:

wget http://danga.com/memcached/dist/memcached-1.2.6.tar.gz
tar xvfz memcached-1.2.6.tar.gz
cd memcached-1.2.6/
./configure && make
sudo make install

By default, memcached gets installed into /usr/local/bin/, and to test run it you can do:

/usr/local/bin/memcached -vv

Again, the double verbose option is there just so you can see that something is happening. For actual production use, you will want to use other options too, for example -d for daemon mode and -m for adjusting the memory limit.

Django and memcached library confusion

I’ve been playing around a bit with Python and Django lately, and I’m pretty impressed with them both so far. Django brings some nice features out of the box to the developer, one of them being the cache framework and the built-in support for memcached. I’ve never actually used memcached before, so I thought this would be a good time to try it out.

Installing memcached itself went fairly smoothly, but I got a bit confused when it was time for the Python client library. According to the documentation, Django supports cmemcache and python-memcached, of which cmemcache is the recommended option since it’s built on top of the C library libmemcache and almost twice as fast as the pure Python library.

On the other hand, the library used by cmemcache, i.e. libmemcache, is according to the memcached documentation wiki and this thread no longer under active development, which the early-2006 timestamps of the file listings agree with. It’s a bit strange that a seemingly abandoned library is recommended by the Django documentation, especially since memcached surely must have evolved a bit during the last 3 years.

To the detriment of both libraries, neither seem to support the consistent hashing introduced in memcached by libketama. Consistent hashing makes it possible to add or remove memcached nodes to a live environment without having all keys become remapped to different servers, effectively nuking the cache. This will not be a problem for all environments of course, but it makes dynamic scaling less attractive.

Consistent hashing is also supported by libmemcached (not to be confused with libmemcache), which is still actively maintained and used by python-libmemcached (not to be confused with python-memcached). Despite the low version number, it looks like a more appealing choice for Python clients, given that it’s actively developed, presumably fast, and supports consistent hashing. A pity that Django doesn’t support it.

So far this is just theory and speculation, of course, and only testing may tell what’s the best choice for a Django site. What’s the worst offense; the slower performance of python-memcached or the outdated underlying library of cmemcache? Does its age make any difference in practice? Thoughts, anyone?