Working with Linux: Guiding Principles

In this post I will layout the basic principles that I have come to rely upon when working in a Linux environment to keep my data and code clean and nicely integrated. They may seem very simple and obvious to those who have worked with Linux for any time, but a) I don't see everyone else following them and b) coming from a Windows background it took me a while to work out what was important and how to manage things.

1st. Do not interfere with the base system. The bases system in this case is everything outside of your home directory. And by not interfering I mean do not add / remove files or edit configs other than using the distribution's standard UI (and try not to edit configs in /etc at all if you can help it). So for example, using Ubuntu, I will only put programs into /usr/lib/ using apt-get. This goes for svn builds of up-to-date libraries I am using - no "sudo make install" for me. It took me a while to realise how important this is for smooth operation of a machine. At first I just shoved my built code and 3rd party (not yet packaged) stuff into /usr/lib. I soon hit versioning problems and things became a mess. It also makes it hard to tell whether your code will run on another machine or not.

2nd. Document your changes. Rules are made to be broken, particularly the rule above - you will of course encounter some 3rd party lib that will only work if it is copied to /usr/share/whatever with a link in /etc, and if you want to use it you have to break the 1st rule and put it where it asks to be put. The thing is a) this should be rare and b) you should make a note of what you have done.

So how do we implement these policies in practice. Well, for the first rule you create a bin, lib, include and share directory in your home drive. With hindsight I think I should have created an opt directory too, so if you are starting from scratch do that. Then everything that would go in /usr/lib goes in ~/lib and so on for the other dirs. I have tried using /usr/local in a similar manner, but to be honest that worked out as more hassle than it was worth. Also, using your home directory will work on machines where you do not have admin rights or where your home dir is on a network share and accessed by multiple machines.

As for the second principle, I recommend keeping a text file on Dropbox or some other file sharing service (like UbuntuOne if you can get it to work). This document should also record config changes you have to make to get hardware working etc. That way when you need to undo these hacks you can easily see all the changes you have made. When you do this you will find that updating 3rd party libs is easy as you can just clean out the old version manually (or check that the automatic cleanup worked - it's quite amazing how many poorly written uninstall scripts don't remove everything they put there[1]). Also, if you suddenly want to work on another machine (got a new laptop or desktop machine) you know what changes you need to make to get things up and running. Or, when you have a catastrophic harddrive failure (believe me, they happen) you can get back up and running a lot faster. And finally, if you keep a note of all those config hacks you have been accumulating over time, you can try removing them when your distribution upgrades so you don't accumulate crud in /etc and can take advantage of improved services as they become available. This is particularly applicable to laptops - when you get a new laptop you often end up rewriting /etc to try to get wireless / mouse / soundcard working. In 12 moths stuff will work out the box, but unless you go back and restore the config files, your distribution will keep hold of the manual changes you have made, often with negative consequences.

While you are doing this, you might want to keep a list of the packages you have installed on the machine. This way when you change machines you can do so easily and quickly. In fact, if you can keep much of these documents as python scripts you are really onto a winner because then restoring (or moving) your entire work environment can be as easy as copying over your home directory and running a couple of scripts!

So those are the principles - don't go hacking around in the bowels of your system and keep a record of what you do. Pretty simple eh? Next post I will look at how we actually put these into practice and show you how to organise your work in some sensible directories and how to set up an environment so everything just works using the magic of .profile.

[1] - If you are writing something to install on a user's machine, write out an installation manifest to /var/lib/libmystuff/installation.manifest or somewhere else sensible and write to that every change you make to the user's system. Then people can do a clean uninstall of libmystuff3 even if they have deleted the installation files for libmystuff3 and already got hold of libmystuff4. You could even check for the file and do an automatic cleanup when installing a newer version!