Sunday, January 22, 2006

Migrating CVS from Debian to RedHat

A busy week behind me, moving our CVS repository from Debian to RedHat Enterprise 4 on Wednesday and Thursday.

Ready

Three years ago we began development of a new (Java) software solution for our main customer. To handle the several teams and the code they produced, we introduced our very first CVS server. Because many people in our company have a thorough liking for Windows we used a Windows 2000 Server installation with CVSNT. Nobody really cared about the warnings on the net using Eclipse (2.1 at the time) with CVSNT.

However it did not take too long until (almost) everybody realised that a Linux server should be the way to go. We went for a RedHat Enterprise Linux 2 but installed a VMware GSX server on it. Inside that VMware we created two virtual servers with Debian Woody. The one was to be a "dedicated" CVS server, the other one a build and test machine. For almost three years now this has been a (mostly) reliable setup. Sometimes the VMware would crash and take both virtual machines down with it, however this happened only about once per year.

Over these three years our repository has grown steadily to currently about 20GB in size, containing all sorts of stuff. Most of it in terms of the number of files is Java source code and ressource files, however there are also lots of office files, zips, exes, rpms and so on. Especially branching and merging began to show the limits of the existing hardware (Dual P4, 2.4GHz, 2GB RAM, RAID5 minus the overhead of VMware). Some time ago we already moved the build server to a separate physical machine, to ease the load and speed up CVS again. However in November 2005 we ordered a new server (Dual Xeon, 3.2GHz, 4GB RAM, SAN connection) which was finally (I won't elaborate on the unfortunate details between then and now) handed over to us on Wednesday, readily installed with RedHat Enterprise 4.

Set

The main goal was to keep downtime at a minimum and have the developers not notice anything apart from the increased speed. All accounts and passwords were to be migrated to the new machine. Moreover the ViewCVS (now ViewVC, see its new homepage) service was to be installed on the new machine, too.

First thing I did was copy the relevant lines of /etc/passwd and /etc/shadow over. Luckily the transition went from Debian to RedHat and not the other way round, because Debian assigns UIDs from 1000 up while RedHat already starts at 500. So there were no clashes there that would have required later updates to the file permissions and ownerships.

Then I rsynced the old servers three repositories to the new machine. I had thought about using some combination of netcat and tar, but I decided against it, because with rsync I could easily copy the data over and keep the old server online at the same time. Later I would just take it offline and transfer the delta, which I expecetd to save dramatically on the period of time during which the old server was already taken away and the new one not available yet.

Seemingly everything went fine, but some of the lines of rsync's output had scrolled by and somehow looked strange. When I looked a little closer I saw that all filenames containing non ASCII characters had been somehow corrupted. This being a German system with lots of people working on provided for quite some of these cases. The problem was the difference in file system name encodings between the somewhat older Debian installation and the new RedHat system. While the Debian system had stored all file names in ISO8859-15 (German, including the EURO Symbol) RedHat now used an UTF-8 based file name encoding. I did not realize till then, that file names are just bytestreams to the system and that whatever meaning they have soleley depends on the selected encoding of those who read them. This is somewhat different from Windows, where the file system contains some meta information about which encoding was used when the file was created. It did not dive into the details, but started looking a for a solution to this. In the end I came across a perl script called convmv that can recursively rename whole directory trees and translate between file name encodings on the fly. A dry run (very nice feature of the script, you have to explicitly state "--notest" on the command line to have it actually change something on your disk) looked good, so I decided to convert the filenames once the last rsync run would have finished.

Another way to go would have been to change RedHat's idea about file names to ISO8859-15, but as that would probably only have pushed the problem farther into the future but would not have solved it, I decided against it.

The next step was to make ViewCVS work on the new machine as well. Although on both Debian and RedHat I used Apache2, there were some nasty little problems I came across. My first guess would have been that a simple copy and paste cycle on the Apache configuration (taking into the account the IMHO much nicer organisation of the config files on Debian) would do it. However in some details the servers seem to have been compiled with different presets, so I could not get it run just like that. The most important thing was the order of two aliases that would allow me to have both http://cvs-server/viewcvs and http://cvs-server/viewcvs/ (see the trailing slash) be the same as http://cvs-server/cgi-bin/viewcvs.cgi. For some reason the alias definition for the URL including the slash had to be first, which on RedHat led to a recursive cycle that Firefox complained about. However, after some fiddling to get the stylesheets and logos right it finally worked like charm (and way faster, especially when generating a graphical view of the branches and tags of a file with cvsgraph).

After checking the file permissions on some special files (e. g. the CVSROOT directory and its contents) I configured xinetd to offer the CVS pserver. However just to make sure that nobody who had somehow learned the new servers address connected yet, I had it listen on a different port than 2401 by changing the /etc/services file. Connecting my (Windows) Eclipse to this new location was simple enough, however I was somewhat taken aback when I saw the file names of files with Umlauts showing as the UTF-8 sequences instead of the "real" characters. Changing the Server Encoding setting in the repository location's properties dialog solved the problem, however that now meant that all developers would need to update this setting, too. This was against the principle of not having anyone make any changes to work with the new machine, so the discussion about keeping the file names in ISO8859 format came up once again.

Making matters worse is what seems to be an Eclipse bug I filed unter Bugzilla #124499. Eclipse always changes the encoding of a configured repository location back to the default when I change any of the other settings in the dialog. All users of the new CVS server now have to keep this in mind to avoid problems with file name encodings. However finally we still stuck to UTF-8 and prepared an instruction video with Wink, a freeware equivalent of Camtasia or the like. We will still have to see how many problems we will run into...

Go!

Then, finally, I took the old server offline and rsynced the latest changes to the new machine. I must say I had not used rsync before, but it fully met my expectations. While the initial transfer of all data had taken about 75 minutes, updating the new machine with recent updates took only around 12 minutes, and a great share of that time was just for preparing the (rather long) file list. I then ran the convmv script on the data and had our network people update the DNS alias for the CVS server to point to the new machine.

Finally, after some last tests, I could send the mail I had prepared earlier and tell people that CVS was back online, and that they needed to change an Eclipse setting. From what I can tell up to now, there have not been any problems worth mentioning. Some people needed their passwords reset because they had forgotten them and somehow managed to have Eclipse also forget it, but that was about all.

What?s left

One of the open ends is the cvsdb feature of ViewCVS. It allows you to query a Bonsai compatible database of CVS checkins, logs, people etc. ViewCVS provides scripts to fill such a database from an existing repository and scripts to hook to CVS?s handlers. However what is missing is the code to also store CVS tags in the database. This however would be the most important feature for us, because right now we use cvs2cl to generate changelogs between release builds. Those are marked with CVS tags. Currently generating a changelog requires a full update of the corresponding branch and a quite long running collection of log messages and other information. At the moment it takes about 70-120 minutes to generate a changelog between to tags. I very much hope to dramatically accelerate this by using database queries instead of having to bother CVS every time for all files? histories. However I have not been able to get it working so far. On the ViewVC mailing list someone pointed out a few patches, but those aim at older version of ViewCVS. I still have not figured out how to make it work with the current revision. Maybe my Python is just not good enogh (yet J).

No comments: