Saturday, April 25, 2009

CVS Performance: I/O Bottleneck because of locks

Today we found the reason for our sometimes abysmal CVS performance, even though the repository is located on a fast SAN: Locks were written to the local disks rather than the SAN.

For years we have been using CVS as our internal version control system. The repository has grown to about 20GB and covers 6 years of code and resources in several hundred thousand files and their history revisions.

Some time ago we migrated the repository data from an internal disk subsystem to a SAN based on 15k hard drives. The server is a Dual Xeon HP box with 4GB of RAM serving about 50 users.

Even though we thought we had a decently fast setup here, sometimes – especially when more than 10 or 15 people started to checkout or synchronize their Eclipse workspaces – performance would start to really degrade. In some cases working on branches only was possible after increasing the client timeouts to 3 minutes!

Our datacenter administrators provided us with I/O statistics in which we recently realized (don’t ask why they did not tell us about this earlier when we complained about bad I/O rates) that while the SAN interface was never really getting busy, local disks where 100% busy most of the time, especially during times when we had the most complaints from users.

Turns out that in the early days when we set up the repository configuration and obviously did not know enough about it, we configured the CVS LockDir variable in CVSROOT/config to /var/lock/cvs. This had never been changed, not even when we moved machines and put the data to the SAN. In effect each and every read or write operation on the repository caused locks to be created and deleted on the local hard drive of the server – completely breaking down the supposedly superb I/O performance.

As this is Friday afternoon we changed the configuration to place the locks on the SAN as well but could not really see the effect, because most people had already left for the weekend. We will certainly have a close look at the system on Monday to see if performance gets better. Probably we’ll set up a tmpfs ram disk to hold the locks if we see any improvements.

Up to this point I never even knew for sure how and when CVS put locks. I was under the impression that they were only required for commit or tagging operations. Because of that (false) assumption I would never have thought we could get a problem with local I/O, but as we have very complex directory structures lots and lots of per-directory read locks could of course pose a problem for a rather slow local disk.

For more information on CVS locks (not related to RCS locks) see the CVS manual.

Technorati Tags: ,,

Tuesday, April 14, 2009

Folder Action: Strip Subversion .svn metadata

I regularly need to send a zipped copy of a folder that is a local Subversion working copy. Every time I create the archive I end up with a file that’s larger than needed, because it contains the hidden Subversion metadata “.svn” subfolders.

Here is quickly hacked together AppleScript than delegates the actual work to the Unix “find” command, but can be attached as a Folder Action. I now have folder on my Desktop called “Strip SVN Metadata” and I just Option-Drag anything from a Subversion working copy there (making a copy, not moving) and a few seconds later I can just zip it from there without the .svn folders.

on adding folder items to this_folder after receiving added_items
    tell application "Finder"
        set fold_name to the name of this_folder
        try
            repeat with i from 1 to number of items in added_items
                set new_item to item i of added_items
                set the item_path to the quoted form of the POSIX path of new_item
                do shell script ("/usr/bin/find " & item_path & " -name '.svn' -type d -exec rm -rf {}")
            end repeat
        end try
    end tell
end adding folder items to

Sunday, April 12, 2009

Solved problem: Access Mac OS X SMB Shares from Vista – no more System Error 1326

Today I tried to map a network drive to a folder shared via SMB from Mac OS X. I enabled sharing in System Preferences and set up the user account appropriately. But whenever trying to connect from Vista, I ended up with “Systemfehler 1326” (“System error 1326 has occurred. Logon failure: unknown user name or bad password”) complaining about invalid username or password.

First I suspected a problem with my longish password that contains special characters, but that was not it. Turns out it is a compatibility problem/feature between the Samba configuration in OS X (the component responsible for sharing folders via the SMB protocol) and Vista’s default security settings.

First a solution for the impatient:

On Vista launch regedit.exe and navigate to “HKLM/SYSTEM/CurrentControlSet/Control/Lsa”. Check the value of “LmCompatibilityLevel” and set it to 1 – it defaults to 3.

For a list of settings for this key, see Microsoft Knowledge Base Entry 239869

On my system I did not have to reboot, I could connect to the Mac share immediately.

LM Compatibility Level 3 means the client will only try NTLMv2 authentication. This will not work against OS X in the default configuration, which only offer NTLMv1. By setting this to 1 you tell Vista to use v2 if the server supports it, but fall back to v1 if not.

While this is a quick and rather simple fix, it degrades security. By default Vista only connects to SMB servers that support the NTLMv2 authentication mechanism, because it is superior to the older variant from a cryptographic point of view. See http://davenport.sourceforge.net/ntlm.html and the Wikipedia entry on NTLM for more details.

In general you should prefer increasing security instead of loosening restrictions. To do so, you should configure XP and Windows 2000 to the same level 3 setting as Vista (the registry key is the same) and also set up Mac OS X to support NTLMv2.

Edit /var/db/smb.conf (using sudo vim) and make sure, the following two lines are present:

ntlm auth = no
lanman auth = no

If not, add or edit them to appear like this. Do not change anything else in that file!

When you are done, relaunch the Samba daemons:

sudo launchctl stop org.samba.smbd
sudo launchctl stop org.samba.nmbd
sudo launchctl start org.samba.nmbd
sudo launchctl start org.samba.smbd

From now on, Mac OS will only accept NTLMv2 connections, matching the higher security standards and refuse v1 clients – so make sure, you configure all your XPs accordingly.