Monday, December 29, 2008

XP SP3: STOP 0x0000007E (0xC000001D) (solved)

Update: If you arrived here, looking for a solution to a blue screen problem that started to occur on Feb 12th, 2010 - this is a different kind of problem. The solution to that can be found here: Heise Online (English). Basically, it is a side-effect of a Microsoft update (MS10-015) that triggers a flaw in an already present bootkit malware, which then leads to the blue screen of death problem.

From time to time (i. e. every workday ;-)) I still use Windows XP. Recently I kept getting complaints about the system partition being too full. I guess on all operating systems this can have all kinds of funny (or disastrous) effects, so I tried to clean up a little. Apart from the usual suspects (temporary internet files, some huge downloads on the desktop and the like) I noticed that there were literally dozens of $NtUninstall...$ folders containing the setup/uninstall files for a lot of updates that had accumulated over time.

I realized that those could be deleted, however I prefer having a "clean" system, so I went for Service Pack 3 first which should include all the individual patches anyway.

The SP3 network install package first complained about the RDP 6.0 MUI Pack (KB925877) being incompatible with SP3 and that I had to remove it first. Doing so via the Control Panel did not work for some reason: It disappeared from the list, however the SP3 setup still complained. I had to manually delete the registry key pointing to it (HLKM/Software/Microsoft/WindowsNT/CurrentVersion/Hotfix/KB925877) before the SP3 setup would start.

After a while it reported a successfully finished installation and asked for a reboot. So I did.... After the POST I got a quick glance at the XP boot screen which was very quickly replaced with a blue screen of death (official: "STOP error") with these details:

An attempt was made to execute to non-executable memory ... STOP: 0x000000FC (0xBA5038E8, 0xBE903964, 0xBA503848, 0x00000001) ...

I looked at my BIOS settings and found the "XD Technology" option set to "on". According to Intel's support glossary this is the "Execute Disable" feature, so I tried again having it switched off. This just replaced the error codes, the BSOD remained:

... STOP: 0x0000007E (0xC000001D, 0xF78DE8EA, 0xF78DE7F4, 0xF78DE4F0) ...

Neither Safe Mode nor regular boot succeeded. Using a different machine I came across some other people with the same problem. This forum entry gave me the necessary clue - probably one of the next things I would have tried anyway: It's a USB driver problem. I removed all USB peripherals - good thing to have a PS/2 keyboard lying around and that "legacy" port on your mainboard - and Windows immediately booted normally. I did not try to reattach the USB devices, but went right to the device manager and removed everything in the USB category. Even without another reboot I just let it then search modified hardware, having it re-detect all the stuff I had removed a minute before. One more reboot, with everything attached an everything is back to normal. Now I can finally delete the superfluous files for which I started this whole odyssey...

Wednesday, December 24, 2008

OS X Development

While waiting for my in-law's Ubuntu machine to download the update from 7.10 to 8.04 over a "DSL light" (384kbit/s) line I thought I might just write a bit about my first developer experiences on the Mac.

Back in August I switched from the PC to a 24" iMac. Now that I got a little more acquainted with it I thought I might just as well have a look at what the developer experience is like. People tend to say all sorts of things about Objective-C and that for a Java guy like me it would be a huge step backwards.

For that reason I was glad to read that Objective-C 2.0 comes with support for garbage collection, because manual memory management after years of Java would have been a real pain-point for me. I find it interesting to still see a discussion going on in the Mac developer world about whether or not GC is a good idea or not. I would have expected everyone to jump at it and leave the manual stuff behind with delight. Well, seems like a little bit of history repeating and all the same arguments against it being dug up again (most prominently of course it being slow and leading to several *seconds* of delay in the GUI all the time). But I am sure, this will pass.

Since then I have gotten used to the square bracket syntax oddities and XCode 3.1 is a whole lot better than the stuff I looked at several years ago when I worked for a company that targeted the Mac as well, but sadly enough had to shut its doors before anything meaningful could happen.

My self-education project is a sync tool that will allow me to keep my FRITZ!Box's internal phonebook aligned with the OS X AddressBook application. This is what I have come up with so far:

Unfortunately AVM - the makers of the FRITZ! line of products - do not offer an official API to upload the XML to the box. So I will have to figure out a way to either remote-control the web interface (yuck!) or somehow send it over via the telnet port (still yuck, but a little less... ;-)).

Apart from the unfamiliar language and library I am somewhat pissed about the level of comfort XCode offers. While it certainly is a good product and people who are knowledgable in it will disagree, it still seems to have a long way to go to catch up with either Netbeans or Eclipse in terms of productivity. After all refactoring is a little more to me than "Rename in scope" and "Extract method" - not even mentioning the differences in SCM integration.

But anyway, I am sure I will learn to live with and like it. I am currently on the look for a suitable book: I am proficient enough in Java and OO that I do not need an introduction to Objective-C, but I would definitely like a very thorough introduction to Cocoa, binding and the development model on the Mac in general. Of course it should include XCode 3 and be based on Objective-C 2.0 so I don't have to translate each and every step to the new version or even worse get used to deprecated ways of doing things. If you have any suggestions, please leave a note in the comments.

Sunday, December 07, 2008

US power sockets in the eyes of a European

I recently went on a trip to the U.S. For 10 days I stayed in different hotels, and in each of them I wondered about the state of the power outlets...

Armed with an adapter for my German/European plugs I tried to charge the camera's or phone's batteries. I don't know if it's just me, but I expect to just plug a device in and get a firm and secure connection to the power outlet. At least that's what I am used to at home.

However in all the hotel rooms the sockets seemed to have no "grip" on the plugs which then tended to either slip out completely or at least lose contact all the time. I first thought it might just be the adapter model I brought, but even the lamps and TVs in the rooms had flaky connections which would easily slip out even when just brushing the cable slightly. I believe this may have to do with the fact that in the US system the power outlets are just flat while in Europe they recess into the wall and provide a much firmer mechanical hold on the plugs. Schutzkontakte_steckdose.jpg The image is taken from German Wikipedia.

Maybe I am a little too "German" here, but I have to admit I felt really uncomfortable sleeping in a room with electrical connections like this. I have never seen such a thing here at home. If this is really normal, I wonder if there are more fires caused by problems with electricity in America than in Europe.

Wednesday, November 05, 2008

W-JAX 2008 OSGi Day

After the rather basic Power Workshop on Monday I attended the OSGi related sessions of the main conference yesterday. Though I was skeptical before - from my experience I often find conference speakers put too much focus on introductions and basic stuff at the beginning just to run short on time later - I was rather happy with the overall outcome.

The first session "OSGi Today and Tomorrow" was held by Peter Kriens of OSGi alliance fame and provided some insight into the history and ideas of the platform which he co-invented. The "tomorrow" portion was rather short - about 5 minutes out of 75 - I would have liked some more information here. However as it was the first session of the main conference I did not expect to dive into much detail here.

After that Matthias Lübken and Martin Lippert of akquinet it-agile GmbH talked about strategies and caveats when "porting" existing applications to use OSGi. I liked that talk even though I do not see why it was marked as "advanced" in the conference program.

The best session of the day (at least for me) was Kai Tödter and Gerd Wütherich's "Patterns and Best practices für dynamische OSGi Applikationen". As far as I am concerned this one would have been better rated as "advanced" material. Using a nice sample application that was used to demonstrate the use of the whiteboard and extender patterns (the underpinnings of declarative services) they provided some interesting ideas and insights. The sample application "Person Manager" is publicly available.

In a higher-level session about "Mobile OSGi for mobile enterprise applications" Joachim Ritter of ProSyst GmbH talked about the problems you face should you try to develop enterprise applications that are to run on mobile devices - especially phones - and a possible solution: their Mobile OSGi platform. It is designed to hide the enormous complexity of mobile operating systems, runtime environments, remote management and provisioning etc. If you are interested in that platform go to their web site and follow the link to Sprint (the U.S. telco provider) and their Titan platform. Though I did not check myself yet there should be a download for a Windows Mobile version including some Eclipse based developer tools - even with debugging and profiling options.

The last session of the day I had already heard at JAX08 in Wiesbaden. Martin Lippert went into the details of class loading in an OSGi environment and how it can both be your friend and enemy :-) This is really a low-level experts talk and even though I listened for the second time I still found it highly interesting and a good conclusion of my conference stay.

Monday, November 03, 2008

w-jax 2008 OSGi Power Workshop

Today I attended the Einf├╝hrung in die OSGi Service Platform (Introduction to the OSGi service platform) power workshop at w-jax 2008 in Munich. I had hoped to get some more detailed insight into using that stuff in practice. While the workshop was clearly not a waste of time I was a little disappointed about the lack of practical relevance. The workshop was packed with about 50 people - with only 20 registrations.

The first 90 minutes were spent installing Eclipse 3.4 and the Equinox SDK onto the participants' laptops. That time could have been put to better use had the session description not only told to bring a machine prepared with Java 5 but with the whole stack installed. So we had to hand around 5 CDs an some pen drives to get everyone up and ready.

After that and some introductory words about OSGi we went through increasingly "complex" examples and tutorials. Though I understand the idea to explain the inner workings of bundle lifecycle, Activators, package im-/exports and ServiceTrackers in great detail, the examples were really simple and did not display the use of the features in "real world" scenarios.

While I found my understanding of the OSGi services stuff confirmed, I think it was a little sad for those who had no idea about OSGi before, because I doubt the potential of the platform became clear and from what I can tell many people see the whole stack with a lot of scepticism.

Tomorrow I'll attend several sessions about OSGi, one of them about how to port existing applications to OSGi. I hope this will be a little more in-depth.

Thursday, September 04, 2008

Short note on code readability

Yesterday I came across a piece of code which at first had me a little startled. My first impression was, that the events would not be sent to any listener. Then I thought it would throw a NullPointerException. Only after that I realized that its fully functional, but a little awkward style-wise:

    public static void handle(String data) {
        DataListener dl;
        for (Iterator iterator = dataListeners.iterator(); iterator.hasNext(); dl.eventOccurred(data)) {
            dl = (DataListener);
            System.out.println("Sending data to " + dl.toString());

Did you get it? I have never before seen someone do the "actual work" in the third part of the for loop... Reminded me that even a short piece of code written in an unorthodox style can be somewhat challenging to grasp at first sight.

Monday, July 07, 2008

As promised...

... the next post comes from OS X :-)

Wednesday, July 02, 2008

Changing platforms

It's been a while since the last post. This is mostly due to me entering new territory in several ways. For one, I have been digging into JavaME development lately (platform change #1), building a mobile data entry and manipulation application that uses a an embedded database and talks to its server via Webservices, if connected. Otherwise data will be queued up locally and sent as soon as the network comes back.

The whole thing is build as an Eclipse embedded rich client platform app (eRCP) leaning on the included Equinox OSGi platform (#2). One of my favorites there is Declarative Services which helps cut down boilerplate code and at the same time encourages modular design. Really cool stuff, however somewhat difficult to get into if you are completely new.

I plan on posting some sort of tutorial on this. Of course there are already several out there, that's how I learned to do it (and still am). However some of the material seems a little outdated, so maybe someone will find it useful. I cannot tell yet when I'll get to it, but it's definitely coming.

Oh, and apart from that I have also decided to dump my current PC and go for a 24" iMac (#3). I have been thinking about this for several months now, the first time when I experienced a lot of freezes of my Ubuntu installation. However when it turned out not to be a hardware problem I stayed with the P4@3GHz. I even upgraded to Vista - which will now find its way into a boot camp partition I guess. But when I recently read an article in c't magazine where they had built a complete PC for €199 that was benchmarked to outperform any Pentium 4 machine that gave me the last push. Considering how long I kept this machine (at least 4 years now) I decided the higher price for the Apple machine would not make much of a difference in the end. (Even though I must say that Apple does not seem to be in their right mind when setting prices here in Germany. The same configuration - even when adding taxes to the U. S. price - is still more that 300€ more expensive (currently about $450)!)

According to the order status page it should arrive Friday or Monday. Can tell you, I am pretty excited about this, especially because as outside temperatures currently are in the 30°C (86°F) range my PC makes a lot of noise, which is really bugging me.

Alright. The next post will be from OS X I guess :-)

Monday, May 26, 2008

A little more on Windows Drag and Drop

A few days ago I complained about the silly drag and drop behavior of several ZIP tools, my favorite 7-Zip included. My post contained a remark that this is actually a Windows problem, but I did not have any further information on it other than a few links to the 7-Zip forum where people talked about this.

Thanks to an anonymous commenter on the earlier post I can now point to a good source of more information: The also fabulous WinSCP tool comes with a special drag and drop extension that works around the limitation of first unpacking to a temp file and then copying to the final destination, making unzip operations very slow. From the documentation page at you can learn this:

Here is short explanation: Windows drag&drop mechanics does not allow source application of drag&drop operation to find out easily, where the files are dropped. It is up to target application (Windows Explorer usually) to transfer files to destination. It is rather reasonable, because source application can hardly transfer files to all possible destinations. [...]

To allow direct drag&drop downloads, the shell extension was developed. It misuses Windows Explorer CopyHook’s. CopyHook is COM object (DLL library) that is called by Windows Explorer whenever directory (not file) is transferred within file system. When you drag anything from WinSCP, it creates empty dummy folder in temporary directory and pretends that you as user drag that directory. Once your drop it to Windows Explorer, it calls the CopyHook’s (including the WinSCP shell extension), telling it what and where was dragged. This way WinSCP knows the actual destination. It cancels the drag&drop operation, so the dummy directory is not copied by Windows Explorer and transfers your actual selection to now-known destination.

Go read the whole text in case you'd like to know the full story. Unfortunately the link to the MSDN library seems dead, so here is a (currently working) replacement:

BTW, why do they keep changing their URLs all the time? Even most private blogs jump through hoops to keep their permalinks valid when they change the site, why can't MS?

This is definitely a reason for me to use the installer package in the future instead of the standalone executable download I used to grab.

Wednesday, May 21, 2008

Vista still stands in its own way with USB drives

Some things will apparently never change. After I learned that the notorious temp-dir-copy-cycle most zip tools go through is probably a Windows feature, today I came across an issue I thought should finally have been addressed in "Windows 6" (Vista).

I had inserted a USB pen drive ("USB DISK") earlier and just wanted to eject it.


From my point of view this is a fairly typical scenario. When you insert the drive Windows will offer to open it in an explorer window. This is just what I did. I copied a file from there to the desktop and wanted to remove the device:


"Auswerfen" is the German word for "Eject". Guess what...


The same old story once again... This lengthy message tells me that some application is using the drive and that removing it might cause data-loss, kittens could die and the whole blah blah.

Of course I can click on "Fortsetzen" (Proceed), but this is not what I would expect of a 21st century user interface... The least thing would be a list of process names to get any clue of what program might still be accessing the drive and whether it is okay to close their handles or not...

Why on earth is it so difficult to just detect that no one else but the very same explorer window that offered me the context menu has a handle on that drive? Maybe I would even accept the warning if the pen drive's content was showing in a second window, but come on, how difficult can it be?

Tuesday, May 13, 2008

7-Zip, drag and drop and the temp-dir mystery

Today I downloaded the Eclipse Ganymede M6 Java EE package. It is a 178MB ZIP archive containing the IDE. Downloading took a few minutes and then I was ready to go. Well, almost. Of course the ZIP needed to be extracted first. I use 7-Zip as the archive tool of choice. It has proven to be reliable and fast.

However it shares one "habit" I have seen so far with every archive tool I tried: Wherever you want to extract the contents of the archive, it will first put all files into the %temp% directory - often on the system partition - and only then copy the unpacked content to the final location.

With the Ganymede package this means 217MB of data in 4180 files and folders are first read from the archive and stored in a folder. On my P4@3GHz this takes about 110s. After that a copy of that some data - almost 4200 files - takes another 90s. Apart from that, instead of 217MB I now need twice as much almost half a gigabyte to store both the temp copy and the final one. Why is that necessary?

There are some threads on the 7-Zip forum also discussing this. Apparently (however I could not find it in the linked MSDN articles) this has to do with how Windows handles drag and drop. And indeed, extracting via the "two-folder mode" (F9 in 7-Zip, I did not know that before) puts the archive contents directly to the destination, saving almost half the time.

If anyone more knowledgeable in Windows development could shed some light on this, I would surely be grateful. For now, knowing the workaround is fine, too.

Just one more thing: When I first hit F9 in a 7-Zip window apparently nothing happened. Turns out the split control was all the way to the left, almost invisible next to the window border, so if you don't see a second pane come up, have a close look there :)

Sunday, April 20, 2008

Stale /etc/group.lock & /etc/gshadow.lock on RH9

Last week for the first time in quite a long time an automated update process failed me. Even though it had been in use for several months already, and had seen numerous successful test runs, one machine was (almost) completely screwed up after the process. It may be interesting to know that all this is taking place on rather "antique" Red Hat 9 systems. I have not had the time to double check whether the following still holds true for more modern systems, any comments are appreciated.

The update logs contained a strange line:

useradd: unable to lock group file

Well, while the message itself is one of the more explicit ones, I could not understand what would be causing a lock on the group file. There is indeed a useradd command in the upgrade script, however it is the only one, and the update takes place right after a reboot. Hence no other process could possibly hold a lock on /etc/group.

Looking at the /etc directory I found these two files: /etc/group.lock and /etc/gshadow.lock.

I was quite surprised, when I could readily add a new group manually on the affected machine with the exact same command line from the update script.

After that a quick sweep over all production systems that are to be upgraded revealed, that each and every one of them contained the two files named above, usually with a creation date at about the time of their initial setups. So how could any of those ever have been upgraded successfully? Apparently my manual useradd succeeded as well...

Turns out that the .lock files are not just simple flags. At first I thought they were just by their presence acting as a signal to any command trying to change the contents of the group files. Apparently that's not the case. Their content is relevant as well: They both contain the process ID (PID) of the command that last acted upon the /etc/group and /etc/gshadow files. Unfortunately due to a bug described in a RedHat Errata Entry (on RH Desktop 3, not specifically RH9) those files do not get properly unlocked, when the useradd command exits (even normally).

So you get a stale lock file lying around. Next time useradd tries to modify the group files, it will not complain if there is no process currently running that has the PID denominated by the .lock file! This is really nasty, because it will work almost every time you try, but there is of course a realistic chance that some random process will just by chance get the same PID the first useradd once had and be running at the time you call it the second time. In that case, useradd is mistaken to believe that some other process is currently modifying the group or gshadow file and aborts with the error message above!

I have not found an updated shadow-utils RPM for RH9, so right now I need to scan all our scripts for group modifications and replace the groupadd calls to a function or script that will take care of removing the lock file once the command has finished.

groupadd ${*}
rm -f /etc/group.lock
rm -f /etc/gshadow.lock

Great that this hit us on a test system before rolling out into production.... Maybe sometimes Murphy is a nice guy after all ;-)

Saturday, April 12, 2008 Browser Switch

Just noticed that seems to deem Firefox users "not worthy" of their new site design ;-) (Firefox 3 Beta 5) (IE7)

Strangely enough show the same (blue) layout in both browsers.

Just wondering...

Monday, April 07, 2008

Locale.getDefault() FindBugs Detector

I just hacked together a little FindBugs plugin that will just detect calls to the static Locale.getDefault() method. This may be undesirable for application code to do - e.g. in cases where you have an application framework that provides a more flexible way of localization.

This is nothing I deem worthy of contributing to the FindBugs team, because technically there is nothing wrong with Locale.getDefault(). Nevertheless maybe you can use it, too.

Just drop it into the plugin directory of your FindBugs installation. The plugin contains the source code of its single class inside the jar. Because this page is not yet full, I post it here as well for a quick glance:

package de.shipdown.fb.detect;

import java.util.Locale;

import org.apache.bcel.classfile.Code;
import org.apache.bcel.generic.ObjectType;

import edu.umd.cs.findbugs.BugAccumulator;
import edu.umd.cs.findbugs.BugInstance;
import edu.umd.cs.findbugs.BugReporter;
import edu.umd.cs.findbugs.bcel.OpcodeStackDetector;
import edu.umd.cs.findbugs.classfile.ClassDescriptor;
import edu.umd.cs.findbugs.classfile.DescriptorFactory;

 * This is a FindBugs plugin designed to detect calls to the
 * {@link Locale#getDefault()} method. This may be undesirable in certain
 * contexts.
 * @author Daniel Schneller
public class LocaleGetDefaultDetector extends OpcodeStackDetector {

	/** The reporter to report to */
	final private BugReporter reporter;
	final private BugAccumulator bugAccumulator;

	 * {@link ObjectType} for {@link java.util.Locale}
	private final ClassDescriptor localeType = DescriptorFactory

	 * Creates a new instance of this Detector.
	 * @param aReporter
	 *            {@link BugReporter} instance to report found problems to.
	public LocaleGetDefaultDetector(BugReporter aReporter) {
		reporter = aReporter;
		bugAccumulator = new BugAccumulator(reporter);

	Subtypes2 subtypes2 = AnalysisContext.currentAnalysisContext()

	public void visit(Code obj) {

	 * Checks for method invocations ({@link org.apache.bcel.generic.INVOKESTATIC})
	 * on a {@link Locale#getDefault()}
	 * @param seen
	 *            An opcode to be analyzed
	 * @see edu.umd.cs.findbugs.visitclass.DismantleBytecode#sawOpcode(int)
	public void sawOpcode(int seen) {
		// we are only interested in static method calls
		if (seen != INVOKESTATIC) {

		try {
			String className = getClassConstantOperand();
			if (className.startsWith("[")) {
				// Ignore array classes
			ClassDescriptor cDesc = DescriptorFactory

			// if it is not compatible with Locale we are not
			// interested anymore
			if (!subtypes2.isSubtype(cDesc, localeType)) {

			String invokedName = getNameConstantOperand();
			if (!invokedName.equals("getDefault")) {

			bugAccumulator.accumulateBug(new BugInstance(this,
					this).addCalledMethod(this), this);

		} catch (ClassNotFoundException e) {


The file can be downloaded here: LocaleGetDefaultDetector.jar

Friday, March 28, 2008

Windows Live Writer

During the transition from JRoller to Blogger one of the factors that drove my decision were the available editors. Both offer some sort of WYSIWYG HTML editors to compose blog posts. However I was not too pleased with either one of them. Both tended to produce somewhat bloated markup with lots of divs and spans, combined with inline style attributes - in the end not really much better than <font> tags...

In the end I gave up on the "rich edit" modes and wrote the markup by hand, usually with Bluefish.

While I was generally fine with that I still kept looking for some decent blogging tool a little more sophisticated than either the browser's text-area control or Bluefish. I tried ScribeFire for a very short time, but was not pleased with it. Maybe it suits people better who usually blog about stuff they find while surfing the web, but for me it just did not work. Finally - I cannot really remember when or how I got to it - I stumbled across the Windows Live Writer.

It has a lot of nice features, one of them being the ability to download the layout from your actual blog (BTW I just tried this with my Blogger account, even though there are other types supported and I just assume they offer the same functionality). This allows you to write stuff with a very close approximation of what the final posting will look like.

You also get a "preview" mode in which you can see the full page layout complete with graphics and style-sheets. This is really nice to get a final look before publishing, because frankly the preview mode you get in Blogger's web interface plain sucks. JRoller at some point upgraded to a newer version of the Roller engine apparently and got that better from then on.


And apart from the plain GUI niceties, the HTML code that gets generated is really nice and clean. You may even choose between standard HTML mode (in which e. g. <p> tags are not closed) or XHTML compliant code (which I prefer).

Images you want to insert are automatically uploaded, so no hassle there, either. And you can put borders around it in different styles as well:

Windows Live Writer Screenshot

The one thing I missed though was the ability to insert custom snippets of HTML code, e. g. a <pre> tag with a specific class attribute. To do I had to switch from "rich editing" to the HTML mode and insert that manually. Today I found a nice template plugin (did I mention there is a plugin-architecture? ;-)) that allows me to do just that. It can do even more, because you can include snippets of C# code inside the templates. E. g. the following code was first written, then selected and finally the template with that code in it applied.

<pre class="prettyprint">
<%= _selection %>

I have come to like this tool so much that even when working on Ubuntu (which is most of the time) I fire up a VirtualBox'ed instance of it.

So in case you have not found your favorite editor yet, are always ready for something new and - that one might be a show-stopper ;-) - you are not completely anti-Microsoft, go have a look and try for yourself.

Thursday, March 27, 2008

Follow-up: Strange InnoDB Locking Behavior

On March 4th I wrote about an observation we made concerning InnoDB locking behavior in combination with foreign key constraints.

Since then we have been in contact with MySQL support and they in turn with the InnoDB developers. One of the conclusion I can already draw from the whole thing is that another level of indirection (InnoDB being a part of Oracle, i. e. an external resource to MySQL, too) does not tend to speed things up...

But be that as it is, here is a more thorough explanation we got about how the more aggressive locking comes to pass in case a foreign key constraint references a parent table's primary key. As the original response from the support staff spans several emails and includes some internal stuff I will paraphrase to protect the innocent (i. e. me ;-)).

First - just to remind myself - some notes on how the foreign key checks work. For any update you do to a child table, the DBMS needs to make sure that you cannot concurrently remove the parent table's row the FK refers to. Otherwise you could end up with a request to roll back the transaction that was intended to change a child row and be left without a suitable parent value. To do so, locks are placed on the index records being referenced by a child table's foreign key.

In the example transaction 1 locks a row in the parent table to update it. It does so by locking the primary key. It uses an X (exclusive) lock on the particular value of the PK for one row in this table. An exclusive lock blocks access to the entire row, because the primary key is a clustered index in InnoDB. That means the key itself is part of the record - in contrast to non-clustered indexes where a copy of index columns' values is placed in a location outside the actual records. So regardless of whether or not the key values themselves are changed, the record is locked completely.

Again: Because of InnoDB's structure all primary key locks are X locks, as the key itself is part of the row. In contrast to that an S (shared) lock only locks entries in a secondary index, which are stored outside the actual records.

Now, transaction 2 intends to update a record in a dependent table. It has to take a lock on the foreign key there whose reference key is the primary key in the parent table. However the exact row the child record refers to in the parent table has already been locked by transaction 1, hence operation 2 has to wait.

Transaction 3 needs to change a column in a row belonging to a child table, too. This column has a foreign key to the parent table, but not one whose parent is a primary key. It instead references a secondary key of the parent table. That means transaction 3 does not have to wait, because it does not require X lock on the entire row in the parent table. It only requires an S lock on the secondary key. As no other transaction has taken such a lock on that index entry, transaction 3 can proceed.

While all this explains perfectly why we experience a locking situation in one case, but not in the other, it still leaves the question open on how to go on from there.

Basically the one conclusion that can be drawn here already is this:

Either you get undesirable locks when foreign keys reference their parent tables' primary keys, our you have to define superfluous indexes that cost disk space and slow down write operations. In some cases you cannot even choose, because the optimizer might still decide to go for the primary key, even though you defined a secondary one, depending on the specific conditions for any given statement.

The latter part of that is especially true if you go for the - usually suggested - short primary keys in InnoDB tables, e. g. single numeric key columns. While those are generally a good idea especially for InnoDB they work against you in the scenario at hand.

Right now, the only suggestion we got from the MySQL folks is to rather go for less indexes, because the cost they introduce is usually significant in contrast to the locking problems. This is hard to quantify however and may depend heavily on your application's access patterns. Making matters still worse is InnoDB's current (5.0) limitation that it will lock all rows that are examined while executing any particular DML statement. Only in 5.1 only rows that get actually changed will be locked, alleviating the situation to some degree.

There might still be something coming here, however right now I do not think there is a big chance of this being changed any time soon. The one thing I will ask for however is to improve the documentation on foreign keys to have people know about this stuff in advance.

Monday, March 10, 2008

Enable Explorer Window Titles on Vista

Something struck me recently about the appearance of the Vista desktop. I did not consciously realize it at first. What's "wrong" with this picture:

Some Explorer Windows

If you do not see it, compare it with this one:

Some More Explorer Windows

See the difference? The second screenshot was taken with "AeroBar" loaded. The only thing it does is make use of the otherwise wasted space in the title bar by displaying the folder name. Does anyone know why Microsoft would not do this by default?

Sunday, March 09, 2008

iPhone SDK - Sweet with a smack of bitterness

When Apple released the iPhone SDK to finally allow 3rd party applications on the iPhone and the iPod Touch, I was immediately excited. Even though Apple claimed that web applications were a great way of developing software for their devices, I did never think so. It may be a feasible way for some simpler types of software, but not being able to leverage the full potential of a "fat client" (even on a device as thin as that ;-)) always struck me as a severe limitation.

However the sun does not shine so brightly after all. The Register's Bill Ray writes about his view on things in "Cometh the hour, cometh the iPhone SDK". I tend to largely agree with him. When I watched the "March 6 event" video I didn't even realize that the $99 fee for the development tools was supposed to be a yearly subscription, but got the impression it was simply the price for a license to use the tools in the SDK. This is still what I understand from the iPhone Developer Program website. But maybe its just hidden somewhere in the fine-print I did not care to look at.

However much more important to me is the fact that the Apple "App Store" will be the one and only way to distribute software to end users. This has huge implications, even though at first glance you might be tempted to follow Apple's portrayal of this being a good thing: a very easy way for software authors to get their stuff deployed to the largest number of customers possible. They underline this will be a free service for free applications. And maybe even the 30% of the price for non-free products is a fair deal, compared with what others claim. Nevertheless this makes each and every software developer - be it a company or an individual - dependent on Apple's good will.

Jobs already made clear that there will be restrictions on what kinds of software will be sold and which will not. Of course he presented examples that most people will tend to agree with: porn, illegal and malicious apps and those compromising your privacy. One might possibly argue that this may lead to a higher quality standard. But think about it a little longer: it also means that Apple takes the role of the "guardian of the public morals". This might even be seen as a good thing, but this also means, that they have full control over the software market. They can easily delay or even reject anything that is not to their liking, i. e. is a competition to their own products.

Think of a company that specializes in iPhone apps and becomes really successful. Maybe at some time in the future Apple decides that 30% of the revenue is not enough and wants to sell a similar product of their own. In a regular kind of software market, the customers still have the choice whether they use the one or the other vendor's products. But with the App Store Apple could just decide that the competing product cannot be sold though their channel any more, effectively killing the other company. Maybe it will need a little tweaking of the terms and conditions, but hey: what are you gonna do if you don't agree with them then? The only thing you can do is leave with your product, which is exactly what you were supposed to do in the first place. Now, that's what I call a monopoly.

I definitely like Apple's products, they are really great. Seeing what people were able to do with the beta-version of the development kit was really impressive. I expect a lot of great new stuff to come out in the near future. From what I see going on here people should consider if shouting "foul" at e. g. Microsoft is always a call in the right direction.

Nevertheless, maybe this is just the beginning of a new round of jail-breaking. In the past you couldn't run non-Apple software, so smart people found ways around that. Now you can have 3rd party software, but only through Apple. I guess someone will work around that, as well...

Saturday, March 08, 2008

iGoogle for iPhone

While iGoogle is nothing really new, I only recently (and by accident) noticed, that there now is an iPhone optimized version of it. While the regular Google home page for mobile devices has seen some improvements specific to the Apple mobile devices as well (tabs at the top of the screen), I really like the combined iGoogle and small display optimized combination iGoogle/i offers.

Unfortunately the latter one does not have the nice tabs to quickly switch between GMail, Calendar, Google Reader and the other apps they offer. This would be the last thing to make it close to perfect for me.

I wonder how this got past me without noticing it earlier...

For some screenshots see (iGoogle/i) or TechCrunch (Google/m). It seems you have to use an iPhone or an iPod to see the differences, the "regular mobile version" appears to be unchanged when opened with e. g. Internet Explorer.

Thursday, March 06, 2008

Monsters of the Programming World

Through a comment on my post about my latest encounter with unexpected/unexplained locking phenomena David Linisin's blog caught my attention. Looking through some older postings there I found a post called "The Cruel World Of Programming" which contains some hilarious drawings of "bug species" everyone knows. Go have a look, a hard copy of this is now pinned to the wall behind my desk :-)

Tuesday, March 04, 2008

Importing Audiobooks to iTunes on Windows

I may have found a new passion: Audiobooks. While for quite some time I was rather skeptical, even when my girlfriend gave me "The Hobbit" as a Christmas present a year ago I hardly tried it. I just could not imagine sitting around and listening to someone reading a story to me.

However back then I did not have an iPod (or any other portable music player for that matter), so now - listening to podcasts most of the time I drive, walk or do my household chores - I reconsidered when I got hold of a "Glenkill" set of CDs.

My first shot was to just import them into iTunes. However that way all 5 CDs with all their tracks showed up in my "Music" library section where I really did not like them. Even though you can change the files' properties to exclude them from shuffling this was not really acceptable for me.

A colleague told me that he had found "Doug's AppleScripts for iTunes" when he was trying to do the same thing for his children's audiobooks and they served him well. However there is one slight detail that makes them (at least for now) unsuitable for me. Let's just say there is a part of the site's name that does not lend itself to a Windows or Linux user...

There are some suggestions, but most of them involve re-importing the CDs as a single file per CD into iTunes - something I did not really want to do if not absolutely unavoidable, because I consider swapping media a rather boring task.

So I went and looked for something similar to the above mentioned Apple scripts for those being stuck with the Windows based version of iTunes. Eventually I found just the right tool for the job: Mp4Box or rather its GUI front-end YAMB.

Once I had downloaded and installed it I could join all the single m4a files I already had into a single large one. The GUI is a little rough around the edges (try and re-order the tracks to see what I mean), but that's alright. After some time (the progress indicator is not really helping) I ended up with a new MP4 file that contained the concatenated tracks I had fed into YAMB.

Importing it into iTunes once the file suffix had been changed to "m4b" (the file extension for audiobooks) automatically put it into the "Audiobooks" category. I have not found out yet, if I can define chapters or bookmarks - that (and syncing it to the iPod) will have to wait until tomorrow.

InnoDB Primary Key and Secondary Index Locking Strangeness

Recently we noticed a strange locking problem within our application. While the previous release worked flawlessly the current version produced a number of Lock Wait Timeouts.

The code itself did not change in the area we experienced the problems, however our persistence framework had been updated to a new release in conjunction with our app.

One notable difference was the schema generation strategy (the schema is generated based on an UML class model). In the previous version primary key columns were always created alphabetically. Because that is not necessarily a desirable thing it always created a secondary index, too, that contained the same columns in a user-specified order. In case you really wanted the alphabetical order you ended up with two identical indexes. Especially for InnoDB this is a very inefficient thing to do. (See my Index Analyzer tool for more info on why.)

Often we ended up with something like this:

CREATE TABLE `parent` (
    `A` char(3) NOT NULL default '',
    `B` char(2) NOT NULL default '',
    `C` decimal(12,2) default NULL,
    KEY `IDX_P` (B, A)

In this table the Primary Key and IDX_P are both referring to the same columns, just in a different order.

With the new persistence framework release those secondary indexes are not generated anymore. Instead, the primary key order is used as defined in the object model.

Although this is generally an improvement we appreciate a lot, it turns out this did have a negative side-effect of accidentally introducing a different locking behavior we would not have expected:

create database if not exists testcase;
use testcase;
drop table if exists child1;
drop table if exists child2;
drop table if exists parent;

CREATE TABLE `parent` (
    `A` char(3) NOT NULL default '',
    `B` char(2) NOT NULL default '',
    `C` decimal(12,2) default NULL,
    KEY `IDX_P` (B, A)

CREATE TABLE `child1` (
    `X_parent_A` char(3) default NULL,
    `X_parent_B` char(2) default NULL,
    `id` bigint(20) NOT NULL default '0',
    PRIMARY KEY (`id`),
    CONSTRAINT `child1_parent` 
       FOREIGN KEY (X_parent_A, X_parent_B) 

CREATE TABLE `child2` (
    `X_parent_A` char(3) default NULL,
    `X_parent_B` char(2) default NULL,
    `id` bigint(20) NOT NULL default '0',
    PRIMARY KEY (`id`),
    CONSTRAINT `child2_parent` 
       FOREIGN KEY (X_parent_B, X_parent_A) 

INSERT INTO parent (C,A,B) values (11.95, '111' , '11'); 

This sets up three tables and populates the "parent" table with a single row.

Now we go ahead and try some work on those tables in two separate transactions:

Transaction 1

Note that this transaction has not been committed yet. It makes a change to the single row we have in that table. However it does not change the primary key value!

Now in a second transaction try something with a child table:

lockwait2To me this does not really seem like a difficult task for the database to perform: Insert a new child row, referencing the parent table. Even though there is a concurrent update on that parent row in progress, it does not intend to change the primary key value which means the referential integrity is going to alright, no matter what transaction gets committed first.

In our app this is a very common pattern and we would definitely have noticed if that was something MySQL/InnoDB had problems with.

This is the relevant portion of the InnoDB status output on locks, when the UPDATE statement above has been issued in the first transaction:


As you can see, there is a lock on the primary key of that single row we are updating.

When we try to insert into the "child1" table, this is what we get regarding locks:


This notoriously hard to read output (see Xaprb's Maatkit) says that the INSERT INTO "child1" is waiting for a record lock on the master table to be granted. Because this lock is being held by the update transaction which has not been committed yet, the INSERT will eventually time out and report the error we have seen.

So far, this might not be obviously strange. But look at this now ("transaction 3"):


Compare transactions 2 and 3 carefully. You will see that they are identical in terms of the values. The only difference is that the former inserts into "child1" and the latter into "child2".

Here are the relevant parts from the DDL of tables "child1" and "child2":

CREATE TABLE `child1` ( ...
    CONSTRAINT `child1_parent` ...

CREATE TABLE `child2` ( ...
    CONSTRAINT `child2_parent` ...

As you can see the foreign key constraint defined in "child1" references the primary key columns in the parent table. "child2" uses the same columns, but in the order they are contained in the secondary "IDX_P" index.

According to the manual you need not reference the primary key in foreign key definitions, but any index that begins with the referenced columns in the right order will do. But obviously there is an important difference: Modifications to all child tables will block in case a referenced record's PK has been locked, even though its value is not being changed, whereas using a secondary index will happily allow you to run both statements concurrently without conflicts.

For completeness' sake we made some more tests, e. g. with a secondary index that's defined in the same order as the primary key. In that case you will get the locking in both transactions, because then InnoDB always uses the Primary Key and hence locks updates in both child tables. And of course you will get a locking situation on the child tables when you are going to modify the primary key value in the parent table.

Currently MySQL/InnoDB are looking into this, because from what I hear from their support staff, they find it curious as well. I'll follow up on this one for sure :)

Saturday, March 01, 2008

Vista (Home Premium)

Last weekend I decided to buy a license of Vista to replace the XP install I still kept for the occasional necessity. I went for the Home Premium version which I got from for €89.

As I usually use Ubuntu I decided to do a clean install to replace the XP partition completely. All documents are on the Linux disk anyway, so nothing to lose there.

The install worked flawlessly, even though I do not know how long it took, because I let it run over night after I had entered the product key. In the morning it was asking for a user name and allowed me to log in shortly afterwards.

At first glance I did not notice any striking differences compared to the RC1 build I had tried out some time ago. Nevertheless I quickly realized that the computer was beeping using the internal speaker. A look at the device manager (Ctrl-Pause still works) revealed missing drivers for my sound card.

Windows update pulled about 170MB and wanted a reboot. After that I had new graphics drivers (nvidia Geforce) and sound drivers (Creative Labs Soundblaster SB Live). While the graphics worked fine, the audio sounded somewhat garbled - just as if the volume was too high for the speakers. Only when reducing the volume to around 5% the sound became clear. Looking around the net I found out that Creative Labs has announced the end-of-life for its Audigy and Live series. This means that a perfectly fine sound card that has served its purpose well for is now utterly useless, because of missing drivers.

While I can understand that Creative may have new products to sell, I can tell you that this is not going to drive me into their arms again. For now I have settled with the onboard Intel sound chip that was disabled via BIOS setup up to now.

Apart form that driver issue there seems to be a little more work to do with the sound subsystem. When I played the "refurbished" Solitaire game and switched to a different window the game sound were gone until I closed the game completely and re-launched it...

The "new" start menu (is it still named like that, even though the "Start" caption is gone?) has a nice search feature that allows me to just enter some characters and see all matching entries. However I would have liked it even better had they not removed the old program group cascading menu structure. The way it is now almost forces me to switch between mouse and keyboard, while the old style was a little easier on mouse only usage.

So far I cannot tell I am terribly impressed - only the Aero GUI is nice to look at. So I guess I will continue to use Ubuntu as my main work OS and just go to Vista to test Windows specific stuff and maybe to play from time to time.

Wednesday, February 20, 2008

Blogger Backup

Tonight for the first time I realized that all my posts to JRoller and Blogger are all just present on their respective websites, but not as a copy on my local machine or some other backup. While I have full trust in Google's engineers to keep Blogger up and running as well as my data safe, it is never a bad idea to make a copy of your stuff.

So I just googled for "Blogger Backup" and found just that: A Blogger Backup tool. It is a .NET based tool that utilizes Google's C# data access libraries. Probably it would have worked with Mono, but as I had a VirtualBox'ed XP instance up anyway I used it there.

Once I had downloaded and installed the tool, it was just a matter of a few clicks to get all my posts saved in Atom/XML files locally. One thing has to be noted though: Make sure your blog feed is set to "Full posts", not just the summary. I first got this wrong, so all the backup files just contained at most the first paragraph.

Now all 99 posts I posted to my blog reside in a local disk folder which makes me feel a little better :)

As for JRoller I so far have only found some special template based backup ideas, however none look really too appealing. Maybe I will just use a website grabber to snapshot the whole thing as HTML.

Sunday, February 10, 2008

Back to text mode

Over the past couple of weeks my machine repeatedly just froze - mouse not moving, keyboard dead. I do not know if it was just X11 crashing, because I do not have any remote services active on the machine, so I could not log in remotely. Just as a precaution I did not boot into X11, but now I am in text mode (80x25), backing up my home folder to an external hard disk. After that I will go for a fresh install of Ubuntu 7.10. The current install has seen upgrades from 6.06 to 6.10 to 7.04 to 7.10, spiced and seasoned with some manual tweaks, especially when something did not work in one of the previous Ubuntu versions. So I cannot really tell if something that was need back then now tends to destabilize my machine. It almost feels a little like this "again Windows syndrome" - where people claim that Windows just "needs a reinstall once a year" to remain fast and stable. I am a little reluctant to admit this, so a part of me hopes this is really a hardware problem; that would mean of course I would have to make a decision on staying with the PC and Linux or buy a Mac for the first time in my life... As the backup takes some time, I thought I might just post this here. I am writing with the elinks browser in plain VGA text mode. I did not really remember what a PITA this extremely limited amount of screen real estate is. Usually I would have put in links to my earlier posts about the Toshiba external hard drive (which I finally switched to HDD only mode, hence the new full backup) and the Ubuntu upgrade articles. However I do not even have mouse support here, and copying over the post URLs by hand from another console is a little too tedious. Right now I feel like I am back to DOS - only back then the monitor was a little smaller. I can definitely tell I have no desire to go back to a real text mode interface ever again. Don't get me wrong - I really love command line interfaces. But as one of my professors once said: "Graphical user environments' sole purpose is to provide multiple terminal windows with good fonts and a nice background image.". While I would not go that far, right now I would really like to have just that :)

Friday, February 08, 2008

String.replaceAll() caveat

Sometimes I wonder how it is possible to write Java programs for as long as I have now and still stumble across little gotchas I did not know about.

When writing a class that is intended as a utility to write well-formed queries for our persistence framework, I was bitten by a nice little subtlety of the String class. Just a little background: In our application we use a persistence framework with a SQL-like (but still different) query language. Statements in that language are easier to read because they are closer to the business object model, but in the end they get translated into "real" SQL.

Unfortunately the query parser is based on simple query strings without support for something like prepared statements which makes it susceptible to injection attacks if you put user-entered values into a query directly. While there is a way to escape the input accordingly it is a little cumbersome and makes the code to assemble the query conditions into the final String hard to read.

I decided to build a little wrapper that would allow support for a sort of prepared statements by having people put placeholders into their queries and escape the values upon insertion.

The code for such a query might look like this:

StmtHelper h = new StmtHelper("SELECT firstname FROM customer WHERE name == #n#");
h.replace("n", txtfield.getText());
String queryString = h.assemble();
Inside the StmtHelper class' "replace" method I first escape the parameter value as needed and store it with its placeholder "n" in a map. Upon "assemble" I create a new String based on the template passed to the constructor where all placeholders are replaced with their escaped values.

This is the "assemble" method:

public String assemble() {
String result = template;
for (Map.Entry<String, String> entry : replacements.entrySet()) {
 result = result.replaceAll(entry.getKey(), entry.getValue());
return result;
At first glance this does what it is supposed to do. Unfortunately there is a bug in this code that will only show its ugly face with proper values to be filled in. For example this code will lead to a problem at runtime:
StmtHelper h = new StmtHelper("SELECT firstname FROM customer WHERE name == #n#");
h.replace("n", "O'Reilly");
String queryString = h.assemble();
This should lead to a query string as follows:

SELECT firstname FROM customer where name = 'O\'Reilly'

The single quote in the name must be escaped with a backslash. This is taken care of inside the "replace" method and works reliably - inspecting the replacement Map proves that. Nevertheless the final query string will be wrong: The single quote will not be escaped, effectively breaking the syntax for this particular query and name, but practically opening a gaping security hole for injection attacks:

SELECT firstname FROM customer where name = 'O'Reilly'

I was rather surprised when I came across this, because the replacements worked in all the test cases I had built into the JUnit tests. Unfortunately those did not contain a replacement value with a single quote, even though this is of course one of the first things someone would try in an attack... Looking into this more closely I learned something about String.replaceAll() I had never noticed before. This is from the Java 1.5 API documentation:

Replaces each substring of this string that matches the given regular expression with the given replacement. An invocation of this method of the form str.replaceAll(regex, repl) yields exactly the same result as the expression Pattern.compile(regex).matcher(str).replaceAll(repl)

IMHO this does not suggest anything special concerning the replacement, as long as the regular expression for the first parameter is valid. In my case this was just a simple name surrounded by hash characters, so no problems to expect here. However the backslash in front of O'Reilly's single quote disappears after the call to "replaceAll", leaving the quote unprotected in the resulting String.

Only looking at the Java 6 documentation makes it more clear:

An invocation of this method of the form str.replaceAll(regex, repl) yields exactly the same result as the expression Pattern.compile(regex).matcher(str).replaceAll(repl) Note that backslashes (\) and dollar signs ($) in the replacement string may cause the results to be different than if it were being treated as a literal replacement string; see Matcher.replaceAll. Use Matcher.quoteReplacement(java.lang.String) to suppress the special meaning of these characters, if desired.

That added paragraph suggests that more than one person must have fallen into that trap. While in the Java 5 version of the docs there was a mention of the result being exactly the same as if you used a Pattern/Matcher combination, at least to me it was not obviously hinting at a special behavior concerning the replacement string, but only the first parameter called "regex".

Strangely enough only after that experience I noticed that I must have been lucky all the time before, because I practically always used the "replaceAll()" method instead of the simple "replace()" I not use to replace the key/value pairs. As I do not really need the regular expression capabilities for the search term, this also saves some overhead.

Monday, February 04, 2008

Sun javac / Eclipse compiler difference

Our primary development environment is Eclipse, set to Java 5 compatibility. However during the nightly builds we use Sun's javac, currently version 1.5.0_12 I believe.

Today I noticed one of the builds had failed in this code (reduced to a small test case):

public interface Task<V> {}
public interface RunnableTask extends Runnable, Task {}
public class ProgressMonitorUtil {
public <V> V execute(final Task<V> aBackgroundTask) {
return null;
public void execute(final RunnableTask aBackgroundTask) {}
public class Test {
public static void main(String[] args) {
new ProgressMonitorUtil().execute(new RunnableTask() {
    public void run() {}

Put these classes into an Eclipse project, either with Java 5 or Java 6 compatibility settings. You will see no problems. However compiling this with Sun's javac (also either 5 or 6 will do) leads to this: reference to execute is ambiguous, both method <V>execute(Task<V>) in ProgressMonitorUtil and method execute(RunnableTask) in ProgressMonitorUtil match
new ProgressMonitorUtil().execute(new RunnableTask() {

Note: uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 error

Apparently method dispatching works differently in both compilers. In Eclipse the call goes to the second method (void execute...) which seems reasonable since RunnableTask is a more specific type.

However you can still convince Sun's compiler to compile this if you get rid of the "unchecked" warnings:

public interface RunnableTask extends Runnable, Task<Void> {}
public class ProgressMonitorUtil {
public <V> V execute(final Task<V> aBackgroundTask) {
return null;
public void execute(final RunnableTask aBackgroundTask) {}
public class Test {
public static void main(String[] args) {
    new ProgressMonitorUtil().execute(new RunnableTask() {
        public void run() {}

This compiles without warnings or errors on both compilers and leads to the same dispatching that Eclipse does. I guess I will post this as an Eclipse bug nevertheless, because responses there are usually way quicker than at Sun...

Edit: Making RunnableTask generic is not very useful in this context, but instead just declaring it as "extends Task<Void>" is sufficient as well. I updated the samples above.

Monday, January 21, 2008

SAXParseException: -1:-1: Premature End Of File - Misleading error

Today I had to look at a piece of code a colleague had written, using my XPathAccessor class. She used it in a servlet which gets XML formatted requests. As those are generated by an external 3rd party tool we agreed on some XML schema definitions. Everything they send us needs to conform to its corresponding schema, each reply we send gets validated against a different set.

In order to allow independent testing on either side, we provided a little test kit that allows testing our system without having to set up a servlet engine. Basically it just takes a file, reads it into a String and hands that to the handler.

First it gets parsed without validation. This is necessary to find out which type of request we were send (the address is the same for all of them). After the root element is known, it will be read again, this time using the right schema to verify the request.

Once that is done, some reply is put together and sent back to the client. So far, so good.

When I looked at the code I could see nothing wrong. Nevertheless each time a request was sent, we got a

2008-01-21 10:02:12,889 INFO  [STDOUT] [Fatal Error] :-1:-1: Premature end of file.
org.xml.sax.SAXParseException: Premature end of file.
    at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
    at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)

I had a look at the source of the XML input and first suspected a superfluous CR-LF after the closing element of the root tag. On the net some people claimed that this might be a cause for the error above. But removing that did not help.

This is the relevant code that handles the request:

public String readXmlFromStream(InputSource anInputSource) {
    String tResult = null;
    try {
        XPathAccessor reader = new XPathAccessor(anInputSource);
        String type = reader.xp("rootElem", "reqType");
        if (type.startsWith("K")) {
            Schema schemaK = XElement.getSchema(this.getClass()
            XPathAccessor validatingReader = new XPathAccessor(anInputSource, schemaK);

The last line throws the "Premature end of file" SAXParseException. The constructors of XPathAccessor look like this:

public XPathAccessor(InputSource aSource) throws SAXException, IOException, ParserConfigurationException {
    this(aSource, null);

public XPathAccessor(InputSource aSource, Schema aSchema) 
    throws SAXException, IOException, ParserConfigurationException {

    Validator validator = null;
    builder = factory.newDocumentBuilder();
    document = builder.parse(aSource);
    if (aSchema != null) {
        validator = aSchema.newValidator();
        validator.validate(new DOMSource(document));

Curiously in case of the Servlet no files are involved at all. Everything is in memory, so "Premature end of file" is not too helpful anyway. The solution to this mess can be found - sometimes it turns out to be helpful - in the API documentation for the InputSource:

An InputSource object belongs to the application: the SAX parser shall never modify it in any way (it may modify a copy if necessary). However, standard processing of both byte and character streams is to close them on as part of end-of-parse cleanup, so applications should not attempt to re-use such streams after they have been handed to a parser.

This last sentence is the clue: Because the InputSource has been used to find out the type of request, it cannot be used again for the validating XPathAccessor. In that light the error message at least makes a little sense: The underlying stream has been read to its end and been closed, so one might call that the "end of file"; and because the (2nd) XPathAccessor has just tried to read it from the start, "premature" might be a valid qualifier...

Knowing that also explained why the test suite worked fine; it read the XML contents into a String, which another set of overloaded constructors for XPathAccessor can accept. Of course strings can be read as often as you like, so no problems there.

As the docs do not give an immediate hint, I hope someone finds this post to save him/her some time.

Wednesday, January 16, 2008

Can't start server: Bind on TCP/IP port: No such file or directory (some progress)

Back in October last year I wrote about a peculiar MySQL error message:

Can't start server: Bind on TCP/IP port: No such file or directory

This error only seems to occur on Windows (Server 2003 in this case). While the first part is clear - a port could not be bound, because it was already occupied by another process - the second part does not make any sense.

We got in contact about this with MySQL support recently, because we came across the problem again on a machine that had received the "treatment" described in the earlier article which did not seem to work. Turned out that server had not been rebooted ever since, so the registry change was never activated.

However as a nice side-effect we now know (almost for sure) what that strange "No such file or directory" is all about: see MySQL Bug #33137. It has been categorized as a feature request - I doubt it will get any serious attention soon. So I will just summarize here to have it all in one place.

Apparently when the bind() call that is issued on MySQL server startup fails, on Linux and other platforms the cause for that failure can be queried from the error number (errno) return value. This does not seem to be the case on Windows. According to Axel Schwenke of MySQL support's research this is well documented on the MSDN reference of the Winsock bind function. According to that errno is not set to any meaningful value as a result of the bind() call. Instead you are advised to use WSAGetLastError() to find out if your call succeeded or what went wrong.

MySQL currently does not follow that advice, so the error displayed (errno=ENOENT=-1) does not have anything to do with the bind call at all. As a matter of fact there is currently no way to tell what really happened when that call failed. The only "real world" reason that comes to mind is in fact - as mentioned above - another process already occupying that port.

Be sure, should that feature request ever be implemented, you will find the (final) post about it here :)

Tuesday, January 08, 2008

Preload-mania killing aging machines

Over the holidays a friend of mine asked me to have a look at his machine, because it was extremely slow and barely responding at all anymore. When he tried to print a few documents it took more than 10 minutes for the first page to come out of the printer after the job had been started.

First I suspected a hardware problem, but when I booted it up I quickly realized that this was a bad case of "freeware congestion" combined with "pre-load-mania". When I had last set up the machine I had just put F-Secure Anti-virus, the necessary hardware drivers, Word and Firefox on the machine.

Hardware-wise a Celeron 2,4GHz with 512MB RAM is not exactly a high-end machine, but for web surfing, some emails and the occasional letter it should be very sufficient.

Now, about a year and a half later the login screen was followed by a desktop building up icon-wise and a colorful ICQ login even before the anti-virus software's splash screen. After that the hard drive kept working furiously for several minutes, trying hard to bring in about 15 icons to trickle into the system tray.

Most of those were related to some clever tool, e. g. a weather monitor, a world clock, a picture-of-the-day screen saver control panel, graphics- and soundcard driver helpers, and various other utilities, including some Yahoo Widgets.

Of course most of these had registered themselves as an auto run, some even with a background service. Once all those programs were started the task manager showed around 540MB of committed memory - more than the whole physical memory available and without having opened only one "useful" foreground process.

Using Sysinternals' AutoRuns tool I had a look at all the different places that can be used for running software on logon or boot, and apart from all the (presumably) tiny gadgets and widgets I also found a lot of the ubiquitous pre-loading parts of all sorts of common software: Adobes PDF Reader, Microsoft Office, something from iTunes and several others.

This is something I have always hated, and the older your machine gets, the worse it becomes: Automatic updates to the latest and shiniest new version of any given piece of software have become absolutely commonplace. While this takes the burden off the user to keep up-to-date and get patches for security vulnerabilities is has a serious downside, too. With every new version software tends to become bigger and more bloated. Moreover every vendor seems to believe that the primary and sole purpose of any machine will be to run their and only their software. Given that - completely wrong - assumption it must seem all natural to them to pre-load seemingly all of their software on boot, sitting ready in the background, just waiting for you to click the icon to issue the final call to "MainWindow.setVisible(true)" and be up almost instantaneously.

While this might work if you really use only that one program all the time and have a sufficiently large amount of RAM in your machine, this might actually work out. But this is complete unrealistic. Nobody in their right minds would boot their machine in the morning and manually launch Acrobat Reader, all MS office apps and every application they might possible use that day just to have them ready. It is immediately apparent to even the novice user that this is probably not making the machine more responsive. 

But this is - almost - exactly what happens with all the auto run entries: You just don't see them on the screen immediately. So one thing I always do after installing any software is double-check whether it just registered some sort of auto run and if so remove it.

Along that road I tend to replace more and more of the "big" products with leaner and slicker alternatives. E. g. instead of the Adobe Reader I nowadays usually install FoxIt Reader. Instead of Paintshop Pro (nowadays owned by Corel) I use Paint.NET and IrfanView etc. etc.

This really helps a lot - try for yourself, especially the difference between Adobe and FoxIt Reader is impressive.

Back to my friends machine: For about half an hour I tried to tidy things up, however the sheer amount of auto run tools, services etc. was too overwhelming. So I decided to do a fresh install. Not wanting to backup all user data (around 70GB) to DVDs and not having an external drive at hand I could not reformat the drive during the installation.

That meant that doing a new install in parallel to the existing one would reuse the existing "Program Files" and "Documents and Settings" folders - something I did not want in order to keep the new setup as clean as possible.

Because you cannot rename those folders from a running Windows I booted from a current Knoppix CD and mounted the NTFS partition. From here I just changed the folder names to "DocsAndSettings.old", "Programs.old", and "Windows.old".

Once I had done that I reinstalled XP from the CD I had prepared for the previous install already with XP ISO Builder. The allowed me to just leave the machine working, because it would automatically configure a user account, install all hardware drivers - even those not part of the default Windows CD-ROM - and set up the network to use DHCP. After about an hour I came back and inserted the c't offline update disk to bring Windows up to date.

Once all necessary applications had been reinstalled - which - given that you have all setup disks available and don't have to go download anything - should be done with the anti-virus program being last to save some more time - I just moved the "My Documents" folder back in place.

Boot time from POST to desktop is now around 35 seconds with everything ready to go after about 1:15 minutes. Memory load is around 310MB with a significant portion being taken by the inevitable anti-virus/anti-spyware software. However launching a browser or a home-banking app is now a matter of seconds instead of minutes, and printing a letter also no longer requires you to take a day off from work.

When I presented the results my friend had a hard time believing that I did nothing but completely dispose of all the gimmicks he had found useful and funny over time. We'll see how long he can resist this time.

Wednesday, January 02, 2008

Switch/Case and Autoboxing/Autounboxing

Because I have seen people trying this several times in the last couple weeks I decided I might just as well write a post about it: A seemingly common misconception of Java 5's auto(un)boxing.

While it has been a while now since we moved to Java 5 only now are people slowly getting familiar with the new syntax features. Most of them learned Java beginning with 1.4 and have a history in the DBase or FoxPro world. So object oriented programming and Java as one implementation of it are understood, however maybe not as deeply as you would expect. Some are especially impressed by the ease of use autoboxing and -unboxing bring to the wrapper classes for primitives. I also find that feature quite useful, because objects falling out of the persistence framework have full-blown object types for boolean values or numbers. This makes it rather cumbersome to work with them. Autounboxing helps a lot there:

if (theBusinessObject.isCompleted().booleanValue() && theBusinessObject.getNumber().intValue() > 5) {
if (theBusinessObject.isCompleted() && theBusinessObject.getNumber() > 5) {

Undeniably the second example is much easier to read. It becomes even more obvious once you start doing calculations based on wrapped primitives. Nevertheless problems may arise if you do not know what this new syntax will do under the covers. In the above case it is quite clear that the compiler will just put the calls to ".booleanValue()" and ".intValue()" into the bytecode on your behalf.

Consider this example:

public class Box {

 private static final Integer one = 1;
 private static final Integer two = 2;
 private static final Integer three = 3;

 public static void main(String[] args) {
  int myInt = three;
  switch (myInt) {
   case 1:
   case 2:
   case 3:

Thanks to autoboxing the reference type variables "one", "two" and "three" can be assigned using a primitive int on the right hand side of the "=" sign. And because of the autounboxing of "three" in the first line of "main()" it can be assigned to "myInt". After that you find just a regular switch/case construct on that primitive int.

Decompiling this using e. g. javap reveals the "magic" behind this:

D:\temp>c:\jdk1.5.0_12\bin\javap -c Box
Compiled from ""
public class Box extends java.lang.Object{
public static void main(java.lang.String[]);
   0:   getstatic       #2; //Field three:Ljava/lang/Integer;
   3:   invokevirtual   #3; //Method java/lang/Integer.intValue:()I

static {};
   0:   iconst_1
   1:   invokestatic    #10; //Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
   4:   putstatic       #11; //Field one:Ljava/lang/Integer;
   7:   iconst_2
   8:   invokestatic    #10; //Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
   11:  putstatic       #12; //Field two:Ljava/lang/Integer;
   14:  iconst_3
   15:  invokestatic    #10; //Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
   18:  putstatic       #2; //Field three:Ljava/lang/Integer;
   21:  return

At index 3 in "main()" you can see the automatically inserted call to java.lang.Integer.intValue(). This is the autounboxing. In the static initializer it goes the other way round: The compiler inserts java.lang.Integer.valueOf(int) at indexes 1, 8, and 15. Here the autoboxing takes place.

So far so easy. Now look at this:

public class Box {

 private static final Integer one = 1;
 private static final Integer two = 2;
 private static final Integer three = 3;

 public static void main(String[] args) {
  int myInt = three;
  switch (myInt) {
   case one:
   case two:
   case three:


Trying to compile this will fail:

D:\temp>c:\jdk1.5.0_12\bin\javac constant expression required
                        case one:
                             ^ constant expression required
                        case two:
                             ^ constant expression required
                        case three:
3 errors

I have seen this pattern numerous times, and whenever someone comes across it they seem to wonder what the difference is compared to the first example and why they get a compile error. They expect unboxing to happen at each "case". However they do not realize that this is not the same as putting the primitive value there, but is resolved to a method call under the covers - which of course is illegal in that context.

I have found it helpful to show people the bytecode output that gets generated in the first case. As a side effect they also usually learn for the first time about the existence of decompilers :)

One last piece of advice: Eclipse has a feature to specify a different syntax coloring for places where autoboxing and -unboxing occur. I recommend defining a clearly recognizable format, e. g. I use underlined text in a dark red color. I find it rather helpful to remind me that sometimes in such situations a null-check is a good idea - after all reference types might be null opposed to primitive values.