Tuesday, September 26, 2006

Windows: TCP Port conflicts above 1024

We have repeatedly run into problems with our JBoss application servers not being able to start after a Windows system reboot, because their configured network ports (e. g. 1099) had already been claimed by some other process.

It took quite a while to find the reason, because often just trying again several times without stopping or starting any other programs, it would suddenly succeed.

What turned out to be the reason is the dynamic port allocation for ports above 1024 (so called ephemeral ports). If any process requests a random port, it may get one of those you would like to use for your own applications.

On Windows 2000/2003 Server installations as well as on Windows XP Pro you can reserve port ranges (even if they only cover a single port) for your applications. Effectively they are not reserved for anything specific, but just excluded from the dynamic allocation. To do so, create or edit the following registry value (type REG_MULTI_SZ/Multi-String Value):

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ReservedPorts

In this value specify port ranges in the format xxxx-yyyy with xxxx and yyyy being the lowest and highest port of the range to be reserved. To reserve a single port, just use the same values for both (e. g. 1099).

What I find interesting is the suprisingly high frequency of this problem occuring. Even though I do not know for sure, I suspect Windows just starting random allocations at 1024 and counting upwards. That would at least to some degree explain why the problem occurred so often for 1099 but not for other, higher ports.

Responsible for assigning those random ports is the "RPC endpoint mapper", itself reachable via TCP port 135. There are several components that make use of this service, many of them included in Microsoft products. Most of these can be individually configured to request specific ranges or single ports to be used, but if you need specific ports to be reserved for your application under all circumstances, using the parameter above seems more sensible.

For further information see the original Microsoft Knowledgebase item #812873 which describes the ReservedPorts setting and several others, reachable from #832017 which explain in more detail which products can be individually configured.

Sunday, September 24, 2006

Video colors wrong on Ubuntu

Today I was suprised to see a video in Totem with very strange colors. It almost looks as if one color component is missing. I was alredy looking for problems with Totem on the net, when I noticed that on the second screen of my dual head setup everything was fine. Moving the window half-way between the two screens gives me one half with the right and one half with wrong colors.

It seems to affect different players (Xine and Mplayer, too) and all sorts of video material (tried DVD, mpg and xvid).

Though I do not know for sure, I suspect the latest upgrade package that also upgraded the nvidia driver. If I do not find another solution I will try downgrading to the previous version again.

Friday, September 22, 2006

MySQL Index Analyzer: 0.04 released

Over on the MySQL Index Analyzer site I have just released version 0.04 as a downloadable package.

This is the first GUI version that allows real analysis and has more features than the command line version.

A quick overview of what is new:

  • Swing GUI
  • Analysis features as on the command line
  • Copying of generated ALTER TABLE statements to the clipboard
  • Information on data and index size distribution
  • Rudimentary analysis of possible disk space savings

So go have a look :)

Thursday, September 21, 2006

The Register: Interesting Article on software standards

"The Register" has an interesting article called Dumb customers and dumber software standards that I recommend reading.

It is about interoperability between companies and their software products and that much innovation could be gained on all sides should they aim to work closer together. A core sentence is Customers should not be deterred from participating in standards groups for fear of giving away business plans or valuable competitive information, Snow added. [...] "We have stuff we do in the financial services business - I don't speak about that stuff at places like this or in standards organizations because there's a potential loss of competitive advantage. On the other hand, we have a lot of other stuff that isn't core to our business that we'd be better served to commodify [sic]," he said..

I definitely recommend having a look.

Tuesday, September 19, 2006

MySQL Index Analyzer: First GUI version

Over on my MySQL Index Analyzer blog I just posted two screenshots of the first working GUI drafts. If you are interested, you will have to check this version out from the SVN repository, I will prepare a new distribution package soon.

Monday, September 18, 2006

Ubuntu kernel update broke X11

Yesterday I installed an Ubuntu (Dapper Drake) kernel upgrade through the system upgrade mechanism. The new version is 2.6.15-27-686. Today, when I tried to boot, I just got a console logon, X11 refused to start because of problems with the configuration.

The following log was output to /var/log/Xorg.0.log, but from what I could see, the messages were contradictory. First the NVIDIA GPU is found and several messages from the module are displayed, but at the end of the log it says Failed to load the NVIDIA kernel module!.

(II) NVIDIA X Driver  1.0-8762  Mon May 15 13:09:21 PDT 2006
(II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
(II) Primary Device is: PCI 01:00:0
(--) Chipset NVIDIA GPU found
(--) Chipset NVIDIA GPU found
...
(II) Setting vga for screen 0.
(II) Setting vga for screen 1.
(**) NVIDIA(0): Depth 24, (--) framebuffer bpp 32
(==) NVIDIA(0): RGB weight 888
(==) NVIDIA(0): Default visual is TrueColor
(==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
(**) NVIDIA(0): Option "NoLogo" "0"
(**) NVIDIA(0): Option "AllowGLXWithComposite" "true"
(**) NVIDIA(0): Option "UseDisplayDevice" "CRT"
(**) NVIDIA(0): Option "UseEdidDpi" "FALSE"
(**) NVIDIA(0): Enabling RENDER acceleration
(II) NVIDIA(0): Support for GLX with the Damage and Composite X extensions is
(II) NVIDIA(0):     enabled.
(EE) NVIDIA(0): Failed to load the NVIDIA kernel module!
(EE) NVIDIA(0):  *** Aborting ***
(**) NVIDIA(1): Depth 24, (--) framebuffer bpp 32
(==) NVIDIA(1): RGB weight 888
(==) NVIDIA(1): Default visual is TrueColor
(==) NVIDIA(1): Using gamma correction (1.0, 1.0, 1.0)
(**) NVIDIA(1): Option "NoLogo" "0"
(**) NVIDIA(1): Option "AllowGLXWithComposite" "true"
(**) NVIDIA(1): Option "UseDisplayDevice" "DFP"
(**) NVIDIA(1): Option "UseEdidDpi" "FALSE"
(**) NVIDIA(1): Enabling RENDER acceleration
(EE) NVIDIA(1): Failed to load the NVIDIA kernel module!
(EE) NVIDIA(1):  *** Aborting ***
(II) UnloadModule: "nvidia"
(II) UnloadModule: "ramdac"
(II) UnloadModule: "fb"
(II) UnloadModule: "nvidia"
(II) UnloadModule: "ramdac"
(II) UnloadModule: "fb"
(EE) Screen(s) found, but none have a usable configuration.

Fatal server error:
no screens found

It took me some time to notice that the automatic upgrade had only downloaded the new kernel package, but left out the linux-restricted-modules package. After manually installing a version matching the new kernel, I got X11 back immediately:

$ sudo apt-get install linux-restricted-modules-2.6.15-27-686

I do not know if this is "normal" behaviour, but I would definitely like to not have to repair my system after a seemingly flawless upgrade....

Friday, September 15, 2006

Prevent Windows automatic updates reboot

Another one of those reminder posts: How to prevent the reboot triggered by the Windows automatic update feature.

  • Disable automatic updates completely. Not recommended.

  • Change the policy. Recommended.

    1. Start - Run - gpedit.msc

    2. Local Group Policy - Computer Configuration - Administrative Templates - Windows - Components - Windows Updates

      If you do not see the last element, use the context menu on Administrative Templates to add a new template. Pick the wuau.adm template and close the dialogs. Windows Updates should now show up.

    3. Activatge "No auto-restart for scheduled Automatic Updates Installations" and/or use a different warning interval.

On XP Home go to or create the following registry key/value HKEY_LOCAL_MACHINE \Software\Policies \Microsoft\Windows \WindowsUpdate\AU\NoAutoRebootWithLoggedOnUsers (Dword) and set it to either 0 (allow reboots) or 1 (disallow reboots).

Friday, September 08, 2006

Log Buffer #9: a Carnival of the Vanities for DBAs

As did last week's Craig Mullins of dbazine.com I wrote a short post about the LogBuffer recently when I found it in my Blog's referrer list and was promptly asked if I would like to compile one myself.

So here am I am, welcoming all of you to the the 9th issue of the Log Buffer, a Carnival of the Vanities for the DBA community. Once again you will find a plethora of links to all sorts of information on the one thing that keeps many of us both fed and sometimes close to blank despair.

Myself being some sort of a mixture between a software developer and database admin I have had a fair amount of time over the last years to get experience especially with MySQL. I did some work with the MS SQL server, too, so I am happy to have at least one item to cover for this faction:

Joseph Sack made sort of a mental note about SQL Server 2005 Instant File Initialization on his SQL Server blog. If you have been unnerved by waiting for your SQL Server to physically fill up its data files before allowing you to use them, go have a look. This can save a lot of time and may be especially interesting when expanding an existing installation.

While this is sadly the only item I have for the Microsoft fans, the MySQL people seem to be a lot chattier.

Peter Zaitsev is one of them. Formerly a MySQL AB employee he started his own consulting firm and was recently joined by his former - and new - colleague Vadim Tkachenko. During their time at MySQL AB they were already very active on their MySQL Performance Blog, but leaving the company does not seem to have affected their ambitions in any way.

There were three very interesting articles published during the last week alone, probably the most noteworthy, even if not completely new or even limited to databases, being Peter's warning that Even Minor Upgrades Are Not Always Safe. I am sure many of us (*raises hand*) have been through the trouble of upgrading to a bug-fix release of (insert favorite software package here) and only after having transferred it into production been confronted with its nasty little side effects. So from personal experience I can only support Peter's view on the matter.

Roland Bouman of MySQL AB published a detailed article on Refactoring MySQL Cursors, both on the O'Reilly Databases blog as well as his own blogspot blog. While the post itself centers on MySQL's new (since version 5) support of cursors, it surely holds valuable general information that can be applied to any other system as well. I have to admit that when reading it I felt reminded of several situation where I found myself or other people doing exactly what Roland describes. So if you have ever written a cursor, maybe you should have a look at this and after it decide again whether your choice was wise.

Jay Pipes has two interesting items, even more so as they point in somewhat different directions. The first one is his mentioning of an Interview with Rohit Nadhani from Webyog concerning the recent open-sourcing of their popular SQLyog MySQL GUI frontend which is now available in both a community and professional edition.

If you have some ressources to spare and are still looking for something useful to do with them, be sure to have a look at Jay's second article entitled Build Farm Scripts Available on MySQL Forge which is an announcement of the progress of the very interesting MySQL Build Farm initiative. They aim to provide a unified way of building MySQL on a variety of platforms and concentrate the results in order to improve quality.

Another post from Peter Zaitsev is about the Slow Query Log Analysis Tools. These include a patch to the MySQL server which increases the slow-query log threshold resolution from seconds to milliseconds. While most people will probably not want to use a patched server in production it might still be worth a look for benchmarking and performance tuning during development.

Also on the MySQL Performance Blog, Vladim Tkachenko provides a helpful hint towards the useful GROUP_CONCAT extension to GROUP BY. It can help prevent looping on the application side by combining values of a group into a single field value. Example snippets for PHP are provided, but can easily be adapted to suit other needs as well.

While we are at it, Mark Rittman is not only writing about GROUP BY...ROLLUP, but also about his plans for Comparing Oracle 10g Aggregation Techniques. He is going to present his findings about initial aggregation speed, ease of use and speed as well as space requirements at the UK Oracle User Group Conference in Birmingham this November. Since Mark says he is going to keep us informed about his progress, a regular look at his blog is definitely advisable.

This week Pete Finnigan gives a link to a Nice idea on audition using trace events in his Oracle security forums. The basic thought is to use Oracle error messages and attach events to them to improve Oracle security auditing, possibly finding some intruder poking around.

Although written using some Oracle wording, Dominic Giles post On the subject of I/O is really an excellent and mostly database-agnostic article. It covers the common problem of non-technical folks (and even some of the technical ones, too) underestimating the extreme importance of a well-balanced system design regarding the I/O subsystem, memory and CPU power.

Written by another Oracle expert - Chris Foot - but also interesting for all DBAs out there, regardless of the products they use, is The Non-Technical Art of Being a Successful DBA - Naming Conventions. It contains recommendations and best-practice advice to help prevent what Chris calls Moonlight Script Hunting, and outlines how to keep even large installations with lots of different databases running in a well-structured way.

Staying with Oracle for one more item: Eddi Awad has published some examples for the Cool Undocumented SQL Function: REVERSE which seems to have been available since Oracle 8i, if not officially documented. While its practical use may not be immediately evident, reading this article might get you some fresh ideas.

Two interesing posts come from Xapbr. The first started as he looking for some recommendations in a MySQL IRC channel and ended up as A review of the Glom graphical database frontend for PostgreSQL, somewhat similar to Microsoft Access or Filemaker Pro, but with a "real" database behind it.

Xapbr's second post is a notice that Version 0.1.149 of innotop has been released. While previous versions of this great tool for keeping an eye on the low level status of MySQL's InnoDB storage engine had some stability problems, those have been addressed now. I have already tried it and so far, I havent't encountered any of the problems I used to have.

More on PostgreSQL can be found among Greg's Postgres stuff who provides some tools that might come in handy at times for Finding Multi-Column Keys using the INFORMATION_SCHEMA. Apart from being outright useful they might also be seen as a starting point for more things to be done with database metadata.

Re-inventing Greg's method to prevent re-inventing nicely shows PostgreSQL's ability to use different sorts of languages to implement custom functionality. Robert Treat demonstrates how to implement an email validity check in pl/php using the Pear::Validate module, following the same feature, however implemented earlier by Greg in pl/perl with EMail::Valid.

To finish this weeks edition I'd like to point you to Sean McCown's What's a Tiny Mistake? on the Database Underground blog. While I primarily find it to be a nice obituary for "The Crocodile Hunter" Steve Irwin, Sean also draws a parallel to the IT business by giving out a warning to software developers and database folks reminding them of that small changes can have disastrous effects.