Wednesday, June 13, 2007

MySQL server_errno=1236 when disk full

Yesterday I was asked for help concerning a replication problem with one of our test systems. My colleague had already installed a fresh dump he had created with mysqldump ... --master-data. The dump looked ok and contained a master configuration statement:

...
CHANGE MASTER TO MASTER_LOG_FILE='master-bin.000127',MASTER_LOG_POS=4462223;
...

The slave was provided with the correct user, password and host name for this master. Nevertheless issuing a START SLAVE did not work. In the slave .err file we found this:

 
...
050327 20:54:51 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position (server_errno=1236)
050327 20:54:51 [ERROR] Got fatal error 1236: 'Client requested master to start replication from impossible position' from master when reading data from binary log
...

After some fiddling around and searching the net for solutions we had a look at the server as well. SHOW MASTER STATUS revealed seemingly correct values and SHOW MASTER LOGS also listed several binlog files, up to the one mentioned above (127).

Only by looking at the Windows console of the server we found out what was wrong. Upon logging in the operating system popped up a nice balloon message about "Low Disk Space On Drive E:". In fact low was slightly understated: The drive was full up to the last byte.

Interestingly MySQL happily moved the current position counter in the output of SHOW MASTER STATUS forward, even though the file could not be written. The manual chapter on How MySQL Handles a Full Disk claims that the server should wait until there is enough space available again, and I remember having seen some "Waiting for someone to free space" before. I will have to take a look into this again and file a support request if I cannot find an explanation.

2 comments:

Gary said...

Do you recall how you resolved this issue? I may be experiencing something similar. I'm getting:

070831 23:53:25 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236)
070831 23:53:25 [ERROR] Got fatal error 1236: 'Client requested master to start replication from impossible position' from master when reading data from binary log

070831 23:53:25 [Note] Slave I/O thread exiting, read up to log 'db100-bin.000056', position 879532399

Daniel Schneller said...

Sorry, but as it was not a critical system we did not have the time to look further into it. What we did was just purge some of the older binlogs and reduced the time they are kept on the master. The slave was re-initialized.