B3 had to be rebooted to remove out-of-control mysld process

[daggett]

Dear Excito/B3 users.

Firstly, here's some evidence of myslqld running out of control on my B3. This was taken from, by my count, more than 80 defunct mysql processes at the time of my observation.

Code: Select all

root@b3:/var/run/mysqld# ps auxww | grep mysql
root      1179  0.0  0.1   1872   760 ?        R    Feb27   0:44 /bin/sh /usr/bin/mysqld_safe
mysql    26758 20.0  3.0  97528 15768 ?        Rl   21:40   0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
root     26759  0.0  0.1   1632   552 ?        S    21:40   0:00 logger -t mysqld -p daemon.error
root     26760  0.0  0.0      0     0 ?        Z    21:40   0:00 [mysqld_safe] <defunct>
root     26762  0.0  0.0      0     0 ?        Z    21:40   0:00 

...

root     27127  0.0  0.0      0     0 ?        Z    21:40   0:00 [mysqld_safe] <defunct>
root     27129  0.0  0.0      0     0 ?        Z    21:40   0:00 [mysqld_safe] <defunct>

I couldn't kill the mysqld (or mysqld_safe) processes(are 'defunct' processes processes already killed?). I have never had this experience before. I saw no alternative but to reboot my B3 (using "shutdown -r now") and log back in (after which I found the out-of-control Mysqld had been terminated). My attempts to reststart myslqd failed. Not having a knowledge of how to diagnose the problem and fix it, I see no alternative now, but to uninstall Mysql and reinstall it. However, I need to know if Mysql is the standard mysql-server debian package. (Apache2 on the B3 is non-standard, BTW. It is apache2-mpm-prefork, I think, and not apache2. This can be a trap for the unwary and was for me.)

What preceded this was my attempt to set up on my B3 a testbed mirror Drupal content-managed web site. This entailed building the Mysql tables of the site I was mirroring, the size of the MySQL dump of tables was 761 Megabytes. I tried to build the MySQL database by reading in the Mysql dump of tables (that is, essentially a long series of Mysql "INSERT INTO TABLE ..." statements). The Mysql session that was reading the MySQLdump file froze, so, I terminated it with Control C. When I attempted to restart it I got messages which implied that that the files in /var/lib/mysql/... were too large. I found I could not kill off mysqld or whatever it was that was spawning of that huge number of defunct mysqld processes, so I rebooted my B3 (as I mentioned above).

I would be interested if anyone could offer advice on what to do.

Is it possible that what happened was just a one-off ocurrence of bad luck? Or is a mysqldump of size 761Mb just too large for the B3 or the particular version of Mysql that comes installed on the B3? If the latter, can I replace that version of Mysql with another? If so, how can I do it?

An alternative to running the mirror drupal/mysql server on my B3 is to run it on my desktop Ubuntu Linux system. which I would prefer not to do, but if I have no alternative, I will have to give that a try.

[Ubi]

FIrst of all you can *always* kill mysqld (or any process) with the -9 switch

Code: Select all

kill -9 mysqld

Changing to a new version of mysql is probably not going to help you. 761 is very large for a db, but should be within the reach of the B3 (it's certainly within the reach of a well-tuned mysql system)

mysql log files are a bit of a pain, but see if it gives you some info. As for reading the dump file my guess would be that it runs out of memory before anything else. CHeck this by uesing "top". ALso make sure this isn't a disk-full problem.

And finally, you can mirror a mysql by copying the raw data files to a new machine. There's no need for dump. Read the Replicaiton chapter from the mysql manual

[ryz]

You cant kill all process with

Code: Select all

kill -9

First of all you cant kill defunct processes since they are already dead. They just wait for the parent to read there exit status. You can neither kill processes that are in interruptible sleep. Processes in interruptible sleep usually waits for some kind of IO. When it comes to mysql I have no knowledge at all so
I am am sorry but I can not help you there.

[daggett]

Thanks, ryz,

(Thanks also, Ubi. Sorry I Did not notice your post earlier. Some links concerning Mysql replication are forge.mysql.org and at [dev.mysql.com/url])

But I am still not sure what alternative there was to rebooting my B3. I couldn't find the process ID of the process that was parent to the huge number of defunct processes. The file /var/run/mysqld/mysqld.pid did not exist. ([url=http://forum.excito.net/posting.php?mod ... =9&p=14145]Ubi's seems to offer a solution: "kill -9 mysqld")

B3 does not have the grunt to handle a mysql database with Mysqldump size of 700MB?

My guess of the cause of the problem is that my B3 simply does not have the grunt to handle a Mysql database with a size of 700 Mb (but Ubi said that B3 and a well-tuned mysqld server may be able to handle a database of that size if i replicate the mysql binary data rather than use the more verbose mysqldump of SQL statements). So, if I am to use the B3 for a test drupal site I will have to create a smaller version of the same database (and therefore it can't be a true mirror). (That may be a lot easier said than done.) Otherwise, I will have to use another Linux box (my desktop Machine?) as the testbed mirror.

[Ubi]

I don't know if the B3 can or cannot handle a 700MB mysql database. All I mentioned was that is was very large.

If you want to know the mother process of mysql, simply type

Code: Select all

ps auxw| grep mysql

and kill -9 the process with the lowest number. That should take along all child processes. Sometimes killall also works (killall mysqld).