[Systems] [Fwd: [Systems-logs] Cron <mirrorbrain at sunjammer> ulimit -c unlimited ; mirrorprobe -t 20]

[Peter Pöml]

Hi Sascha + Bernie,

Le 16.09.2010 à 14:53, Sascha Silbe a écrit :
> All entries look like this (with "ps aux|grep mirrorbrain"):
> 
> postgres   606  0.0  0.0 106492 13460 ?        Ss   08:16   0:00 postgres: mirrorbrain mirrorbrain 127.0.0.1(50153) idle                                                                     
> 
> The PIDs are rapidly increasing (about 2Hz).
> There's nothing running as user mirrorbrain and no PostgreSQL processes
> running other than those for mirrorbrain and 6 special purpose ones
> (autovacuum, wal writer etc.).
> Nothing useful in /var/log/postgresql/postgresql-8.3-main.log and
> neither in /var/log/syslog.

MirrorBrain mostly runs inside Apache, thus we need to look into the MPM and its configuration, as well as the database connection pool configured and used.

Some periodically scheduled processes run as user mirrorbrain:
1) the mirrorprobe
2) the scanner ("mb scan")
3) possibly, the process that updates the hashes (metalink-hasher/mb makehashes), if you happen to use it.

From what you write, the mirrorprobe isn't causing the problem, it just trips over it because it doesn't get a connection when it wants to run. So, provided there are no stuck processes that hold connections open, Apache is the one to look at. We need to improve your Apache configuration to stop it using so many connections.

To double check, could you please show me the output of "netstat -tupan | grep 5432"?

In addition, it would be meaningful to see "ps aufxw | grep htt" (or maybe you'd have to grep for "apache", depending on your distro).

Then, the Apache server-status should provide further insight (load mod_status, switch ExtendedStatus on, and look at /server-status).

The, which MPM do you use (worker/prefork)? (You could use apache2 -V to find out.) And how is it configured? /etc/apache2/apache2.conf defaults -- or what did you change?

What is the mod_dbd configuration? Grep for DBDKeep. (If you don't have that, it would be a good starting point by the way.) I think that the Debian/Ubuntu install instructions at http://mirrorbrain.org/docs/installation/debian/ don't actually mention that part. 


> At least some cron daemons do not run the same job in parallel if it
> takes longer than its frequency. Not sure what Vixie cron does, I would
> have to dive into the source (the documentation doesn't specify
> behaviour in this case).
> However it isn't our problem: The cron job is deactivated and we still
> get the same error (at least in the logs - no idea who/where the client
> is).

I have used only Vixie cron, and it does no such checking. I don't know about the newer descendants, but would a bit surprised if they handle that -- after all, they can't know for certain what a process or process group is going to entertain and what the definition if "completed" is... but would be great if they do...

Peter