[Systems] [Fwd: [Systems-logs] Cron <mirrorbrain at sunjammer> ulimit -c unlimited ; mirrorprobe -t 20]

[Peter Pöml]

Hi Bernie,

On Wed, Sep 15, 2010 at 05:15:14PM +0200, Bernie Innocenti wrote:
> > What could be the source of this problem?
> 
> Is this number of connections to be expected?
> 
>  sunjammer:~# ps afx | grep postgres | wc -l
>  63
>  sunjammer:~# ps afx | grep mirrorbrain | wc -l
>  54
> 
> "mirrorbrain -t 20" works well as root.


What are these processes? Could you share the output without "|wc -l"?

Something might be tuned not optimally. I don't know your setup well
enough, but many connections are not normally needed.


> > -------- Forwarded Message --------
[...]
> I'm getting one of these every 5 minutes. Disabling the cron job until
> we fix the issue.


What do you get -- the backtrace, shown below, that indicates
"connection limit exceeded"?


On Wed, Sep 15, 2010 at 05:10:15PM +0200, Bernie Innocenti wrote:
> What could be the source of this problem?
> 
> -------- Forwarded Message --------
> From: Cron Daemon <root at sugarlabs.org>
> To: root at sugarlabs.org
> Subject: [Systems-logs] Cron <mirrorbrain at sunjammer> ulimit -c unlimited ; mirrorprobe -t 20
[...]
> psycopg2.OperationalError: FATAL:  connection limit exceeded for non-superusers


The source is a usage of too many connections (probably about 100, if
you use the PostgreSQL defaults). I cannot tell you the reason yet. But
if you can provide details, I will understand it and can tell you more.

One thing I can already say. For all cron jobs which are run
periodically and might not complete within the interval they are
scheduled for, it is advantagous to run them under a locking wrapper.
This applies to the mirrorprobe and to the scan jobs, too.


So, instead of
    mirrorprobe -t 20
you run
    flock -n LOCK-mirrorprobe -c "mirrorprobe -t 20"
or
    withlock LOCK-mirrorprobe mirrorprobe -t 20

This prevents that these jobs ever could start to stack up, if they take
unexpectedly long.

flock should be installed with every Linux system.
withlock is available at http://code.google.com/p/withlock/

This is only possibly related to the issue you are seeing, but
nevertheless a good idea in general. Also for rsync jobs and stuff. It
rules out several possible complications.
(I realize that I should start prominently suggesting that in the
MirrorBrain documentation.)

Peter
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.sugarlabs.org/private/systems/attachments/20100915/520892ac/attachment.pgp