[Systems] Apache stalled on sunjammer today

[Bernie Innocenti]

On 18/07/15 20:01, Bernie Innocenti wrote:
> SysV ipc resources are global and don't get released when a process
> crashes. So they are inherently prone to leaks.
>
> Dunno why anyone sane would want to use them when there are better
> alternatives in Linux.

I figured out that the mechanism is configurable in Apache 2.2:

   http://httpd.apache.org/docs/2.2/mod/mpm_common.html#acceptmutex

And for Apache 2.4:

   http://httpd.apache.org/docs/2.4/mod/core.html#mutex

If it hangs again, it may be worth switching to flock or fcntl.


> On July 18, 2015 7:14:04 AM EDT, "Sam P." <sam at sam.today> wrote:
>
>     Hi All,
>
>     I was just loading up the wiki when sunjammer decided to hang on
>     me.  Couldn't load the ASLO either.
>
>     Exactly the same errors in the log as Sebastian reported.  `sudo
>     ipcs -s` has a similar output:
>
>     ------ Semaphore Arrays --------
>     key        semid      owner      perms      nsems
>     0x0052e2c1 414580736  postgres   600        17
>     0x002fa327 32769      root       666        2
>     0x0052e2c2 414613506  postgres   600        17
>     0x0052e2c3 414646275  postgres   600        17
>     0x0052e2c4 414679044  postgres   600        17
>     0x0052e2c5 414711813  postgres   600        17
>     0x0052e2c6 414744582  postgres   600        17
>     0x0052e2c7 414777351  postgres   600        17
>     0x0052e2c8 414810120  postgres   600        17
>     0x0052e2c9 414842889  postgres   600        17
>     0x0052e2ca 414875658  postgres   600        17
>     0x0052e2cb 414908427  postgres   600        17
>     0x0052e2cc 414941196  postgres   600        17
>     0x0052e2cd 414973965  postgres   600        17
>
>     Restarting apache2 and postgresql-8.4 services fixed the issue.
>
>     Still, it isn't good to get the same issue in like 6 hours.
>
>     Thanks,
>     Sam
>
>     On Sat, Jul 18, 2015 at 2:47 PM Bernie Innocenti <bernie at codewiz.org
>     <mailto:bernie at codewiz.org>> wrote:
>
>         Nice, let's kill buildbot then.
>
>         As for postgresql + mysql: unfortunately, mysql is used by a lot of
>         stuff and postgres is a hard requirement for mirrorbrain, which
>         needs a
>         a special geolocation index not supported by mysql.
>
>         On 17/07/15 23:44, Sam P. wrote:
>          > Hi Sebastian,
>          >
>          > We don't need buildbot running for sure - it hasn't had any
>         bots running
>          > for maybe even 12 months.
>          >
>          > We only use buildbot for rebuilding developer.slo - but I can
>         replace
>          > that with something like the help rebuilder [1] and move that
>         over to
>          > freedom.
>          >
>          > Thanks,
>          > Sam
>          >
>          > [1] https://github.com/sugarlabs-infra/help-rebuilder
>          >
>          > On Sat, Jul 18, 2015 at 1:37 PM Sebastian Silva
>          > <sebastian at fuentelibre.org <mailto:sebastian at fuentelibre.org>
>         <mailto:sebastian at fuentelibre.org
>         <mailto:sebastian at fuentelibre.org>>> wrote:
>          >
>          >     Apache error.log was reporting:
>          >     [Fri Jul 17 10:14:00 2015] [emerg] (22)Invalid argument:
>         couldn't
>          >     grab the accept mutex
>          >
>          >     And apache services were down. It was refusing to restart
>         itself.
>          >
>          >     I asked Bernie and he  told me "looks like we ran out of
>         semaphores..."
>          >
>          >     So i learned the *icps* command.
>          >
>          >     $ sudo ipcs -s
>          >
>          >     ------ Semaphore Arrays --------
>          >     key        semid      owner      perms      nsems
>          >     0x002fa327 32769      root       666        2
>          >     0x0052e2c1 348618763  postgres   600        17
>          >     0x0052e2c2 348651532  postgres   600        17
>          >     0x0052e2c3 348684301  postgres   600        17
>          >     0x0052e2c4 348717070  postgres   600        17
>          >     0x0052e2c5 348749839  postgres   600        17
>          >     0x0052e2c6 348782608  postgres   600        17
>          >     0x0052e2c7 348815377  postgres   600        17
>          >     0x0052e2c8 348848146  postgres   600        17
>          >     0x0052e2c9 348880915  postgres   600        17
>          >     0x0052e2ca 348913684  postgres   600        17
>          >     0x0052e2cb 348946453  postgres   600        17
>          >     0x0052e2cc 348979222  postgres   600        17
>          >     0x0052e2cd 349011991  postgres   600        17
>          >
>          >     So I restarted postgres database and after that, apache
>         was able to
>          >     restart.
>          >
>          >     Sunjammer is running a lot of stuff, I also notice
>         buildbot using a
>          >     lot of memory, is it under use?
>          >
>          >     Do we need 2 databases? mysql and postgres are running...
>          >
>          >     Perhaps it would be nice to purge sunjammer of unneeded
>         services?
>          >
>          >     Thanks Bernie for expert advice.
>          >
>          >     Regards,
>          >     Sebastian
>          >     _______________________________________________
>          >     Systems mailing list
>          > Systems at lists.sugarlabs.org
>         <mailto:Systems at lists.sugarlabs.org>
>         <mailto:Systems at lists.sugarlabs.org
>         <mailto:Systems at lists.sugarlabs.org>>
>          > http://lists.sugarlabs.org/listinfo/systems
>          >
>          >
>          >
>          > _______________________________________________
>          > Systems mailing list
>          > Systems at lists.sugarlabs.org <mailto:Systems at lists.sugarlabs.org>
>          > http://lists.sugarlabs.org/listinfo/systems
>          >
>
>
>         --
>            _ // Bernie Innocenti
>            \X/ http://codewiz.org
>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.


-- 
  _ // Bernie Innocenti
  \X/  http://codewiz.org