[Systems] Error on trac reading a bug #4863

[Samuel Cantero]

Hi All,

On Tue, Jul 28, 2015 at 6:52 AM, Sam P. <sam at sam.today> wrote:

> Hi Bernie,
>
> On Tue, Jul 28, 2015 at 2:12 PM Bernie Innocenti <bernie at codewiz.org>
> wrote:
>
>> 1. We need monitoring for containers health with notifications when
>> parameters go off thresholds. Similar to what Munin does.
>>
>
> I'm not sure how that would help us.  A container running out of resources
> doesn't crash - parsoid runs almost using all the allocated memory and
> parsoid+trac+others hit their cpu quotas sometimes and just keep running.
>

A container running out of memory dies. By default, the memory limit value
for the container is set to twice as much as the memory parameter we
specify while starting a container. The limit value is the sum of memory
and swap.

We do not have swap accounting enabled yet on freedom, but for sure we are
going to have it at the next reboot.

You can test it with the stress image (as we did before for the CPU):

sudo docker run --rm -m 128m --name testmem scg/stress --vm 1 --vm-bytes
256M

If you allocate over twice the amount of memory we assign you will get the
following error:

stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [1] (415) <-- worker 7 got signal 9
stress: WARN: [1] (417) now reaping child worker processes
stress: FAIL: [1] (421) kill error: No such process
stress: FAIL: [1] (451) failed run completed in 3s

By default, Docker kills processes in a container if an out-of-memory (OOM)
error occurs. We can change this behaviour on a container but we are not
doing it now.

>
> Maybe it would be more useful to monitor client metrics, like can we reach
> the service over http?
>

I guess It would be useful. We can do it with nagios.

>
>
>>
>> 2. Containers should autorestart if they crash for any reason (with the
>> usual measures to deal with services which crash repeatedly).
>>
>
> That is a feature built into docker.  I've enabled it in container.yml (as
> a global default) and restarted trac with the new parameters.
>

Great. Maybe we need to enable historical resource usage in cadvisor. This
would help us to understand if a container has crashed because it has run
out of resources.

>
> Thanks,
> Sam
>
>
>>
>> 3. Since high-profile services have been deployed using Docker for a
>> while, software for doing (1) and (2) certainly exists already :-)
>>
>>
>> On 27/07/15 22:32, Samuel Cantero wrote:
>> > I didn't shut it down. The trac container was already off when I checked
>> > it.
>> >
>> > On Mon, Jul 27, 2015 at 9:40 PM, Sam P. <sam at sam.today
>> > <mailto:sam at sam.today>> wrote:
>> >
>> >     Did you check the logs before you shut it down?
>> >
>> >
>> >     On Tue, 28 Jul 2015 9:38 am Samuel Cantero <scg at sugarlabs.org
>> >     <mailto:scg at sugarlabs.org>> wrote:
>> >
>> >         Hi Gonzalo,
>> >
>> >         The trac container was down. I just recently turned it on.
>> >
>> >         Somebody turned it off for some reason? If it not the case,
>> >         maybe the container has reached its maximum memory or processor
>> >         limit. The docker container stops in that case.
>> >
>> >         Greetings,
>> >
>> >
>> >
>> >         On Mon, Jul 27, 2015 at 5:01 PM, Gonzalo Odiard
>> >         <godiard at sugarlabs.org <mailto:godiard at sugarlabs.org>> wrote:
>> >
>> >             Today, I get a error
>> >
>> >             502 Bad Gateway
>> >             nginx/1.8.0
>> >
>> >             when try access trac.
>> >
>> >             Gonzalo
>> >
>> >             On Sat, Jul 25, 2015 at 8:40 PM, Gonzalo Odiard
>> >             <godiard at sugarlabs.org <mailto:godiard at sugarlabs.org>>
>> wrote:
>> >
>> >                 Thanks Sams :)
>> >
>> >                 On Sat, Jul 25, 2015 at 3:58 PM, Samuel Cantero
>> >                 <scg at sugarlabs.org <mailto:scg at sugarlabs.org>> wrote:
>> >
>> >                     Hi All,
>> >
>> >                     I just recently check this email. The good news is
>> >                     that Sam P already dockerized this service and now
>> >                     we have the 1.0.7 Trac version.
>> >
>> >                     Greetings,
>> >
>> >                     On Fri, Jul 24, 2015 at 11:06 AM, Gonzalo Odiard
>> >                     <godiard at sugarlabs.org
>> >                     <mailto:godiard at sugarlabs.org>> wrote:
>> >
>> >                         I am pretty sure dnarvaez updated the trac
>> >                         instance a time ago.
>> >
>> >                         On Fri, Jul 24, 2015 at 12:03 PM, Bernie
>> >                         Innocenti <bernie at codewiz.org
>> >                         <mailto:bernie at codewiz.org>> wrote:
>> >
>> >                             On 07/24/2015 10:34 AM, Gonzalo Odiard
>> wrote:
>> >                             > When try to read bughttp://
>> bugs.sugarlabs.org/ticket/4863
>> >                             > get the following error:"Genshi
>> UnicodeEncodeError error while rendering
>> >                             > template (unknown template location)"
>> >                             >
>> >                             > A quick google search show this bug [1]
>> >                             >
>> >                             > The error is originated by simply enter a
>> link like this
>> >                             > "<http://trac.edgewall.org/>"
>> >                             > to a ticket. According to this [2] the
>> bug is already solved on Trac 1.0.2
>> >                             >
>> >                             > Then:
>> >                             > Developers: Please don't add links
>> between <> until the error is solved.
>> >                             > Systems: I don't know what version of
>> trac we have running, could be
>> >                             > possible update?
>> >
>> >                             Who's maintaining Trac these days? The wiki
>> >                             points at dnarvaez and me,
>> >                             but the former is MIA and the latter has
>> >                             installed it in 2008 and not
>> >                             touched it ever since:
>> >
>> >                             https://wiki.sugarlabs.org/go/Service/bugs
>> >
>> >                             Perhaps one of the two Sams would like to
>> >                             upgrade Trac in-place or move
>> >                             it into a docker container?
>> >
>> >                             --
>> >                               _ // Bernie Innocenti
>> >                               \X/ http://codewiz.org
>> >
>> >
>> >
>> >
>> >                         --
>> >                         Gonzalo Odiard
>> >
>> >                         SugarLabs - Software [for | by] children
>> learning
>> >
>> >                         _______________________________________________
>> >                         Systems mailing list
>> >                         Systems at lists.sugarlabs.org
>> >                         <mailto:Systems at lists.sugarlabs.org>
>> >                         http://lists.sugarlabs.org/listinfo/systems
>> >
>> >
>> >
>> >
>> >
>> >                 --
>> >                 Gonzalo Odiard
>> >
>> >                 SugarLabs - Software [for | by] children learning
>> >
>> >
>> >
>> >
>> >             --
>> >             Gonzalo Odiard
>> >
>> >             SugarLabs - Software [for | by] children learning
>> >
>> >
>> >         _______________________________________________
>> >         Systems mailing list
>> >         Systems at lists.sugarlabs.org <mailto:Systems at lists.sugarlabs.org
>> >
>> >         http://lists.sugarlabs.org/listinfo/systems
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Systems mailing list
>> > Systems at lists.sugarlabs.org
>> > http://lists.sugarlabs.org/listinfo/systems
>> >
>>
>>
>> --
>>   _ // Bernie Innocenti
>>   \X/  http://codewiz.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/private/systems/attachments/20150728/a46df47c/attachment.html>