[Systems] Error on trac reading a bug #4863

Bernie Innocenti bernie at codewiz.org
Tue Jul 28 19:48:09 EDT 2015


On 07/28/2015 09:27 AM, Samuel Cantero wrote:
> Hi All,
> 
> On Tue, Jul 28, 2015 at 6:52 AM, Sam P. <sam at sam.today
> <mailto:sam at sam.today>> wrote:
> 
>     Hi Bernie,
> 
>     On Tue, Jul 28, 2015 at 2:12 PM Bernie Innocenti <bernie at codewiz.org
>     <mailto:bernie at codewiz.org>> wrote:
> 
>         1. We need monitoring for containers health with notifications when
>         parameters go off thresholds. Similar to what Munin does.
> 
> 
>     I'm not sure how that would help us.  A container running out of
>     resources doesn't crash - parsoid runs almost using all the
>     allocated memory and parsoid+trac+others hit their cpu quotas
>     sometimes and just keep running.
> 
> 
> A container running out of memory dies. By default, the memory limit
> value for the container is set to twice as much as the memory parameter
> we specify while starting a container. The limit value is the sum of
> memory and swap. 
> 
> We do not have swap accounting enabled yet on freedom, but for sure we
> are going to have it at the next reboot.

In the best Google tradition, freedom doesn't have any swap space at all :-)


> You can test it with the stress image (as we did before for the CPU):
> 
> sudo docker run --rm -m 128m --name testmem scg/stress --vm 1 --vm-bytes
> 256M
> 
> If you allocate over twice the amount of memory we assign you will get
> the following error:
> 
> stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
> stress: FAIL: [1] (415) <-- worker 7 got signal 9
> stress: WARN: [1] (417) now reaping child worker processes
> stress: FAIL: [1] (421) kill error: No such process
> stress: FAIL: [1] (451) failed run completed in 3s
> 
> By default, Docker kills processes in a container if an out-of-memory
> (OOM) error occurs. We can change this behaviour on a container but we
> are not doing it now.

See my response to Sam P.: killing the entire container is almost always
the right response to an OOM.


>     Maybe it would be more useful to monitor client metrics, like can we
>     reach the service over http?
> 
> I guess It would be useful. We can do it with nagios. 

Cool, does it support Docker? I used Nagios a long time ago at the FSF
and it was pretty good. The reason I was using munin at SL is that it
was easier to setup and I didn't know Nagios back then. Moreover, Munin
seems a bit of a dead project nowadays, so I'd be happy to switch.

Can we run the Nagios frontend in a container too?


>     That is a feature built into docker.  I've enabled it in
>     container.yml (as a global default) and restarted trac with the new
>     parameters.
> 
> Great. Maybe we need to enable historical resource usage in cadvisor.
> This would help us to understand if a container has crashed because it
> has run out of resources.

Cool, another useful feature I didn't know about. Yes, please, let's
enable resource graphing. Is there an http frontend to look at the numbers?


>     Thanks,
>     Sam
>      
> 
> 
>         3. Since high-profile services have been deployed using Docker for a
>         while, software for doing (1) and (2) certainly exists already :-)
> 
> 
>         On 27/07/15 22:32, Samuel Cantero wrote:
>         > I didn't shut it down. The trac container was already off when
>         I checked
>         > it.
>         >
>         > On Mon, Jul 27, 2015 at 9:40 PM, Sam P. <sam at sam.today
>         > <mailto:sam at sam.today <mailto:sam at sam.today>>> wrote:
>         >
>         >     Did you check the logs before you shut it down?
>         >
>         >
>         >     On Tue, 28 Jul 2015 9:38 am Samuel Cantero
>         <scg at sugarlabs.org <mailto:scg at sugarlabs.org>
>         >     <mailto:scg at sugarlabs.org <mailto:scg at sugarlabs.org>>> wrote:
>         >
>         >         Hi Gonzalo,
>         >
>         >         The trac container was down. I just recently turned it on.
>         >
>         >         Somebody turned it off for some reason? If it not the
>         case,
>         >         maybe the container has reached its maximum memory or
>         processor
>         >         limit. The docker container stops in that case.
>         >
>         >         Greetings,
>         >
>         >
>         >
>         >         On Mon, Jul 27, 2015 at 5:01 PM, Gonzalo Odiard
>         >         <godiard at sugarlabs.org <mailto:godiard at sugarlabs.org>
>         <mailto:godiard at sugarlabs.org <mailto:godiard at sugarlabs.org>>>
>         wrote:
>         >
>         >             Today, I get a error
>         >
>         >             502 Bad Gateway
>         >             nginx/1.8.0
>         >
>         >             when try access trac.
>         >
>         >             Gonzalo
>         >
>         >             On Sat, Jul 25, 2015 at 8:40 PM, Gonzalo Odiard
>         >             <godiard at sugarlabs.org
>         <mailto:godiard at sugarlabs.org> <mailto:godiard at sugarlabs.org
>         <mailto:godiard at sugarlabs.org>>> wrote:
>         >
>         >                 Thanks Sams :)
>         >
>         >                 On Sat, Jul 25, 2015 at 3:58 PM, Samuel Cantero
>         >                 <scg at sugarlabs.org <mailto:scg at sugarlabs.org>
>         <mailto:scg at sugarlabs.org <mailto:scg at sugarlabs.org>>> wrote:
>         >
>         >                     Hi All,
>         >
>         >                     I just recently check this email. The good
>         news is
>         >                     that Sam P already dockerized this service
>         and now
>         >                     we have the 1.0.7 Trac version.
>         >
>         >                     Greetings,
>         >
>         >                     On Fri, Jul 24, 2015 at 11:06 AM, Gonzalo
>         Odiard
>         >                     <godiard at sugarlabs.org
>         <mailto:godiard at sugarlabs.org>
>         >                     <mailto:godiard at sugarlabs.org
>         <mailto:godiard at sugarlabs.org>>> wrote:
>         >
>         >                         I am pretty sure dnarvaez updated the trac
>         >                         instance a time ago.
>         >
>         >                         On Fri, Jul 24, 2015 at 12:03 PM, Bernie
>         >                         Innocenti <bernie at codewiz.org
>         <mailto:bernie at codewiz.org>
>         >                         <mailto:bernie at codewiz.org
>         <mailto:bernie at codewiz.org>>> wrote:
>         >
>         >                             On 07/24/2015 10:34 AM, Gonzalo
>         Odiard wrote:
>         >                             > When try to read
>         bughttp://bugs.sugarlabs.org/ticket/4863
>         <http://bugs.sugarlabs.org/ticket/4863>
>         >                             > get the following error:"Genshi
>         UnicodeEncodeError error while rendering
>         >                             > template (unknown template
>         location)"
>         >                             >
>         >                             > A quick google search show this
>         bug [1]
>         >                             >
>         >                             > The error is originated by
>         simply enter a link like this
>         >                             > "<http://trac.edgewall.org/>"
>         >                             > to a ticket. According to this
>         [2] the bug is already solved on Trac 1.0.2
>         >                             >
>         >                             > Then:
>         >                             > Developers: Please don't add
>         links between <> until the error is solved.
>         >                             > Systems: I don't know what
>         version of trac we have running, could be
>         >                             > possible update?
>         >
>         >                             Who's maintaining Trac these days?
>         The wiki
>         >                             points at dnarvaez and me,
>         >                             but the former is MIA and the
>         latter has
>         >                             installed it in 2008 and not
>         >                             touched it ever since:
>         >
>         >                           
>          https://wiki.sugarlabs.org/go/Service/bugs
>         >
>         >                             Perhaps one of the two Sams would
>         like to
>         >                             upgrade Trac in-place or move
>         >                             it into a docker container?
>         >
>         >                             --
>         >                               _ // Bernie Innocenti
>         >                               \X/ http://codewiz.org
>         >
>         >
>         >
>         >
>         >                         --
>         >                         Gonzalo Odiard
>         >
>         >                         SugarLabs - Software [for | by]
>         children learning
>         >
>         >                       
>          _______________________________________________
>         >                         Systems mailing list
>         >                         Systems at lists.sugarlabs.org
>         <mailto:Systems at lists.sugarlabs.org>
>         >                         <mailto:Systems at lists.sugarlabs.org
>         <mailto:Systems at lists.sugarlabs.org>>
>         >                       
>          http://lists.sugarlabs.org/listinfo/systems
>         >
>         >
>         >
>         >
>         >
>         >                 --
>         >                 Gonzalo Odiard
>         >
>         >                 SugarLabs - Software [for | by] children learning
>         >
>         >
>         >
>         >
>         >             --
>         >             Gonzalo Odiard
>         >
>         >             SugarLabs - Software [for | by] children learning
>         >
>         >
>         >         _______________________________________________
>         >         Systems mailing list
>         >         Systems at lists.sugarlabs.org
>         <mailto:Systems at lists.sugarlabs.org>
>         <mailto:Systems at lists.sugarlabs.org
>         <mailto:Systems at lists.sugarlabs.org>>
>         >         http://lists.sugarlabs.org/listinfo/systems
>         >
>         >
>         >
>         >
>         > _______________________________________________
>         > Systems mailing list
>         > Systems at lists.sugarlabs.org <mailto:Systems at lists.sugarlabs.org>
>         > http://lists.sugarlabs.org/listinfo/systems
>         >
> 
> 
>         --
>           _ // Bernie Innocenti
>           \X/  http://codewiz.org
> 
> 
> 
> 
> _______________________________________________
> Systems mailing list
> Systems at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/systems
> 


-- 
 _ // Bernie Innocenti
 \X/  http://codewiz.org


More information about the Systems mailing list