[Sugar-devel] [Systems] Error on trac reading a bug #4863
Sam P.
sam at sam.today
Thu Jul 30 08:12:32 EDT 2015
Hi All,
Sorry about my naivety with the memory limits. I now see how this is an
issue :)
I'll look into resource monitoring and alerting systems for docker.
The HN article was pretty interesting. In some respects, many of those
issues are not applicable to us (we use simple images and don't need
secrets at build time). The article does point out some interesting
avenues (OverlayFS) and highlights the momentum of the docker world. The
comments were very negative about docker (particularly for HN), but docker
does provide us value in being containerization to un-tangle our services
and the Dockerfile buid system.
Thanks,
Sam
On Wed, 29 Jul 2015 4:10 pm Bernie Innocenti <bernie at codewiz.org> wrote:
> By the way, found this relevant article today on Hacker News:
>
> http://sirupsen.com/production-docker/
>
> The comments are also interesting:
>
> https://news.ycombinator.com/item?id=9961537
>
>
> On 28/07/15 19:48, Bernie Innocenti wrote:
> > On 07/28/2015 09:27 AM, Samuel Cantero wrote:
> >> Hi All,
> >>
> >> On Tue, Jul 28, 2015 at 6:52 AM, Sam P. <sam at sam.today
> >> <mailto:sam at sam.today>> wrote:
> >>
> >> Hi Bernie,
> >>
> >> On Tue, Jul 28, 2015 at 2:12 PM Bernie Innocenti <
> bernie at codewiz.org
> >> <mailto:bernie at codewiz.org>> wrote:
> >>
> >> 1. We need monitoring for containers health with notifications
> when
> >> parameters go off thresholds. Similar to what Munin does.
> >>
> >>
> >> I'm not sure how that would help us. A container running out of
> >> resources doesn't crash - parsoid runs almost using all the
> >> allocated memory and parsoid+trac+others hit their cpu quotas
> >> sometimes and just keep running.
> >>
> >>
> >> A container running out of memory dies. By default, the memory limit
> >> value for the container is set to twice as much as the memory parameter
> >> we specify while starting a container. The limit value is the sum of
> >> memory and swap.
> >>
> >> We do not have swap accounting enabled yet on freedom, but for sure we
> >> are going to have it at the next reboot.
> >
> > In the best Google tradition, freedom doesn't have any swap space at all
> :-)
> >
> >
> >> You can test it with the stress image (as we did before for the CPU):
> >>
> >> sudo docker run --rm -m 128m --name testmem scg/stress --vm 1 --vm-bytes
> >> 256M
> >>
> >> If you allocate over twice the amount of memory we assign you will get
> >> the following error:
> >>
> >> stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
> >> stress: FAIL: [1] (415) <-- worker 7 got signal 9
> >> stress: WARN: [1] (417) now reaping child worker processes
> >> stress: FAIL: [1] (421) kill error: No such process
> >> stress: FAIL: [1] (451) failed run completed in 3s
> >>
> >> By default, Docker kills processes in a container if an out-of-memory
> >> (OOM) error occurs. We can change this behaviour on a container but we
> >> are not doing it now.
> >
> > See my response to Sam P.: killing the entire container is almost always
> > the right response to an OOM.
> >
> >
> >> Maybe it would be more useful to monitor client metrics, like can
> we
> >> reach the service over http?
> >>
> >> I guess It would be useful. We can do it with nagios.
> >
> > Cool, does it support Docker? I used Nagios a long time ago at the FSF
> > and it was pretty good. The reason I was using munin at SL is that it
> > was easier to setup and I didn't know Nagios back then. Moreover, Munin
> > seems a bit of a dead project nowadays, so I'd be happy to switch.
> >
> > Can we run the Nagios frontend in a container too?
> >
> >
> >> That is a feature built into docker. I've enabled it in
> >> container.yml (as a global default) and restarted trac with the new
> >> parameters.
> >>
> >> Great. Maybe we need to enable historical resource usage in cadvisor.
> >> This would help us to understand if a container has crashed because it
> >> has run out of resources.
> >
> > Cool, another useful feature I didn't know about. Yes, please, let's
> > enable resource graphing. Is there an http frontend to look at the
> numbers?
> >
> >
> >> Thanks,
> >> Sam
> >>
> >>
> >>
> >> 3. Since high-profile services have been deployed using Docker
> for a
> >> while, software for doing (1) and (2) certainly exists already
> :-)
> >>
> >>
> >> On 27/07/15 22:32, Samuel Cantero wrote:
> >> > I didn't shut it down. The trac container was already off
> when
> >> I checked
> >> > it.
> >> >
> >> > On Mon, Jul 27, 2015 at 9:40 PM, Sam P. <sam at sam.today
> >> > <mailto:sam at sam.today <mailto:sam at sam.today>>> wrote:
> >> >
> >> > Did you check the logs before you shut it down?
> >> >
> >> >
> >> > On Tue, 28 Jul 2015 9:38 am Samuel Cantero
> >> <scg at sugarlabs.org <mailto:scg at sugarlabs.org>
> >> > <mailto:scg at sugarlabs.org <mailto:scg at sugarlabs.org>>>
> wrote:
> >> >
> >> > Hi Gonzalo,
> >> >
> >> > The trac container was down. I just recently turned
> it on.
> >> >
> >> > Somebody turned it off for some reason? If it not the
> >> case,
> >> > maybe the container has reached its maximum memory or
> >> processor
> >> > limit. The docker container stops in that case.
> >> >
> >> > Greetings,
> >> >
> >> >
> >> >
> >> > On Mon, Jul 27, 2015 at 5:01 PM, Gonzalo Odiard
> >> > <godiard at sugarlabs.org <mailto:godiard at sugarlabs.org
> >
> >> <mailto:godiard at sugarlabs.org <mailto:godiard at sugarlabs.org>>>
> >> wrote:
> >> >
> >> > Today, I get a error
> >> >
> >> > 502 Bad Gateway
> >> > nginx/1.8.0
> >> >
> >> > when try access trac.
> >> >
> >> > Gonzalo
> >> >
> >> > On Sat, Jul 25, 2015 at 8:40 PM, Gonzalo Odiard
> >> > <godiard at sugarlabs.org
> >> <mailto:godiard at sugarlabs.org> <mailto:godiard at sugarlabs.org
> >> <mailto:godiard at sugarlabs.org>>> wrote:
> >> >
> >> > Thanks Sams :)
> >> >
> >> > On Sat, Jul 25, 2015 at 3:58 PM, Samuel
> Cantero
> >> > <scg at sugarlabs.org <mailto:scg at sugarlabs.org
> >
> >> <mailto:scg at sugarlabs.org <mailto:scg at sugarlabs.org>>> wrote:
> >> >
> >> > Hi All,
> >> >
> >> > I just recently check this email. The
> good
> >> news is
> >> > that Sam P already dockerized this
> service
> >> and now
> >> > we have the 1.0.7 Trac version.
> >> >
> >> > Greetings,
> >> >
> >> > On Fri, Jul 24, 2015 at 11:06 AM, Gonzalo
> >> Odiard
> >> > <godiard at sugarlabs.org
> >> <mailto:godiard at sugarlabs.org>
> >> > <mailto:godiard at sugarlabs.org
> >> <mailto:godiard at sugarlabs.org>>> wrote:
> >> >
> >> > I am pretty sure dnarvaez updated
> the trac
> >> > instance a time ago.
> >> >
> >> > On Fri, Jul 24, 2015 at 12:03 PM,
> Bernie
> >> > Innocenti <bernie at codewiz.org
> >> <mailto:bernie at codewiz.org>
> >> > <mailto:bernie at codewiz.org
> >> <mailto:bernie at codewiz.org>>> wrote:
> >> >
> >> > On 07/24/2015 10:34 AM, Gonzalo
> >> Odiard wrote:
> >> > > When try to read
> >> bughttp://bugs.sugarlabs.org/ticket/4863
> >> <http://bugs.sugarlabs.org/ticket/4863>
> >> > > get the following error:"Genshi
> >> UnicodeEncodeError error while rendering
> >> > > template (unknown template
> >> location)"
> >> > >
> >> > > A quick google search show this
> >> bug [1]
> >> > >
> >> > > The error is originated by
> >> simply enter a link like this
> >> > > "<http://trac.edgewall.org/>"
> >> > > to a ticket. According to this
> >> [2] the bug is already solved on Trac 1.0.2
> >> > >
> >> > > Then:
> >> > > Developers: Please don't add
> >> links between <> until the error is solved.
> >> > > Systems: I don't know what
> >> version of trac we have running, could be
> >> > > possible update?
> >> >
> >> > Who's maintaining Trac these
> days?
> >> The wiki
> >> > points at dnarvaez and me,
> >> > but the former is MIA and the
> >> latter has
> >> > installed it in 2008 and not
> >> > touched it ever since:
> >> >
> >> >
> >> https://wiki.sugarlabs.org/go/Service/bugs
> >> >
> >> > Perhaps one of the two Sams would
> >> like to
> >> > upgrade Trac in-place or move
> >> > it into a docker container?
> >> >
> >> > --
> >> > _ // Bernie Innocenti
> >> > \X/ http://codewiz.org
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Gonzalo Odiard
> >> >
> >> > SugarLabs - Software [for | by]
> >> children learning
> >> >
> >> >
> >> _______________________________________________
> >> > Systems mailing list
> >> > Systems at lists.sugarlabs.org
> >> <mailto:Systems at lists.sugarlabs.org>
> >> > <mailto:Systems at lists.sugarlabs.org
> >> <mailto:Systems at lists.sugarlabs.org>>
> >> >
> >> http://lists.sugarlabs.org/listinfo/systems
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Gonzalo Odiard
> >> >
> >> > SugarLabs - Software [for | by] children
> learning
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Gonzalo Odiard
> >> >
> >> > SugarLabs - Software [for | by] children learning
> >> >
> >> >
> >> > _______________________________________________
> >> > Systems mailing list
> >> > Systems at lists.sugarlabs.org
> >> <mailto:Systems at lists.sugarlabs.org>
> >> <mailto:Systems at lists.sugarlabs.org
> >> <mailto:Systems at lists.sugarlabs.org>>
> >> > http://lists.sugarlabs.org/listinfo/systems
> >> >
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > Systems mailing list
> >> > Systems at lists.sugarlabs.org <mailto:
> Systems at lists.sugarlabs.org>
> >> > http://lists.sugarlabs.org/listinfo/systems
> >> >
> >>
> >>
> >> --
> >> _ // Bernie Innocenti
> >> \X/ http://codewiz.org
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Systems mailing list
> >> Systems at lists.sugarlabs.org
> >> http://lists.sugarlabs.org/listinfo/systems
> >>
> >
> >
>
>
> --
> _ // Bernie Innocenti
> \X/ http://codewiz.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20150730/bfb365e1/attachment-0001.html>
More information about the Sugar-devel
mailing list