Hi All, Sorry about my naivety with the memory limits. I now see how this is an issue :) I'll look into resource monitoring and alerting systems for docker. The HN article was pretty interesting. In some respects, many of those issues are not applicable to us (we use simple images and don't need secrets at build time). The article does point out some interesting avenues (OverlayFS) and highlights the momentum of the docker world. The comments were very negative about docker (particularly for HN), but docker does provide us value in being containerization to un-tangle our services and the Dockerfile buid system. Thanks, Sam <div class="gmail_quote"><div dir="ltr">On Wed, 29 Jul 2015 4:10 pm Bernie Innocenti <<a href="mailto:bernie@codewiz.org" target="_blank">bernie@codewiz.org</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">By the way, found this relevant article today on Hacker News: <a href="http://sirupsen.com/production-docker/" rel="noreferrer" target="_blank">http://sirupsen.com/production-docker/</a> The comments are also interesting: <a href="https://news.ycombinator.com/item?id=9961537" rel="noreferrer" target="_blank">https://news.ycombinator.com/item?id=9961537</a> On 28/07/15 19:48, Bernie Innocenti wrote: > On 07/28/2015 09:27 AM, Samuel Cantero wrote: >> Hi All, >> >> On Tue, Jul 28, 2015 at 6:52 AM, Sam P. <sam@sam.today >> <mailto:<a href="mailto:sam@sam.today" target="_blank">sam@sam.today</a>>> wrote: >> >> Hi Bernie, >> >> On Tue, Jul 28, 2015 at 2:12 PM Bernie Innocenti <<a href="mailto:bernie@codewiz.org" target="_blank">bernie@codewiz.org</a> >> <mailto:<a href="mailto:bernie@codewiz.org" target="_blank">bernie@codewiz.org</a>>> wrote: >> >> 1. We need monitoring for containers health with notifications when >> parameters go off thresholds. Similar to what Munin does. >> >> >> I'm not sure how that would help us. A container running out of >> resources doesn't crash - parsoid runs almost using all the >> allocated memory and parsoid+trac+others hit their cpu quotas >> sometimes and just keep running. >> >> >> A container running out of memory dies. By default, the memory limit >> value for the container is set to twice as much as the memory parameter >> we specify while starting a container. The limit value is the sum of >> memory and swap. >> >> We do not have swap accounting enabled yet on freedom, but for sure we >> are going to have it at the next reboot. > > In the best Google tradition, freedom doesn't have any swap space at all :-) > > >> You can test it with the stress image (as we did before for the CPU): >> >> sudo docker run --rm -m 128m --name testmem scg/stress --vm 1 --vm-bytes >> 256M >> >> If you allocate over twice the amount of memory we assign you will get >> the following error: >> >> stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd >> stress: FAIL: [1] (415) <-- worker 7 got signal 9 >> stress: WARN: [1] (417) now reaping child worker processes >> stress: FAIL: [1] (421) kill error: No such process >> stress: FAIL: [1] (451) failed run completed in 3s >> >> By default, Docker kills processes in a container if an out-of-memory >> (OOM) error occurs. We can change this behaviour on a container but we >> are not doing it now. > > See my response to Sam P.: killing the entire container is almost always > the right response to an OOM. > > >> Maybe it would be more useful to monitor client metrics, like can we >> reach the service over http? >> >> I guess It would be useful. We can do it with nagios. > > Cool, does it support Docker? I used Nagios a long time ago at the FSF > and it was pretty good. The reason I was using munin at SL is that it > was easier to setup and I didn't know Nagios back then. Moreover, Munin > seems a bit of a dead project nowadays, so I'd be happy to switch. > > Can we run the Nagios frontend in a container too? > > >> That is a feature built into docker. I've enabled it in >> container.yml (as a global default) and restarted trac with the new >> parameters. >> >> Great. Maybe we need to enable historical resource usage in cadvisor. >> This would help us to understand if a container has crashed because it >> has run out of resources. > > Cool, another useful feature I didn't know about. Yes, please, let's > enable resource graphing. Is there an http frontend to look at the numbers? > > >> Thanks, >> Sam >> >> >> >> 3. Since high-profile services have been deployed using Docker for a >> while, software for doing (1) and (2) certainly exists already :-) >> >> >> On 27/07/15 22:32, Samuel Cantero wrote: >> > I didn't shut it down. The trac container was already off when >> I checked >> > it. >> > >> > On Mon, Jul 27, 2015 at 9:40 PM, Sam P. <sam@sam.today >> > <mailto:<a href="mailto:sam@sam.today" target="_blank">sam@sam.today</a> <mailto:<a href="mailto:sam@sam.today" target="_blank">sam@sam.today</a>>>> wrote: >> > >> > Did you check the logs before you shut it down? >> > >> > >> > On Tue, 28 Jul 2015 9:38 am Samuel Cantero >> <<a href="mailto:scg@sugarlabs.org" target="_blank">scg@sugarlabs.org</a> <mailto:<a href="mailto:scg@sugarlabs.org" target="_blank">scg@sugarlabs.org</a>> >> > <mailto:<a href="mailto:scg@sugarlabs.org" target="_blank">scg@sugarlabs.org</a> <mailto:<a href="mailto:scg@sugarlabs.org" target="_blank">scg@sugarlabs.org</a>>>> wrote: >> > >> > Hi Gonzalo, >> > >> > The trac container was down. I just recently turned it on. >> > >> > Somebody turned it off for some reason? If it not the >> case, >> > maybe the container has reached its maximum memory or >> processor >> > limit. The docker container stops in that case. >> > >> > Greetings, >> > >> > >> > >> > On Mon, Jul 27, 2015 at 5:01 PM, Gonzalo Odiard >> > <<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a> <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a>> >> <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a> <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a>>>> >> wrote: >> > >> > Today, I get a error >> > >> > 502 Bad Gateway >> > nginx/1.8.0 >> > >> > when try access trac. >> > >> > Gonzalo >> > >> > On Sat, Jul 25, 2015 at 8:40 PM, Gonzalo Odiard >> > <<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a> >> <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a>> <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a> >> <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a>>>> wrote: >> > >> > Thanks Sams :) >> > >> > On Sat, Jul 25, 2015 at 3:58 PM, Samuel Cantero >> > <<a href="mailto:scg@sugarlabs.org" target="_blank">scg@sugarlabs.org</a> <mailto:<a href="mailto:scg@sugarlabs.org" target="_blank">scg@sugarlabs.org</a>> >> <mailto:<a href="mailto:scg@sugarlabs.org" target="_blank">scg@sugarlabs.org</a> <mailto:<a href="mailto:scg@sugarlabs.org" target="_blank">scg@sugarlabs.org</a>>>> wrote: >> > >> > Hi All, >> > >> > I just recently check this email. The good >> news is >> > that Sam P already dockerized this service >> and now >> > we have the 1.0.7 Trac version. >> > >> > Greetings, >> > >> > On Fri, Jul 24, 2015 at 11:06 AM, Gonzalo >> Odiard >> > <<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a> >> <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a>> >> > <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a> >> <mailto:<a href="mailto:godiard@sugarlabs.org" target="_blank">godiard@sugarlabs.org</a>>>> wrote: >> > >> > I am pretty sure dnarvaez updated the trac >> > instance a time ago. >> > >> > On Fri, Jul 24, 2015 at 12:03 PM, Bernie >> > Innocenti <<a href="mailto:bernie@codewiz.org" target="_blank">bernie@codewiz.org</a> >> <mailto:<a href="mailto:bernie@codewiz.org" target="_blank">bernie@codewiz.org</a>> >> > <mailto:<a href="mailto:bernie@codewiz.org" target="_blank">bernie@codewiz.org</a> >> <mailto:<a href="mailto:bernie@codewiz.org" target="_blank">bernie@codewiz.org</a>>>> wrote: >> > >> > On 07/24/2015 10:34 AM, Gonzalo >> Odiard wrote: >> > > When try to read >> bughttp://<a href="http://bugs.sugarlabs.org/ticket/4863" rel="noreferrer" target="_blank">bugs.sugarlabs.org/ticket/4863</a> >> <<a href="http://bugs.sugarlabs.org/ticket/4863" rel="noreferrer" target="_blank">http://bugs.sugarlabs.org/ticket/4863</a>> >> > > get the following error:"Genshi >> UnicodeEncodeError error while rendering >> > > template (unknown template >> location)" >> > > >> > > A quick google search show this >> bug [1] >> > > >> > > The error is originated by >> simply enter a link like this >> > > "<<a href="http://trac.edgewall.org/" rel="noreferrer" target="_blank">http://trac.edgewall.org/</a>>" >> > > to a ticket. According to this >> [2] the bug is already solved on Trac 1.0.2 >> > > >> > > Then: >> > > Developers: Please don't add >> links between <> until the error is solved. >> > > Systems: I don't know what >> version of trac we have running, could be >> > > possible update? >> > >> > Who's maintaining Trac these days? >> The wiki >> > points at dnarvaez and me, >> > but the former is MIA and the >> latter has >> > installed it in 2008 and not >> > touched it ever since: >> > >> > >> <a href="https://wiki.sugarlabs.org/go/Service/bugs" rel="noreferrer" target="_blank">https://wiki.sugarlabs.org/go/Service/bugs</a> >> > >> > Perhaps one of the two Sams would >> like to >> > upgrade Trac in-place or move >> > it into a docker container? >> > >> > -- >> > _ // Bernie Innocenti >> > \X/ <a href="http://codewiz.org" rel="noreferrer" target="_blank">http://codewiz.org</a> >> > >> > >> > >> > >> > -- >> > Gonzalo Odiard >> > >> > SugarLabs - Software [for | by] >> children learning >> > >> > >> _______________________________________________ >> > Systems mailing list >> > <a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a> >> <mailto:<a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a>> >> > <mailto:<a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a> >> <mailto:<a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a>>> >> > >> <a href="http://lists.sugarlabs.org/listinfo/systems" rel="noreferrer" target="_blank">http://lists.sugarlabs.org/listinfo/systems</a> >> > >> > >> > >> > >> > >> > -- >> > Gonzalo Odiard >> > >> > SugarLabs - Software [for | by] children learning >> > >> > >> > >> > >> > -- >> > Gonzalo Odiard >> > >> > SugarLabs - Software [for | by] children learning >> > >> > >> > _______________________________________________ >> > Systems mailing list >> > <a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a> >> <mailto:<a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a>> >> <mailto:<a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a> >> <mailto:<a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a>>> >> > <a href="http://lists.sugarlabs.org/listinfo/systems" rel="noreferrer" target="_blank">http://lists.sugarlabs.org/listinfo/systems</a> >> > >> > >> > >> > >> > _______________________________________________ >> > Systems mailing list >> > <a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a> <mailto:<a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a>> >> > <a href="http://lists.sugarlabs.org/listinfo/systems" rel="noreferrer" target="_blank">http://lists.sugarlabs.org/listinfo/systems</a> >> > >> >> >> -- >> _ // Bernie Innocenti >> \X/ <a href="http://codewiz.org" rel="noreferrer" target="_blank">http://codewiz.org</a> >> >> >> >> >> _______________________________________________ >> Systems mailing list >> <a href="mailto:Systems@lists.sugarlabs.org" target="_blank">Systems@lists.sugarlabs.org</a> >> <a href="http://lists.sugarlabs.org/listinfo/systems" rel="noreferrer" target="_blank">http://lists.sugarlabs.org/listinfo/systems</a> >> > > -- _ // Bernie Innocenti \X/ <a href="http://codewiz.org" rel="noreferrer" target="_blank">http://codewiz.org</a> </blockquote></div>