[Systems] [Sugar-devel] git.sugarlabs.org down for unplanned maintenance
sebastian at fuentelibre.org
Sat Apr 12 14:39:58 EDT 2014
If it is necessary it could be moved.
Sugar Network at this point is "in production" and we do have a process
for deployment (there is testing and devel instance).
Alsroot has had a pretty good track record of keeping git up and
running. I think it's a shame people are moving to github.
What we should be asking I think is how we can provide better service
for our users/developers (for example, having more people monitoring
services and reacting when things crash).
El sáb, 12 de abr 2014 a las 9:59 AM, Gonzalo Odiard
<godiard at sugarlabs.org> escribió:
> I know nothing about our infraestructure but,
> is possible run proyects in development,
> like Sugar Network in a different server/vm
> than critical services like git?
> On Sat, Apr 12, 2014 at 6:07 AM, Sebastian Silva
> <sebastian at fuentelibre.org> wrote:
>> Here I just got home. Sorry for the inconvenience I might have
>> Bernie, do you know which log was/is growing out of hand?
>> Here's a report on everything I know about the issue.
>> We've been experiencing some performance degradation and also some
>> downtime in Sugar Network services (this is documented at
>> http://tareas.somosazucar.org/hxp/issue71 ).
>> We've seen a burst in users since deployment OS images with Sugar
>> Network features ( http://network.sugarlabs.org/stats-viewer/
>> growing pretty fast user_total).
>> There is a notification feature that is polling the sugar network
>> node service.
>> This was causing the allocation and exhaustion of resources (open
>> files). Crashes got to a frequency of every hour or so.
>> It's code I don't understand really well, but I went ahead and
>> patched the Sugar Network with:
>> This made the SN much snappier and it stopped crashing. However logs
>> were saving a traceback several times per second. I thought I had
>> contained the log issue but apparently I missed some other logs (I
>> guess apache logs but they seem clean now).
>> I took a glance at jita and could not find the growing log.
>> Let me know where I can help mitigation.
>> El vie, 11 de abr 2014 a las 7:51 PM, Bernie Innocenti
>> <bernie at sugarlabs.org> escribió:
>>> I was notified that git.sugarlabs.org was showing errors.
>>> After some head scraping I realized that the root filesystem on
>>> jita was
>>> full. I looked around and found giant request logs containing
>>> of requests apparently originating from XOs located in Peru.
>>> We've been DDOSed by our own creature :-)
>>> Anyway, the machine also had a giant, very fragmented mysql database
>>> that I'm currently cleaning up. Gitorious will be back online in
>>> than 1 hour. Contact me on IRC if this is blocking your work, I can
>>> postpone the maintenance.
>>> Bernie Innocenti
>>> Sugar Labs Infrastructure Team
>>> Sugar-devel mailing list
>>> Sugar-devel at lists.sugarlabs.org
>> Sugar-devel mailing list
>> Sugar-devel at lists.sugarlabs.org
> Gonzalo Odiard
> SugarLabs - Software for children learning
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Systems