[Systems] [Sugar-devel] git.sugarlabs.org down for unplanned maintenance

[Sebastian Silva]

If it is necessary it could be moved.
Sugar Network at this point is "in production" and we do have a process 
for deployment (there is testing and devel instance).
Alsroot has had a pretty good track record of keeping git up and 
running. I think it's a shame people are moving to github.
What we should be asking I think is how we can provide better service 
for our users/developers (for example, having more people monitoring 
services and reacting when things crash).

Regards,
Sebastian

El sáb, 12 de abr 2014 a las 9:59 AM, Gonzalo Odiard 
<godiard at sugarlabs.org> escribió:
> I know nothing about our infraestructure but,
> is possible run proyects in development,
> like Sugar Network in a different server/vm
> than critical services like git?
> 
> Gonzalo
> 
> 
> On Sat, Apr 12, 2014 at 6:07 AM, Sebastian Silva 
> <sebastian at fuentelibre.org> wrote:
>> Here I just got home. Sorry for the inconvenience I might have 
>> caused.
>> 
>> Bernie, do you know which log was/is growing out of hand?
>> 
>> Here's a report on everything I know about the issue.
>> We've been experiencing some performance degradation and also some 
>> downtime in Sugar Network services (this is documented at 
>> http://tareas.somosazucar.org/hxp/issue71 ).
>> We've seen a burst in users since deployment OS images with Sugar 
>> Network features ( http://network.sugarlabs.org/stats-viewer/ 
>> growing pretty fast user_total).
>> There is a notification feature that is polling the sugar network 
>> node service.
>> This was causing the allocation and exhaustion of resources (open 
>> files). Crashes got to a frequency of every hour or so.
>> It's code I don't understand really well, but I went ahead and 
>> patched the Sugar Network with: 
>> http://tareas.somosazucar.org/hxp/file66/sn_disable_notifications.patch 
>> This made the SN much snappier and it stopped crashing. However logs 
>> were saving a traceback several times per second. I thought I had 
>> contained the log issue but apparently I missed some other logs (I 
>> guess apache logs but they seem clean now).
>> 
>> I took a glance at jita and could not find the growing log.
>> 
>> Let me know where I can help mitigation.
>> 
>> Regards
>> Sebastian
>> 
>> 
>> El vie, 11 de abr 2014 a las 7:51 PM, Bernie Innocenti 
>> <bernie at sugarlabs.org> escribió:
>> 
>>> I was notified that git.sugarlabs.org was showing errors.
>>> 
>>> After some head scraping I realized that the root filesystem on 
>>> jita was
>>> full. I looked around and found giant request logs containing 
>>> millions
>>> of requests apparently originating from XOs located in Peru.
>>> 
>>> We've been DDOSed by our own creature :-)
>>> 
>>> Anyway, the machine also had a giant, very fragmented mysql database
>>> that I'm currently cleaning up. Gitorious will be back online in 
>>> less
>>> than 1 hour. Contact me on IRC if this is blocking your work, I can
>>> postpone the maintenance.
>>> 
>>> -- 
>>> Bernie Innocenti
>>> Sugar Labs Infrastructure Team
>>> http://wiki.sugarlabs.org/go/Infrastructure_Team
>>> _______________________________________________
>>> Sugar-devel mailing list
>>> Sugar-devel at lists.sugarlabs.org
>>> http://lists.sugarlabs.org/listinfo/sugar-devel
>> 
>> _______________________________________________
>> Sugar-devel mailing list
>> Sugar-devel at lists.sugarlabs.org
>> http://lists.sugarlabs.org/listinfo/sugar-devel
>> 
> 
> 
> 
> -- 
> Gonzalo Odiard
> 
> SugarLabs - Software for children learning 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/private/systems/attachments/20140412/da0cd123/attachment.html>