[Systems] [Sugar-devel] git.sugarlabs.org down for unplanned maintenance

[Gonzalo Odiard]

I know nothing about our infraestructure but,
is possible run proyects in development,
like Sugar Network in a different server/vm
than critical services like git?

Gonzalo


On Sat, Apr 12, 2014 at 6:07 AM, Sebastian Silva
<sebastian at fuentelibre.org>wrote:

> Here I just got home. Sorry for the inconvenience I might have caused.
>
> Bernie, do you know which log was/is growing out of hand?
>
> Here's a report on everything I know about the issue.
> We've been experiencing some performance degradation and also some
> downtime in Sugar Network services (this is documented at
> http://tareas.somosazucar.org/hxp/issue71 ).
> We've seen a burst in users since deployment OS images with Sugar Network
> features ( http://network.sugarlabs.org/stats-viewer/ growing pretty fast
> user_total).
> There is a notification feature that is polling the sugar network node
> service.
> This was causing the allocation and exhaustion of resources (open files).
> Crashes got to a frequency of every hour or so.
> It's code I don't understand really well, but I went ahead and patched the
> Sugar Network with:
> http://tareas.somosazucar.org/hxp/file66/sn_disable_notifications.patch
> This made the SN much snappier and it stopped crashing. However logs were
> saving a traceback several times per second. I thought I had contained the
> log issue but apparently I missed some other logs (I guess apache logs but
> they seem clean now).
>
> I took a glance at jita and could not find the growing log.
>
> Let me know where I can help mitigation.
>
> Regards
> Sebastian
>
>
> El vie, 11 de abr 2014 a las 7:51 PM, Bernie Innocenti <
> bernie at sugarlabs.org> escribió:
>
> I was notified that git.sugarlabs.org was showing errors. After some head
> scraping I realized that the root filesystem on jita was full. I looked
> around and found giant request logs containing millions of requests
> apparently originating from XOs located in Peru. We've been DDOSed by our
> own creature :-) Anyway, the machine also had a giant, very fragmented
> mysql database that I'm currently cleaning up. Gitorious will be back
> online in less than 1 hour. Contact me on IRC if this is blocking your
> work, I can postpone the maintenance.
> --
> Bernie Innocenti Sugar Labs Infrastructure Team
> http://wiki.sugarlabs.org/go/Infrastructure_Team_______________________________________________ Sugar-devel mailing list
> Sugar-devel at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/sugar-devel
>
>
> _______________________________________________
> Sugar-devel mailing list
> Sugar-devel at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/sugar-devel
>
>


-- 
Gonzalo Odiard

SugarLabs - Software for children learning
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/private/systems/attachments/20140412/a161d83f/attachment.html>