[Sugar-devel] [Systems] git.sugarlabs.org down for unplanned maintenance

[Bernie Innocenti]

Ok, we're back in business, with a snappier database too.

Today I felt lucky so I also replaced the aging MySQL 5.1 with a shiny
new MariaDB 10. Don't worry, MariaDB should be 100% backwards
compatible, and we do daily dumps in case anything goes wrong.

I also deleted several *millions* of records from the database for old
login sessions and logs of clone actions. You may have to login again,
but Gitorious feels lot faster now.

We still need to keep an eye at those evil XOs that keep reconnecting to
network.sugarlabs.org. Icarito, can you look at implementing some form
of exponential backoff? If the fix can't be deployed within a few days,
we should defend ourselves with iptables rules or at least stop logging
every connection.

As always, please notify us if anything malfunctions. Note that alsroot
said he would be offline until Apr 13.


On 04/11/2014 08:51 PM, Bernie Innocenti wrote:
> I was notified that git.sugarlabs.org was showing errors.
> 
> After some head scraping I realized that the root filesystem on jita was
> full. I looked around and found giant request logs containing millions
> of requests apparently originating from XOs located in Peru.
> 
> We've been DDOSed by our own creature :-)
> 
> Anyway, the machine also had a giant, very fragmented mysql database
> that I'm currently cleaning up. Gitorious will be back online in less
> than 1 hour. Contact me on IRC if this is blocking your work, I can
> postpone the maintenance.

-- 
Bernie Innocenti
Sugar Labs Infrastructure Team
http://wiki.sugarlabs.org/go/Infrastructure_Team