[Systems] mirror testing request

[Adrian Chadd]

I'm finding it hard to organise cacheboy related organisational stuff
from Australia. One of my goals in my next trip to the US is to get
enough people interested in the project to help me found an
organisation and start working more collaboratively.

So I'm open to further help there. :0


Adrian


2009/9/22 Marten Vijn <info at martenvijn.nl>:
> On Tue, 2009-09-22 at 16:15 +0800, Adrian Chadd wrote:
>> The point behind cacheboy is to try and fix this preponderance of
>> differently run mirror systems, with different content distribution
>> methods, most of which don't really have the network topology or
>> network utilisation in consideration.
>>
>> The plan is to be able to put well connected nodes in places with lots
>> of peering but small and/or expensive amounts of transit (eg, places
>> like africa, south america, eastern europe, Australia and New Zealand
>> :) and have all the projects take advantage of it.
>>
> Fully agree with you in a technical way, from a organizational
> perspective CacheBoy seem to be run by only you (or are more people
> involved?)
>
> It is not a case trust, but over time monolithic systems do pass. If
> there is no way to take over the infrastructure when something happens
> to you,  (girlfriend/kids/health issues may happen, and ofcourse I wish
> you are fine) I would not want to relay on CacheBoy only.
>
> I think a dependency in one person,server,uplink is not wise. Therefore
> my idea to have multiple mirror systems is one way of creating more
> redundancy, but the only.
>
> A way the harden CachyBoy would having multiple admins that work for
> CacheBoy. And CacheBoy would become an organization over time
> (foundation or something).
>
> 2ct,
> Marten
>
>
>
>>
>>
>> adrian
>>
>> 2009/9/22 Marten Vijn <info at martenvijn.nl>:
>> > On Mon, 2009-09-21 at 20:20 -0500, David Farning wrote:
>> >> On Mon, Sep 21, 2009 at 7:52 PM, Adrian Chadd <adrian.chadd at gmail.com> wrote:
>> >> > I'd be happy if Cacheboy was handling all of your downloads. My
>> >> > distribution system is slightly more flexible than bouncer and is
>> >> > network topology aware.
>> >>
>> >> This takes us back to the questions.
>> >> Is cacheboy stable enough that we could host the entire infrastructure on it.
>> >> What would we need to do to run sugarlabs.cdn.cacheboy.net at our
>> >> server facility?
>> >
>> > Space on other mirrors do need enough disk space,
>> > atm it really should stay under 20G, not breaking current
>> > agreements.
>> >
>> > And, the other question would what more mirror systems over time?
>> >
>> > The is still possibility at Internet Systems Consortium (ISC) to mirror
>> > a 2G subset. Here is we should decide quick, to use the moment of
>> > opportinuty.  (third time I mention it)
>> >
>> > My idea would to get on a fair amount of mirror systems, and find a way
>> > the balance the load.
>> >
>> > 2ct,
>> >
>> > Marten
>> >
>> >>
>> >> I remember you saying that cacheboy required root on mirror nodes
>> >> because it runs as a cluster.
>> >>
>> >> david
>> >>
>> >>
>> >>
>> >> > Adrian
>> >> >
>> >> > 2009/9/22 David Farning <dfarning at sugarlabs.org>:
>> >> >> On Mon, Sep 21, 2009 at 3:54 PM, Marten Vijn <info at martenvijn.nl> wrote:
>> >> >>> On Mon, 2009-09-21 at 16:35 -0400, Luke Faraone wrote:
>> >> >>>> On Mon, Sep 21, 2009 at 16:29, Marten Vijn <info at martenvijn.nl> wrote:
>> >> >>>>         What is A.SL.O ?
>> >> >>>>
>> >> >>>
>> >> >>> ah clear, these links do not point to download as far i can see or are
>> >> >>> redicted.
>> >> >>
>> >> >> Currently a.sl.o downloads point to activities.sugarlabs.org.  We can
>> >> >> set it up to point to dl.sl.org if there is an advantage.
>> >> >>
>> >> >>> Anyway, currently only:
>> >> >>>
>> >> >>> http://download.sugarlabs.org/ is mirrored. to all mirros.
>> >> >>>
>> >> >>> So if there is copy of activities@ on download@ it will be
>> >> >>> picked up soon 1-24 hrs.
>> >> >>
>> >> >> Ok got it.  Mozilla uses a system called bouncer to control its mirror
>> >> >> net work.  Whenever a file is written to downloads.mozilla.org.
>> >> >> Bouncer starts immediately propagating the file across the mirror
>> >> >> system.  Once bouncer has verified that a particular mirror has a good
>> >> >> copy of the file, it starts redirecting requests to that mirror.
>> >> >>
>> >> >> It seems that that is similar to how things work within cacheboy.  But
>> >> >> we currently have a manual step of rsyncing from dl.sl.o to cacheboy,
>> >> >> dl2.sl.o, and the european mirrors you have set up.  This gives us a
>> >> >> two tier mirror system
>> >> >>
>> >> >> dl.sl.o
>> >> >>  dl2.sl.o
>> >> >>  ftp.nluug.nl
>> >> >>  ftp.nluug.nl
>> >> >>  sugarlabs.cdn.cacheboy.net
>> >> >>
>> >> >> A user manually chooses between one of the above mirrors.  Rsync runs
>> >> >> on a cron job.  Health is monitored by Marten.
>> >> >>
>> >> >> sugarlabs.cdn.cacheboy.net
>> >> >>  *.cdn.cacheboy.net
>> >> >>
>> >> >> If a user choses sugarlabs.cdn.cacheboy.net , they are redirected
>> >> >> around the cacheboy network based on location and load.  Files are
>> >> >> pushed to mirrors as needed.  cacheboy monitors individual mirror's
>> >> >> health and removes them as necessary.
>> >> >>
>> >> >> This leads to two questions:
>> >> >> Is cacheboy stable enough that we could host
>> >> >> sugarlabs.cdn.cacheboy.net at Gnaps and alias dl.sl.o to
>> >> >> sugarlabs.cdn.cacheboy.net ?  This would put the entire download
>> >> >> infrastructure on cacheboy.
>> >> >>
>> >> >> or
>> >> >> Should we look into Bouncer or Fedora Mirror Manager to manage and
>> >> >> redirect within the first level of mirrors?
>> >> >>
>> >> >> david
>> >> >>
>> >> >>> There is no system yet to do geoip based redirections(as far as i know).
>> >> >>>
>> >> >>> Yes it should be there, IHMO it should be in some
>> >> >>> kind of cgi?
>> >> >>>
>> >> >>> apt-get install  libapache2-mod-geoip?
>> >> >>>
>> >> >>> I never played with it, and I don't trust myself
>> >> >>> to write sane cgi's. So if we have talents on board,
>> >> >>> it is time to use them.....
>> >> >>>
>> >> >>> Feel more than free to take over from here. I would prefeer to chase
>> >> >>> some more mirrors.....
>> >> >>>
>> >> >>> cheers,
>> >> >>> Marten
>> >> >>>
>> >> >>>>
>> >> >>>>
>> >> >>>> http://activities.sugarlabs.org/.
>> >> >>>>
>> >> >>>>
>> >> >>>> --
>> >> >>>> Luke Faraone
>> >> >>>> http://luke.faraone.cc
>> >> >>> --
>> >> >>> this email is sent from my mainframe.....
>> >> >>> http://martenvijn.nl                Sugar on a Stick
>> >> >>> http://bsd.wifisoft.org/nek/        The Network Event Kit
>> >> >>> http://opencommunitycamp.org         26th Jul - 2nd August
>> >> >>>
>> >> >>>
>> >> >>
>> >> >
>> >> _______________________________________________
>> >> Systems mailing list
>> >> Systems at lists.sugarlabs.org
>> >> http://lists.sugarlabs.org/listinfo/systems
>> > --
>> > this email is sent from my mainframe.....
>> > http://martenvijn.nl                Sugar on a Stick
>> > http://bsd.wifisoft.org/nek/        The Network Event Kit
>> > http://opencommunitycamp.org         26th Jul - 2nd August
>> >
>> >
> --
> this email is sent from my mainframe.....
> http://martenvijn.nl                Sugar on a Stick
> http://bsd.wifisoft.org/nek/        The Network Event Kit
> http://opencommunitycamp.org         26th Jul - 2nd August
>
>