[Systems] mirror testing request

[Adrian Chadd]

The point behind cacheboy is to try and fix this preponderance of
differently run mirror systems, with different content distribution
methods, most of which don't really have the network topology or
network utilisation in consideration.

The plan is to be able to put well connected nodes in places with lots
of peering but small and/or expensive amounts of transit (eg, places
like africa, south america, eastern europe, Australia and New Zealand
:) and have all the projects take advantage of it.



adrian

2009/9/22 Marten Vijn <info at martenvijn.nl>:
> On Mon, 2009-09-21 at 20:20 -0500, David Farning wrote:
>> On Mon, Sep 21, 2009 at 7:52 PM, Adrian Chadd <adrian.chadd at gmail.com> wrote:
>> > I'd be happy if Cacheboy was handling all of your downloads. My
>> > distribution system is slightly more flexible than bouncer and is
>> > network topology aware.
>>
>> This takes us back to the questions.
>> Is cacheboy stable enough that we could host the entire infrastructure on it.
>> What would we need to do to run sugarlabs.cdn.cacheboy.net at our
>> server facility?
>
> Space on other mirrors do need enough disk space,
> atm it really should stay under 20G, not breaking current
> agreements.
>
> And, the other question would what more mirror systems over time?
>
> The is still possibility at Internet Systems Consortium (ISC) to mirror
> a 2G subset. Here is we should decide quick, to use the moment of
> opportinuty.  (third time I mention it)
>
> My idea would to get on a fair amount of mirror systems, and find a way
> the balance the load.
>
> 2ct,
>
> Marten
>
>>
>> I remember you saying that cacheboy required root on mirror nodes
>> because it runs as a cluster.
>>
>> david
>>
>>
>>
>> > Adrian
>> >
>> > 2009/9/22 David Farning <dfarning at sugarlabs.org>:
>> >> On Mon, Sep 21, 2009 at 3:54 PM, Marten Vijn <info at martenvijn.nl> wrote:
>> >>> On Mon, 2009-09-21 at 16:35 -0400, Luke Faraone wrote:
>> >>>> On Mon, Sep 21, 2009 at 16:29, Marten Vijn <info at martenvijn.nl> wrote:
>> >>>>         What is A.SL.O ?
>> >>>>
>> >>>
>> >>> ah clear, these links do not point to download as far i can see or are
>> >>> redicted.
>> >>
>> >> Currently a.sl.o downloads point to activities.sugarlabs.org.  We can
>> >> set it up to point to dl.sl.org if there is an advantage.
>> >>
>> >>> Anyway, currently only:
>> >>>
>> >>> http://download.sugarlabs.org/ is mirrored. to all mirros.
>> >>>
>> >>> So if there is copy of activities@ on download@ it will be
>> >>> picked up soon 1-24 hrs.
>> >>
>> >> Ok got it.  Mozilla uses a system called bouncer to control its mirror
>> >> net work.  Whenever a file is written to downloads.mozilla.org.
>> >> Bouncer starts immediately propagating the file across the mirror
>> >> system.  Once bouncer has verified that a particular mirror has a good
>> >> copy of the file, it starts redirecting requests to that mirror.
>> >>
>> >> It seems that that is similar to how things work within cacheboy.  But
>> >> we currently have a manual step of rsyncing from dl.sl.o to cacheboy,
>> >> dl2.sl.o, and the european mirrors you have set up.  This gives us a
>> >> two tier mirror system
>> >>
>> >> dl.sl.o
>> >>  dl2.sl.o
>> >>  ftp.nluug.nl
>> >>  ftp.nluug.nl
>> >>  sugarlabs.cdn.cacheboy.net
>> >>
>> >> A user manually chooses between one of the above mirrors.  Rsync runs
>> >> on a cron job.  Health is monitored by Marten.
>> >>
>> >> sugarlabs.cdn.cacheboy.net
>> >>  *.cdn.cacheboy.net
>> >>
>> >> If a user choses sugarlabs.cdn.cacheboy.net , they are redirected
>> >> around the cacheboy network based on location and load.  Files are
>> >> pushed to mirrors as needed.  cacheboy monitors individual mirror's
>> >> health and removes them as necessary.
>> >>
>> >> This leads to two questions:
>> >> Is cacheboy stable enough that we could host
>> >> sugarlabs.cdn.cacheboy.net at Gnaps and alias dl.sl.o to
>> >> sugarlabs.cdn.cacheboy.net ?  This would put the entire download
>> >> infrastructure on cacheboy.
>> >>
>> >> or
>> >> Should we look into Bouncer or Fedora Mirror Manager to manage and
>> >> redirect within the first level of mirrors?
>> >>
>> >> david
>> >>
>> >>> There is no system yet to do geoip based redirections(as far as i know).
>> >>>
>> >>> Yes it should be there, IHMO it should be in some
>> >>> kind of cgi?
>> >>>
>> >>> apt-get install  libapache2-mod-geoip?
>> >>>
>> >>> I never played with it, and I don't trust myself
>> >>> to write sane cgi's. So if we have talents on board,
>> >>> it is time to use them.....
>> >>>
>> >>> Feel more than free to take over from here. I would prefeer to chase
>> >>> some more mirrors.....
>> >>>
>> >>> cheers,
>> >>> Marten
>> >>>
>> >>>>
>> >>>>
>> >>>> http://activities.sugarlabs.org/.
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Luke Faraone
>> >>>> http://luke.faraone.cc
>> >>> --
>> >>> this email is sent from my mainframe.....
>> >>> http://martenvijn.nl                Sugar on a Stick
>> >>> http://bsd.wifisoft.org/nek/        The Network Event Kit
>> >>> http://opencommunitycamp.org         26th Jul - 2nd August
>> >>>
>> >>>
>> >>
>> >
>> _______________________________________________
>> Systems mailing list
>> Systems at lists.sugarlabs.org
>> http://lists.sugarlabs.org/listinfo/systems
> --
> this email is sent from my mainframe.....
> http://martenvijn.nl                Sugar on a Stick
> http://bsd.wifisoft.org/nek/        The Network Event Kit
> http://opencommunitycamp.org         26th Jul - 2nd August
>
>