[Systems] mirror testing request

[Marten Vijn]

On Tue, 2009-09-22 at 16:15 +0800, Adrian Chadd wrote:
> The point behind cacheboy is to try and fix this preponderance of
> differently run mirror systems, with different content distribution
> methods, most of which don't really have the network topology or
> network utilisation in consideration.
> 
> The plan is to be able to put well connected nodes in places with lots
> of peering but small and/or expensive amounts of transit (eg, places
> like africa, south america, eastern europe, Australia and New Zealand
> :) and have all the projects take advantage of it.
> 
Fully agree with you in a technical way, from a organizational
perspective CacheBoy seem to be run by only you (or are more people
involved?)

It is not a case trust, but over time monolithic systems do pass. If
there is no way to take over the infrastructure when something happens
to you,  (girlfriend/kids/health issues may happen, and ofcourse I wish
you are fine) I would not want to relay on CacheBoy only.

I think a dependency in one person,server,uplink is not wise. Therefore
my idea to have multiple mirror systems is one way of creating more
redundancy, but the only. 

A way the harden CachyBoy would having multiple admins that work for
CacheBoy. And CacheBoy would become an organization over time
(foundation or something).

2ct,
Marten 



> 
> 
> adrian
> 
> 2009/9/22 Marten Vijn <info at martenvijn.nl>:
> > On Mon, 2009-09-21 at 20:20 -0500, David Farning wrote:
> >> On Mon, Sep 21, 2009 at 7:52 PM, Adrian Chadd <adrian.chadd at gmail.com> wrote:
> >> > I'd be happy if Cacheboy was handling all of your downloads. My
> >> > distribution system is slightly more flexible than bouncer and is
> >> > network topology aware.
> >>
> >> This takes us back to the questions.
> >> Is cacheboy stable enough that we could host the entire infrastructure on it.
> >> What would we need to do to run sugarlabs.cdn.cacheboy.net at our
> >> server facility?
> >
> > Space on other mirrors do need enough disk space,
> > atm it really should stay under 20G, not breaking current
> > agreements.
> >
> > And, the other question would what more mirror systems over time?
> >
> > The is still possibility at Internet Systems Consortium (ISC) to mirror
> > a 2G subset. Here is we should decide quick, to use the moment of
> > opportinuty.  (third time I mention it)
> >
> > My idea would to get on a fair amount of mirror systems, and find a way
> > the balance the load.
> >
> > 2ct,
> >
> > Marten
> >
> >>
> >> I remember you saying that cacheboy required root on mirror nodes
> >> because it runs as a cluster.
> >>
> >> david
> >>
> >>
> >>
> >> > Adrian
> >> >
> >> > 2009/9/22 David Farning <dfarning at sugarlabs.org>:
> >> >> On Mon, Sep 21, 2009 at 3:54 PM, Marten Vijn <info at martenvijn.nl> wrote:
> >> >>> On Mon, 2009-09-21 at 16:35 -0400, Luke Faraone wrote:
> >> >>>> On Mon, Sep 21, 2009 at 16:29, Marten Vijn <info at martenvijn.nl> wrote:
> >> >>>>         What is A.SL.O ?
> >> >>>>
> >> >>>
> >> >>> ah clear, these links do not point to download as far i can see or are
> >> >>> redicted.
> >> >>
> >> >> Currently a.sl.o downloads point to activities.sugarlabs.org.  We can
> >> >> set it up to point to dl.sl.org if there is an advantage.
> >> >>
> >> >>> Anyway, currently only:
> >> >>>
> >> >>> http://download.sugarlabs.org/ is mirrored. to all mirros.
> >> >>>
> >> >>> So if there is copy of activities@ on download@ it will be
> >> >>> picked up soon 1-24 hrs.
> >> >>
> >> >> Ok got it.  Mozilla uses a system called bouncer to control its mirror
> >> >> net work.  Whenever a file is written to downloads.mozilla.org.
> >> >> Bouncer starts immediately propagating the file across the mirror
> >> >> system.  Once bouncer has verified that a particular mirror has a good
> >> >> copy of the file, it starts redirecting requests to that mirror.
> >> >>
> >> >> It seems that that is similar to how things work within cacheboy.  But
> >> >> we currently have a manual step of rsyncing from dl.sl.o to cacheboy,
> >> >> dl2.sl.o, and the european mirrors you have set up.  This gives us a
> >> >> two tier mirror system
> >> >>
> >> >> dl.sl.o
> >> >>  dl2.sl.o
> >> >>  ftp.nluug.nl
> >> >>  ftp.nluug.nl
> >> >>  sugarlabs.cdn.cacheboy.net
> >> >>
> >> >> A user manually chooses between one of the above mirrors.  Rsync runs
> >> >> on a cron job.  Health is monitored by Marten.
> >> >>
> >> >> sugarlabs.cdn.cacheboy.net
> >> >>  *.cdn.cacheboy.net
> >> >>
> >> >> If a user choses sugarlabs.cdn.cacheboy.net , they are redirected
> >> >> around the cacheboy network based on location and load.  Files are
> >> >> pushed to mirrors as needed.  cacheboy monitors individual mirror's
> >> >> health and removes them as necessary.
> >> >>
> >> >> This leads to two questions:
> >> >> Is cacheboy stable enough that we could host
> >> >> sugarlabs.cdn.cacheboy.net at Gnaps and alias dl.sl.o to
> >> >> sugarlabs.cdn.cacheboy.net ?  This would put the entire download
> >> >> infrastructure on cacheboy.
> >> >>
> >> >> or
> >> >> Should we look into Bouncer or Fedora Mirror Manager to manage and
> >> >> redirect within the first level of mirrors?
> >> >>
> >> >> david
> >> >>
> >> >>> There is no system yet to do geoip based redirections(as far as i know).
> >> >>>
> >> >>> Yes it should be there, IHMO it should be in some
> >> >>> kind of cgi?
> >> >>>
> >> >>> apt-get install  libapache2-mod-geoip?
> >> >>>
> >> >>> I never played with it, and I don't trust myself
> >> >>> to write sane cgi's. So if we have talents on board,
> >> >>> it is time to use them.....
> >> >>>
> >> >>> Feel more than free to take over from here. I would prefeer to chase
> >> >>> some more mirrors.....
> >> >>>
> >> >>> cheers,
> >> >>> Marten
> >> >>>
> >> >>>>
> >> >>>>
> >> >>>> http://activities.sugarlabs.org/.
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Luke Faraone
> >> >>>> http://luke.faraone.cc
> >> >>> --
> >> >>> this email is sent from my mainframe.....
> >> >>> http://martenvijn.nl                Sugar on a Stick
> >> >>> http://bsd.wifisoft.org/nek/        The Network Event Kit
> >> >>> http://opencommunitycamp.org         26th Jul - 2nd August
> >> >>>
> >> >>>
> >> >>
> >> >
> >> _______________________________________________
> >> Systems mailing list
> >> Systems at lists.sugarlabs.org
> >> http://lists.sugarlabs.org/listinfo/systems
> > --
> > this email is sent from my mainframe.....
> > http://martenvijn.nl                Sugar on a Stick
> > http://bsd.wifisoft.org/nek/        The Network Event Kit
> > http://opencommunitycamp.org         26th Jul - 2nd August
> >
> >
-- 
this email is sent from my mainframe..... 
http://martenvijn.nl                Sugar on a Stick
http://bsd.wifisoft.org/nek/        The Network Event Kit
http://opencommunitycamp.org         26th Jul - 2nd August