[Systems] [mirrorbrain] error with multiple ip addresses

[dfarning at sugarlabs.org]

On Fri, Nov 27, 2009 at 1:35 PM, Peter Pöml <poeml at cmdline.net> wrote:
> Hi David, hi Marten,
>
> Am 12.10.2009 um 22:56 schrieb David Farning:
>
>> Peter,
>> Here is an interesting error.  The following site resolves to multiple
>> ip addresses and causes mb new to throw and error.
>
> Sorry for late reply.

It looks like you are catching up after your vacation:)  

I am also ccing this to systems at sl.o so their is a record of this thread in the archive.

> There are two separate issues to note here.
>
>> mb new nluug.nl -H http://ftp.nluug.nl/pub/os/Linux/distr/Sugar -F
>> ftp://ftp.nluug.nl/pub/os/Linux/distr/Sugar
>> warning: 'ftp.nluug.nl' resolves to a multiple IP addresses:
>> 192.87.102.42, 192.87.102.43
>
> At first, this mirror's hostname is resolving to two addresses, which means
> that the clients use any of the addresses at random (DNS round-robin).
> That's fine for load-balancing in general, but in our case it matters
> whether the content on the two replica servers are identical. If the content
> on the two servers behind the URL differs, then it is a problem. The file
> list that the scanner sees could be different at each run, and a client that
> is redirected might run into a "file not found".
>
> Having said that, with some mirrors this doesn't pose a problem, for
> instance when the file tree is mounted from a shared network storage, so it
> is in fact a single tree. It may also not be a problem if the content is
> synced between two storage systems really really quickly and reliably (also
> depending on the frequency of changes in the tree at all). But in many cases
> that I have seen there are actually two (or more) different servers, with
> different storage systems, sometimes different admins. Synchronization can
> lag, or break.
>
> So, DNSrr is not necessarily a problem, but experience shows that it's a
> source of trouble and best avoided.
>
> Therefore, "mb new" spits out the warning that the IP of ftp.nluug.nl
> resolves to more than one address.
>
> It is best to add such a site as two separate mirrors, by using the "direct"
> hostname that resolves to the individual IP addresses. There usually is an
> individual hostname; if not, the IP itself could be used. The admins won't
> mind since the load balancing is happening anyway.
>
> Thus, you would do:
>
> mb new ftp1.nluug.nl -H 'http://ftp1.nluug.nl/pub/os/Linux/distr/Sugar' -F
> ...
> mb new ftp2.nluug.nl -H 'http://ftp2.nluug.nl/pub/os/Linux/distr/Sugar' -F
> ...
>
>
> As a consequence, the mirrors will be scanned separately, and the content of
> each be maintained in the database. For mirrors that have one actual storage
> that may be unwanted, but if that's sure (because the admins confirm this)
> then the mirror could still be entered as "one". As you did so far. That's
> why "mb new" issues only a warning.
>
> As another consequence, the priority of the mirrors should be adjusted,
> because if nluug.nl appears to the redriector as two mirrors, it'll get more
> requests, and thus it makes sense to set the priority to e.g. 70 instead of
> 100. (The best number will depend on the total number on mirrors.)
>
> In the case of nluug.nl, I had them as two mirrors in the openSUSE database
> because there was actually a case where inconsistencies had been seen (which
> is not to blame nluug.nl -- such things happen). In fact, in the beginning
> no problem was anticipated, because the two boxes were a cluster with shared
> storage. (And, from my notes, there was also ftp.surfnet.nl running from the
> same storage.) But eventually problems did occur, and inconsistencies lead
> to clients getting 403s (forbidden) when they reached a particular server.
> This shows that difficulties are not necessarily in differences in the file
> tree. I ended up scanning and monitoring each server individually; no
> problem since then.
>
> The fact that mirror monitoring takes place for each server is an additional
> advantage. DNSrr itself doesn't deal with failing hosts; DNS happily keeps
> resolving to dead addresses.

Thanks for the complete answer.

> Taking this further, there is another level at which similar precautions are
> in order: if a mirror uses GeoDNS instead of DNSrr. GeoDNS is not visible,
> but depending from where you look (resolve an address) you see something
> different. It's important to know about it - because the individual servers
> are far away from each other in this scenario, and a synchronisation lag (or
> major setup differences) are rather the rule than the exception. The
> treatment is according to the same principle: access each mirror separately
> under its own address, thus bypassing GeoDNS. (kernel.org is the only mirror
> doing this that I'm aware of.)

I think ibiblio might be doing something similar.  It seems that they have a network of decentralized mirrors behind the public mirrors.ibiblio.org name.  I didn't think too much about it because it just works.

> (This is information that needs to appear in the documentation, eventually.
> As a quick workaround, I'll add a link to this posting to the warning
> message.)
>
>
> Now to the second issue. In your case, the warning was a bit concealed,
> because it was closely followed by the traceback of a different problem:
>
>
>> Traceback (most recent call last):
>> [...]
>> psycopg2.IntegrityError: duplicate key value violates unique
>> constraint "server_identifier_key"
>
> What happened here is that a mirror with the identifier "nluug.nl", the same
> as you tried to create, already existed in the database. The identifier of
> mirrors must be unique, so the database vetoes (the database schema enforces
> it).
>
> I am fixing this in svn, so now there is a proper error message reported in
> this case:
>
>
>  # mb new nluug.nl -H 'http://ftp.nluug.nl/pub/os/Linux/distr/Sugar'
> Error: a mirror's identifier must be unique.
> There is already a mirror using this identifier. See output of `mb show
> nluug.nl`.
> Exiting.

Thanks, that is much clearer.

> Thank you very much for the report!

Now that you are back, I have a few more built up.... I didn't want your inbox to appear too over whelming when go back to it:)

david

> Peter
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 271 bytes
Desc: OpenPGP digital signature
Url : http://lists.sugarlabs.org/private/systems/attachments/20091128/ce47b1a5/attachment-0001.pgp