[Sugar-devel] download.sugarlabs.org still redirecting to broken mirror

[James Cameron]

Thanks for the IRC contact.  Yes, it works sometimes, and doesn't work
sometimes, and you've noticed that too.

Yeah, I did a hack too, this is what I changed:

>From 2fe58b2dbe32867e10d2994ff1919efc5a4bf17c Mon Sep 17 00:00:00 2001
From: James Cameron <quozl at laptop.org>
Date: Wed, 10 Sep 2014 16:17:21 +1000
Subject: [PATCH] use activity group caching on spurious 404

- when an activity cannot be downloaded and yet we have a copy in cache,
  use the cache as last resort,

  (caused by broken mirror configuration on activities.sugarlabs.org,
  where out of date mirrors are used, not fixed for many weeks);

- build the cache file name localpath earlier,

- remove the retrying message,

- change cache messages so that the cache file is not mentioned (it is
  derived from the URL anyway),

- remove unnecessary spaces in messages about cache.
---
 .../kspost.60.nochroot.activities.py               | 28 +++++++++++-----------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/modules/sugar_activity_group/kspost.60.nochroot.activities.py b/modules/sugar_activity_group/kspost.60.nochroot.activities.py
index d4a87a8..6b65c90 100644
--- a/modules/sugar_activity_group/kspost.60.nochroot.activities.py
+++ b/modules/sugar_activity_group/kspost.60.nochroot.activities.py
@@ -47,6 +47,7 @@ if install_activities:
         if ooblib.cacheonly:
             print >>sys.stderr, "Trying group URL cache file", grpurlcache
             if os.path.exists(grpurlcache):
+                print >>sys.stderr, "Using group URL cache file", grpurlcache
                 name, desc, results = pickle.load(open(grpurlcache))
             else:
                 continue
@@ -74,48 +75,47 @@ if install_activities:
         (version, url) = microformat.only_best_update(info)
         print >>sys.stderr, "Examining %s v%s: %s" % (name, version, url)
 
-        if ooblib.cacheonly:
-            path = urlparse.urlsplit(url)[2]
-            path = os.path.basename(path)
+        path = os.path.basename(urlparse.urlsplit(url)[2])
+        localpath = os.path.join(cache, path)
 
-            localpath = os.path.join(cache, path)
+        if ooblib.cacheonly:
             if os.path.exists(localpath):
-                print >>sys.stderr, "Using: ", localpath
+                print >>sys.stderr, "Using cache; as directed."
                 ooblib.install_sugar_bundle(localpath)
                 continue
             else:
-                print >>sys.stderr, "Cannot find cache for ", url
+                print >>sys.stderr, "Cannot find cache for", url
                 sys.exit(1)
 
         fd = None
         for attempts in range(5):
             if attempts > 0:
-                print >>sys.stderr, 'Retrying.'
                 time.sleep(1)
             try:
                 fd = urllib2.urlopen(url)
                 break
             except urllib2.HTTPError, e:
-                print >>sys.stderr, 'HTTP error: ', e.code
+                print >>sys.stderr, 'HTTP error:', e.code
             except urllib2.URLError, e:
-                print >>sys.stderr, 'Network or server error: ', e.reason
+                print >>sys.stderr, 'Network or server error:', e.reason
 
         if not fd:
-            print >>sys.stderr, 'Could not reach ', url
+            if os.path.exists(localpath):
+                print >>sys.stderr, "Using cache; as last resort."
+                ooblib.install_sugar_bundle(localpath)
+                continue
+            print >>sys.stderr, 'Could not reach', url
             sys.exit(1)
 
         headers = fd.info()
         if not 'Content-length' in headers:
             raise Exception("No content length for %s" % url)
         length = int(headers['Content-length'])
-        path = urlparse.urlsplit(fd.geturl())[2]
-        path = os.path.basename(path)
 
-        localpath = os.path.join(cache, path)
         if os.path.exists(localpath):
             localsize = os.stat(localpath).st_size
             if localsize == length:
-                print >>sys.stderr, "Not downloading, already in cache."
+                print >>sys.stderr, "Using cache; size and name match."
                 ooblib.install_sugar_bundle(localpath)
                 continue
 
-- 
1.8.3.2



On Wed, Sep 10, 2014 at 06:42:04PM +1000, Sam P. wrote:
> Hi James,
> 
> I did a kind of hack and now it is fixed :)
> 
> See the irc for more.
> 
> Sam
> 
> $ wget [1]http://download.sugarlabs.org/activities/4315/words-21.xo
> --2014-09-10 12:41:23--  [2]http://download.sugarlabs.org/activities/4315/
> words-21.xo
> Resolving [3]download.sugarlabs.org ([4]download.sugarlabs.org)...
> 208.118.235.53, 2001:4830:134:7::11
> Connecting to [5]download.sugarlabs.org ([6]download.sugarlabs.org)|
> 208.118.235.53|:80... connected.
> HTTP request sent, awaiting response... 302 Found
> Location: [7]http://mirror.aarnet.edu.au/pub/sugarlabs/activities/4315/
> words-21.xo [following]
> --2014-09-10 12:41:25--  [8]http://mirror.aarnet.edu.au/pub/sugarlabs/
> activities/4315/words-21.xo
> Resolving [9]mirror.aarnet.edu.au ([10]mirror.aarnet.edu.au)...
> 202.158.214.106, 2001:388:30bc:cafe::beef
> Connecting to [11]mirror.aarnet.edu.au ([12]mirror.aarnet.edu.au)|
> 202.158.214.106|:80... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2014-09-10 12:41:25 ERROR 404: Not Found.
> 
> --
> James Cameron
> [13]http://quozl.linux.org.au/
> _______________________________________________
> Sugar-devel mailing list
> [14]Sugar-devel at lists.sugarlabs.org
> [15]http://lists.sugarlabs.org/listinfo/sugar-devel
> 
> References:
> 
> [1] http://download.sugarlabs.org/activities/4315/words-21.xo
> [2] http://download.sugarlabs.org/activities/4315/words-21.xo
> [3] http://download.sugarlabs.org/
> [4] http://download.sugarlabs.org/
> [5] http://download.sugarlabs.org/
> [6] http://download.sugarlabs.org/
> [7] http://mirror.aarnet.edu.au/pub/sugarlabs/activities/4315/words-21.xo
> [8] http://mirror.aarnet.edu.au/pub/sugarlabs/activities/4315/words-21.xo
> [9] http://mirror.aarnet.edu.au/
> [10] http://mirror.aarnet.edu.au/
> [11] http://mirror.aarnet.edu.au/
> [12] http://mirror.aarnet.edu.au/
> [13] http://quozl.linux.org.au/
> [14] mailto:Sugar-devel at lists.sugarlabs.org
> [15] http://lists.sugarlabs.org/listinfo/sugar-devel

-- 
James Cameron
http://quozl.linux.org.au/