[Sugar-devel] Problems downloading PG titles without a web browser

[James Simmons]

Daniel,

Thanks for your tips.  I was able to add a user agent string like this:

OLPCReader/1.0; +mailto:nicestep at gmail.com

and that fixed the problem.  The code has been committed to Git and I'll
release a new .xo file shortly.

I don't know that I'll submit a path for network.py as the user agent
string is not a problem for most uses.

James Simmons


On Sun, Feb 10, 2013 at 7:05 PM, Daniel Narvaez <dwnarvaez at gmail.com> wrote:

> Actually it seems like you might be able to do it with urllib too.
>
> http://docs.python.org/2/library/urllib.html
>
> Search for urlopener, there is an example.
>
> On Monday, 11 February 2013, Daniel Narvaez wrote:
>
>>
>> http://git.sugarlabs.org/sugar-toolkit-gtk3/sugar-toolkit-gtk3/blobs/master/src/sugar3/network.py
>>
>> That code *looks* trivial to port to urllib2. You could do that, submit
>> the patch and in the meantime just copy and paste the ported code in your
>> activity (it doesn't seem to have dependencies on other toolkit stuff).
>> Well, unless it's a gtk2 activity, in that case you can skip the "submit
>> the patch" step :)
>>
>> On Monday, 11 February 2013, James Simmons wrote:
>>
>>> Marcello (and everyone else who replied):
>>>
>>> It looks like the mirrors have the same policy.  I'm currently using
>>> this to download from the URL:
>>>
>>>
>>> http://doc.sugarlabs.org/epydocs/sugar.network.GlibURLDownloader-class.html
>>>
>>> I'm not seeing a way to put a user agent in using this code.  It looks
>>> like urllib2 supports it.  I used the other because I was already using it
>>> for collaboration and it gives me a way to do a progress report on the
>>> download.  I appreciate everyone's detective work and suggestions.
>>>
>>> James Simmons
>>>
>>>
>>> On Sun, Feb 10, 2013 at 1:22 PM, Marcello Perathoner <
>>> marcello at perathoner.de> wrote:
>>>
>>>> On 02/10/2013 06:46 PM, James Simmons wrote:
>>>>
>>>>> I had created an Activity for the One Laptop Per Child project which
>>>>> downloads and reads the Plain Text version of PG titles.  I use the
>>>>> offline
>>>>> catalogue and some Python code to list available titles in the
>>>>> catalogue
>>>>> that match a search string.  For instance the child might enter
>>>>> "Twain" and
>>>>> get a list of all the books by and about Twain.  Then the child can
>>>>> download the book using the same program.
>>>>>
>>>>> This has been working well for years but lately it has stopped working.
>>>>> When I try to debug it I see it trying to download this URL for
>>>>> instance:
>>>>>
>>>>> http://www.gutenberg.org/dirs/**1/1/119/119.zip<http://www.gutenberg.org/dirs/1/1/119/119.zip>
>>>>>
>>>>
>>>>
>>>> Supply a user-agent that clearly defines your app and provides a way to
>>>> contact you eg.
>>>>
>>>>   OLPCReader/1.0; +http://www.olpc-reader.org/**app-info.html<http://www.olpc-reader.org/app-info.html>
>>>>
>>>>   OLPCReader/1.0; +mailto:me at example.com
>>>>
>>>> The standard Python-urllib user agent will not do!
>>>>
>>>>
>>>> or
>>>>
>>>>
>>>> use a mirror site. See: www.gutenberg.org/MIRRORS.ALL
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Regards
>>>>
>>>> --
>>>> Marcello Perathoner
>>>> webmaster at gutenberg.org
>>>>
>>>
>>>
>>
>> --
>> Daniel Narvaez
>>
>>
>
> --
> Daniel Narvaez
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20130214/ad90f3ad/attachment.html>