[IAEP] [XSCE] RE: [UKids] Re: [support-gang] Taking, OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

[Tim Moody]

The idea of pre-rendering some of the tiles and doing the rest on the fly has always appealed to me too.  In fact I thought this is what IIAB was doing.  But I now think that in order to render anything you need the 500+G database, which Braddock and now Anish have put on the large rendering machine, but which seems to me like a lot of space for the schoolserver itself.

In terms of the efficiency of the tile rendering machine itself there are probably improvements that could be made.  I don't know why we need all the data in ssd and not just the region of interest, but I think Anish said it took 1 1/2 weeks just to load the map data into the database, before any rendering.

I have wondered if a distributed model is possible, like the seti and other projects, where people could sign up to have their machines do a small portion of the rendering from a small portion of the data with perhaps btsync to distribute the results.  This would take some work both on the software and the organizational side.

In terms of a disconnected client, the maps likely have to be tiles as we don't expect client machines to have a spare 500G of storage.  There are ajax client-side apps that stitch together tiles (I think slippy map is one), but they would have to be coaxed into caching tiles on the client and using that cache when disconnected.

-----Original Message-----
From: unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com] On Behalf Of Tony Anderson
Sent: Monday, June 15, 2015 5:43 AM
To: xsce-devel at googlegroups.com; unleashkids at googlegroups.com; gagnonje at gmail.com; Jaakko Helleranta; server-devel; IAEP SugarLabs; iaep at lists.sugarlabs.org; Community Support Volunteers -- who help respond to help AT laptop. org; gagnonje at gmail.com; Jaakko Helleranta; server-devel aep; iaep at lists.sugarlabs.org; Community Support Volunteers -- who help respond to help AT laptop. org; Nick Doiron
Subject: Re: Re: [IAEP] [XSCE] RE: [UKids] Re: [support-gang] Taking, OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

There seem to be two independent problems.


First, make an updated version of the IIAB OSM. The goal is to understand what are the computer requirements to do this and how long it takes and then be able to make a new version whenever needed.

Second, make available more detail than IIAB OSM provides for a region selected by a deployment while allowing the user (learner) to display, edit, and save it as a new map from this data.

On the first problem, I am confused. I would guess the job is cpu intensive. However, many comments about SSD suggest it is i/o limited. 
In a former life, I
worked on parallel programming in which the technical problem was to overlap disk i/o with computing. One thread bringing the data for the next case into memory while the processor worked on the current case. 
The trick was to determine the size of data needed to keep the processor busy and not waiting on i/o. The goal was to overlap i/o with processing to minimize processor idle time.

I assume the processing of the world can be broken down into independent pieces (e.g. one tile at the global zoom level). The experiment with Nepal should show the processing time required as well as the disk i/o time.

Based on that information, the work plan would be to load the global data on hard drive and then set up double buffered pipelines from hard disk to SSD to memory which can keep the processor busy.

On the second, it is not clear that the greater detail needs to be OSM tiles. Something like Nick's map.activity/mappack with the map data on the school server (separate from OSM) and a client application (Sugar or Sugar-web or html) accessing the data, displaying it, and enabling the learner to add to it (à la GIS). After discovering that umapper is not umap, umap is clearly open source and could be a good base for the client application.

This application should be able to get data from the school server and, as always, it should be possible for the learner to do meaningful work when not connected to the schoolserver or internet. Thia means knowing what data the learner needs to display and edit a map locally (even though the local store may be as low as 1GB) and how to specify that data to be downloaded to the Journal when connected.

Tony

On 06/15/2015 08:24 AM, iaep-request at lists.sugarlabs.org wrote:
> Send IAEP mailing list submissions to
> 	iaep at lists.sugarlabs.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://lists.sugarlabs.org/listinfo/iaep
> or, via email, send a message with subject or body 'help' to
> 	iaep-request at lists.sugarlabs.org
>
> You can reach the person managing the list at
> 	iaep-owner at lists.sugarlabs.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of IAEP digest..."
>
>
> Today's Topics:
>
>     1. Re: [XSCE] RE: [UKids] Re: [support-gang] Taking
>        OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT /
>        2PM UTC (Nick Doiron)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 14 Jun 2015 23:24:11 -0700
> From: Nick Doiron <ndoiron at mapmeld.com>
> To: xsce-devel at googlegroups.com
> Cc: Internet In a Box Working Group <iiab at sgvhak.org>,	Unleash Kids!
> 	<unleashkids at googlegroups.com>, J?r?me Gagnon-Voyer
> 	<gagnonje at gmail.com>,	Jaakko Helleranta <jaakko at helleranta.com>,
> 	server-devel <server-devel at lists.laptop.org>,	iaep
> 	<iaep at lists.sugarlabs.org>,	"Community Support Volunteers -- who help
> 	respond to help AT	laptop.org" <support-gang at lists.laptop.org>
> Subject: Re: [IAEP] [XSCE] RE: [UKids] Re: [support-gang] Taking
> 	OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
> Message-ID:
> 	<CALG_m-VXbHFM0OvcturwL-hvnU5YCdWPFsrHu6JH5C1vVtq0Yg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> I don't think there's any technical issues with rendering the world at 10
> and specific countries at 16, other than the human knowing where they can
> and cannot zoom
>
> -- Nick
>
> On Sun, Jun 14, 2015 at 11:58 AM, Tim Moody <tim at timmoody.com> wrote:
>
>> couple of observations:
>>
>>
>>
>> As expected, the new tiles have a lot more detail.
>>
>>
>>
>> There are more levels of zoom in the new ones.
>>
>>
>>
>> Some names have changed - the old map had Lalitpur and the new one has
>> Patan (both are used)
>>
>>
>>
>> I don't see any boxes for unprintable characters, but there is a lot less
>> Devanagari. (Google maps has more)
>>
>>
>>
>> Is it possible to merge individually generated regional tiles?  for
>> example if you rendered India and Nepal separately would you get both?
>>
>>
>>
>> What happens if you render the world at level 10 and then specific
>> countries at 16?
>>
>>
>>
>> *From:* xsce-devel at googlegroups.com [mailto:xsce-devel at googlegroups.com] *On
>> Behalf Of *Anish Mangal
>> *Sent:* Saturday, June 13, 2015 10:06 PM
>> *To:* J?r?me Gagnon-Voyer
>> *Cc:* xsce-devel; Community Support Volunteers -- who help respond to
>> help AT laptop.org; Unleash Kids!; server-devel; iaep; Internet In a Box
>> Working Group; Jaakko Helleranta
>> *Subject:* Re: [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap
>> Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>>
>>
>>
>> Hi,
>>
>> Following from the skype call this week, I uploaded Nepal's OSM data in
>> postgres and render tiles to see performance, and as expected, everything
>> is blazing fast.
>>
>> The pbf file went into the postgress in less than 5 minutes, and tiles are
>> being rendered pretty fast as well.
>>
>> You can check it yourself by going to
>> http://home.braddock.com:28112/osm/slippymap.html
>>
>> * Select Mapnik
>>
>> * Zoom out and center over Nepal
>>
>> * Select Local tiles
>>
>> * Zoom in
>>
>> I don't know what this "proves" as the bottleneck still is doing this for
>> the entire planet, which we need to figure out a way for anyway.
>>
>> Best,
>>
>> Anish
>>
>>
>>
>>
>>
>> On Sun, Jun 14, 2015 at 6:02 AM, Anish Mangal <anishmg at umich.edu> wrote:
>>
>> Hi J?r?me,
>>
>> I incorporated one of your comments; as for the other I think Timm, Nick
>> would be better suited for the discussion (python backend v/s frontend js).
>> :-)
>>
>> Best,
>>
>> Anish
>>
>>
>>
>> On Sun, Jun 14, 2015 at 3:22 AM, J?r?me Gagnon-Voyer <gagnonje at gmail.com>
>> wrote:
>>
>> Hi Anish
>>
>>
>>
>> Great document. I've added 2 comments to the document, feel free to
>> incorporate into the document if that makes sense.
>>
>>
>>
>> Also added a TODO for myself to do more research about the various search
>> solutions.
>>
>> I've used Nominatim in the past (on the client side, not the server
>> infrastructure) and it was overall very good, but I'll want to know more
>> about the other ones.
>>
>>
>>
>> On Sat, Jun 13, 2015 at 12:51 PM, Tim Moody <tim at timmoody.com> wrote:
>>
>> Thanks, Anish for an excellent start at getting this down on paper (so to
>> speak).  I think this covers things pretty well and gives us the necessary
>> hooks on which to hang the details as we begin fleshing out solutions to
>> the requirements you documented.
>>
>>
>>
>> Under issues I added one point about rendering non-Roman character sets.
>>
>>
>>
>> *From:* unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com]
>> *On Behalf Of *Anish Mangal
>> *Sent:* Saturday, June 13, 2015 12:15 PM
>> *To:* Community Support Volunteers -- who help respond to help AT
>> laptop.org
>> *Cc:* Unleash Kids!; server-devel; xsce-devel; iaep; Internet In a Box
>> Working Group; Jaakko Helleranta
>> *Subject:* [UKids] Re: [support-gang] Taking OpenStreetMap Offline -
>> DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>>
>>
>>
>> Okay, so I tried to encapsulate whatever we discussed into a design
>> document, which can hopefully serve as a base for reaching out to the OSM
>> community and to better organize our own thoughts.
>>
>>
>> https://docs.google.com/document/d/1LWsN-cPH3lvMuXS-f0Tk8IWVh-3X808WoIt0OJ-QNt8/edit#
>> <https://docs.google.com/document/d/1LWsN-cPH3lvMuXS-f0Tk8IWVh-3X808WoIt0OJ-QNt8/edit>
>>
>> Please feel free to edit the information there as I may have missed some
>> points, or interpreted them differently than as intended.
>>
>> Best,
>>
>> Anish
>>
>>
>>
>> On Fri, Jun 5, 2015 at 12:00 AM, Adam Holt <holt at laptop.org> wrote:
>>
>> Who's attending http://stateofthemap.us at the UN in NYC this weekend?
>>
>> Who's most serious about bringing OpenStreetMap's opportunities and rapid
>> progress into the hands of the world's offline poor -- kids and all,
>> American and Swahili?  Wanna Take This Map Outside, offline and off the
>> grid, where we all belong/began?
>>
>> http://Internet-in-a-Box.org and similar efforts have made a gigantic 1st
>> step: in Ghana and Rwanda I could never have imagined better reviews to our
>> 2015 deployments begun there in recent months.  Many are now changing the
>> game increasingly for the Bottom BillionS, among many who've literally
>> never seen a globe before, nevermind a map of their own towns.  Both in
>> OLPC (school) contexts, in libraries, on Nepali hillsides where folks don't
>> have proper homes, and far beyond --- perspectives (literally) are about to
>> change.
>>
>> But modern phones today contain more than enough gigabytes to display ALL
>> OpenStreetMap map detail within most countries, and yet they do not yet,
>> WHY?  Where are the Khan Academies and KA Lites of OSM (OpenStreetMap)
>> bringing new classroom/journalistic rubrics, freeing everyone's "brain
>> software" to explore and document our own communities in our own languages?
>>
>> *Who Will Take The Next Steps?*
>> *What engineering middleware, distribution vectors, community/economic
>> models, and final field packagings will get us all there & beyond?*
>>
>> *Will offline edit-contributions prove impossible, much like with
>> Wikipedia in Peru, when offline kids edit overly stale
>> OpenStreetMap/Wikipedia images a year before/later?*
>>
>>
>>
>> As such the wider OLPC community is hosting a DESIGN Call to bring forward
>> ideas across the OpenStreetMap landscape, fertilizing our immediate work
>> with school server projects like http://schoolserver.org, http://xsce.org,
>> https://en.opensuse.org/openSUSE:Education-Li-f-e ETC.  Ministries of
>> Education in India elsewhere are watching closely, expressly eager to help
>> if we can point the way.  All giving our "2020 Vision" questions very
>> practical and immediate "customers" well before 2020, much like Garmin GPS
>> units fed a wonderful ecosystem of geo-specific "gmapsupp.img" offline map
>> files over the past decade:
>> http://wiki.openstreetmap.org/wiki/OSM_Map_On_Garmin/Download
>>
>> That decade's now done: what IS our framework for the coming decade?
>> Please join us if you have strategic/partnership ideas towards making
>> Offline OSM Designs happen, *Thank You !!*
>>
>>
>>
>> 10AM New York Time / 2PM UTC
>>
>> Thursday, June 11th
>>
>> RSVP with your Skype username, if we are more than 15-20 we'll use an
>> industrial conference call system instead!
>>
>>
>> CONCLUSION: Could OpenStreetMap be the very ultimate in Constructionist
>> Learning Projects, replacing OLPC in coming years, on a quickly
>> shrinking/endangered but still green-in-parts planet?  Regardless, how to
>> build the OLPC Movement's offline/civic learning successes, consciously
>> learning from its community infrastructural mistakes?  Nick Doiron (
>> http://mapmeld.com) and Anish Mangal (
>> https://in.linkedin.com/in/anishmangal) who've spent years thinking about
>> this topic will lead the discussion, agenda is entirely yours!  OPTIONAL:
>> submit agenda items in advance right here:
>>
>>
>> https://docs.google.com/document/d/1o6QtzLb6e58YKWqMf_junux2XyBRLFm31un8YLcYslg
>>
>>
>>
>> *< forward invitation to Twitter/wherever as appropriate >*
>>
>>
>> --
>>
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>>
>>
>> _______________________________________________
>> support-gang mailing list
>> support-gang at lists.laptop.org
>> http://lists.laptop.org/listinfo/support-gang
>>
>>
>>
>> --
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Unleash Kids" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to unleashkids+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>>
>> --
>>
>> J?r?me Gagnon-Voyer
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.sugarlabs.org/archive/iaep/attachments/20150614/58b5ae9c/attachment.html>
>
> ------------------------------
>
> _______________________________________________
> IAEP -- It's An Education Project (not a laptop project!)
> IAEP at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/iaep
>
> End of IAEP Digest, Vol 87, Issue 11
> ************************************
> .
>

-- 
Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
--- 
You received this message because you are subscribed to the Google Groups "Unleash Kids" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unleashkids+unsubscribe at googlegroups.com.
For more options, visit https://groups.google.com/d/optout.