[IAEP] [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC

[Anish Mangal]

I guess a VM using a single core and 512 MB or 1GB RAM would be a
reasonable starting point. A Rpi-2 would be great as well. You could just
test from the latest master branch, and it *should* work. Currently, a role
called 'Pathagar' is broken, so you may have to disable installing it in
vars/default_vars.yml when you give it a go.

Is using a VM, I often tend to use vagrant. You can use a fedora 21/22
minimal or centos minimal image.

This is the image I used for testing my XSCE install, where things went
smoothly -
https://github.com/holms/vagrant-centos7-box/releases/download/7.1.1503.001/CentOS-7.1.1503-x86_64-netboot.box

If using centOS you will need to enable the epel repo - *yum install
epel-release*

Hope this helps.

Best,
Anish


On Sun, Jun 21, 2015 at 7:44 AM, Jérôme Gagnon-Voyer <gagnonje at gmail.com>
wrote:

> Hi all
>
> I wanted to do some testing of the search solutions on some kind of
> hardware/software that would look similar to what will be deployed in the
> field (to measure the performance impact).
>
> Got this idea from this post where :
> http://www.openstreetmap.org/user/SimonPoole/diary/34857
>
> Should I be testing with XSCE 5.1? XSCE 5.5 beta? Should I get some
> hardware like a Pi2, or just get some VM? Any VM recommendation?
>
> Thanks
>
> On Wed, Jun 17, 2015 at 3:15 PM, Tim Moody <tim at timmoody.com> wrote:
>
>> Thanks, this is very useful information.  I didn't realize that slippy
>> maps is a strategy and leaflet is a library for it that OSM itself uses.
>>
>>
>>
>> A few questions.
>>
>>
>>
>> Which of the filtered datasets did you use for populated places?
>>
>>
>>
>> Are they from
>> http://wiki.openstreetmap.org/wiki/Planet.osm#Country_and_area_extracts ?
>>
>>
>>
>> Is the directory structure such that separately rendered tiles for
>> different regions can be merged by over-copying?
>>
>>
>>
>> I did some sampling.
>>
>>
>>
>> In Manhattan the maximum zoom is
>> http://xsce/iiab/maps/tile/15/9649/12318.png so level 15 (16th)
>>
>>
>>
>> In Northern Ontario the maximum zoom is
>> http://xsce/iiab/maps/tile/13/2048/2704.png level 13 (14th)
>>
>>
>>
>> In the middle of the North Pacific
>> http://xsce/iiab/maps/tile/14/1088/6190.png a surprising level 14
>>
>>
>>
>> Moving slightly left I get http://xsce/iiab/maps/tile/14/1080/6191.png
>> so I'm not quite clear on the generic blue tile point.
>>
>>
>>
>> I'm also not clear on the point about the meta tiles.  From what I read
>> in http://wiki.openstreetmap.org/wiki/Meta_tiles , meta files just
>> concatenate the smaller tiles, so I'm not clear on how that saves space.  I
>> do understand that it greatly reduces the number of nodes in the file
>> system making retrieving them viable.
>>
>>
>>
>> I looked at space usage by zoom level on the current IIAB OSM:
>>
>>
>>
>> Level
>>
>> Size (in K)
>>
>> Cumulative (in K)
>>
>> Cumulative (in G)
>>
>> 0
>>
>> 60
>>
>> 60
>>
>> 0
>>
>> 1
>>
>> 84
>>
>> 144
>>
>> 0
>>
>> 2
>>
>> 152
>>
>> 296
>>
>> 0
>>
>> 3
>>
>> 300
>>
>> 596
>>
>> 0
>>
>> 4
>>
>> 1,004
>>
>> 1,600
>>
>> 0
>>
>> 5
>>
>> 2,800
>>
>> 4,400
>>
>> 0
>>
>> 6
>>
>> 9,064
>>
>> 13,464
>>
>> 0
>>
>> 7
>>
>> 26,328
>>
>> 39,792
>>
>> 0
>>
>> 8
>>
>> 86,280
>>
>> 126,072
>>
>> 0
>>
>> 9
>>
>> 255,576
>>
>> 381,648
>>
>> 0
>>
>> 10
>>
>> 859,540
>>
>> 1,241,188
>>
>> 1
>>
>> 11
>>
>> 2,233,136
>>
>> 3,474,324
>>
>> 3
>>
>> 12
>>
>> 5,905,172
>>
>> 9,379,496
>>
>> 9
>>
>> 13
>>
>> 17,480,260
>>
>> 26,859,756
>>
>> 27
>>
>> 14
>>
>> 43,947,380
>>
>> 70,807,136
>>
>> 71
>>
>> 15
>>
>> 35,454,488
>>
>> 106,261,624
>>
>> 106
>>
>>
>>
>> So 10G gets 13 levels.
>>
>>
>>
>>
>>
>> *From:* unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com]
>> *On Behalf Of *Braddock Gaskill
>> *Sent:* Monday, June 15, 2015 8:39 PM
>> *To:* Internet In a Box Working Group
>> *Cc:* xsce-devel; Unleash Kids!; Jaakko Helleranta; server-devel; iaep;
>> Community Support Volunteers -- who help respond to help AT laptop.org
>> *Subject:* Re: [IIAB] [XSCE] RE: [UKids] Re: [support-gang] Taking
>> OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>>
>>
>>
>> In IIAB we use a dataset of populated places to render at the highest
>> resolution level, and render the rest of the world at a slightly lower
>> resolution.  Our highest resolution is high enough to see individual
>> buildings.  Our total dataset of rendered tiles for the world with detailed
>> populated places coverage is 100GB.
>>
>>
>>
>> Another optimization we made for IIAB that contributed considerable
>> dataset size reduction and helped with filesystem efficiency - we use the
>> OSM "meta" tiles, which normally store 64 tile png images, but we modify
>> them to store 4096 png images per file, and we made an optimization so that
>> identical tiles are removed (think of all the blue of the ocean cover 75%
>> of the earth).
>>
>>
>>
>> We use Leaflet as our javascript web front end, and it works quite well.
>>
>>
>>
>> -braddock
>>
>>
>>
>>
>>
>> On Sun, Jun 14, 2015 at 11:24 PM, Nick Doiron <ndoiron at mapmeld.com>
>> wrote:
>>
>> I don't think there's any technical issues with rendering the world at 10
>> and specific countries at 16, other than the human knowing where they can
>> and cannot zoom
>>
>>
>>
>> -- Nick
>>
>>
>>
>> On Sun, Jun 14, 2015 at 11:58 AM, Tim Moody <tim at timmoody.com> wrote:
>>
>> couple of observations:
>>
>>
>>
>> As expected, the new tiles have a lot more detail.
>>
>>
>>
>> There are more levels of zoom in the new ones.
>>
>>
>>
>> Some names have changed - the old map had Lalitpur and the new one has
>> Patan (both are used)
>>
>>
>>
>> I don't see any boxes for unprintable characters, but there is a lot less
>> Devanagari. (Google maps has more)
>>
>>
>>
>> Is it possible to merge individually generated regional tiles?  for
>> example if you rendered India and Nepal separately would you get both?
>>
>>
>>
>> What happens if you render the world at level 10 and then specific
>> countries at 16?
>>
>>
>>
>> *From:* xsce-devel at googlegroups.com [mailto:xsce-devel at googlegroups.com] *On
>> Behalf Of *Anish Mangal
>> *Sent:* Saturday, June 13, 2015 10:06 PM
>> *To:* Jérôme Gagnon-Voyer
>> *Cc:* xsce-devel; Community Support Volunteers -- who help respond to
>> help AT laptop.org; Unleash Kids!; server-devel; iaep; Internet In a Box
>> Working Group; Jaakko Helleranta
>> *Subject:* Re: [XSCE] RE: [UKids] Re: [support-gang] Taking
>> OpenStreetMap Offline - DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>>
>>
>>
>> Hi,
>>
>> Following from the skype call this week, I uploaded Nepal's OSM data in
>> postgres and render tiles to see performance, and as expected, everything
>> is blazing fast.
>>
>> The pbf file went into the postgress in less than 5 minutes, and tiles
>> are being rendered pretty fast as well.
>>
>> You can check it yourself by going to
>> http://home.braddock.com:28112/osm/slippymap.html
>>
>> * Select Mapnik
>>
>> * Zoom out and center over Nepal
>>
>> * Select Local tiles
>>
>> * Zoom in
>>
>> I don't know what this "proves" as the bottleneck still is doing this for
>> the entire planet, which we need to figure out a way for anyway.
>>
>> Best,
>>
>> Anish
>>
>>
>>
>>
>>
>> On Sun, Jun 14, 2015 at 6:02 AM, Anish Mangal <anishmg at umich.edu> wrote:
>>
>> Hi Jérôme,
>>
>> I incorporated one of your comments; as for the other I think Timm, Nick
>> would be better suited for the discussion (python backend v/s frontend js).
>> :-)
>>
>> Best,
>>
>> Anish
>>
>>
>>
>> On Sun, Jun 14, 2015 at 3:22 AM, Jérôme Gagnon-Voyer <gagnonje at gmail.com>
>> wrote:
>>
>> Hi Anish
>>
>>
>>
>> Great document. I've added 2 comments to the document, feel free to
>> incorporate into the document if that makes sense.
>>
>>
>>
>> Also added a TODO for myself to do more research about the various search
>> solutions.
>>
>> I've used Nominatim in the past (on the client side, not the server
>> infrastructure) and it was overall very good, but I'll want to know more
>> about the other ones.
>>
>>
>>
>> On Sat, Jun 13, 2015 at 12:51 PM, Tim Moody <tim at timmoody.com> wrote:
>>
>> Thanks, Anish for an excellent start at getting this down on paper (so to
>> speak).  I think this covers things pretty well and gives us the necessary
>> hooks on which to hang the details as we begin fleshing out solutions to
>> the requirements you documented.
>>
>>
>>
>> Under issues I added one point about rendering non-Roman character sets.
>>
>>
>>
>> *From:* unleashkids at googlegroups.com [mailto:unleashkids at googlegroups.com]
>> *On Behalf Of *Anish Mangal
>> *Sent:* Saturday, June 13, 2015 12:15 PM
>> *To:* Community Support Volunteers -- who help respond to help AT
>> laptop.org
>> *Cc:* Unleash Kids!; server-devel; xsce-devel; iaep; Internet In a Box
>> Working Group; Jaakko Helleranta
>> *Subject:* [UKids] Re: [support-gang] Taking OpenStreetMap Offline -
>> DESIGN Call - Thur June 11, 10AM EDT / 2PM UTC
>>
>>
>>
>> Okay, so I tried to encapsulate whatever we discussed into a design
>> document, which can hopefully serve as a base for reaching out to the OSM
>> community and to better organize our own thoughts.
>>
>>
>> https://docs.google.com/document/d/1LWsN-cPH3lvMuXS-f0Tk8IWVh-3X808WoIt0OJ-QNt8/edit#
>> <https://docs.google.com/document/d/1LWsN-cPH3lvMuXS-f0Tk8IWVh-3X808WoIt0OJ-QNt8/edit>
>>
>> Please feel free to edit the information there as I may have missed some
>> points, or interpreted them differently than as intended.
>>
>> Best,
>>
>> Anish
>>
>>
>>
>> On Fri, Jun 5, 2015 at 12:00 AM, Adam Holt <holt at laptop.org> wrote:
>>
>> Who's attending http://stateofthemap.us at the UN in NYC this weekend?
>>
>> Who's most serious about bringing OpenStreetMap's opportunities and rapid
>> progress into the hands of the world's offline poor -- kids and all,
>> American and Swahili?  Wanna Take This Map Outside, offline and off the
>> grid, where we all belong/began?
>>
>> http://Internet-in-a-Box.org and similar efforts have made a gigantic
>> 1st step: in Ghana and Rwanda I could never have imagined better reviews to
>> our 2015 deployments begun there in recent months.  Many are now changing
>> the game increasingly for the Bottom BillionS, among many who've literally
>> never seen a globe before, nevermind a map of their own towns.  Both in
>> OLPC (school) contexts, in libraries, on Nepali hillsides where folks don't
>> have proper homes, and far beyond --- perspectives (literally) are about to
>> change.
>>
>> But modern phones today contain more than enough gigabytes to display ALL
>> OpenStreetMap map detail within most countries, and yet they do not yet,
>> WHY?  Where are the Khan Academies and KA Lites of OSM (OpenStreetMap)
>> bringing new classroom/journalistic rubrics, freeing everyone's "brain
>> software" to explore and document our own communities in our own languages?
>>
>> *Who Will Take The Next Steps?*
>> *What engineering middleware, distribution vectors, community/economic
>> models, and final field packagings will get us all there & beyond?*
>>
>> *Will offline edit-contributions prove impossible, much like with
>> Wikipedia in Peru, when offline kids edit overly stale
>> OpenStreetMap/Wikipedia images a year before/later?*
>>
>>
>>
>> As such the wider OLPC community is hosting a DESIGN Call to bring
>> forward ideas across the OpenStreetMap landscape, fertilizing our immediate
>> work with school server projects like http://schoolserver.org,
>> http://xsce.org, https://en.opensuse.org/openSUSE:Education-Li-f-e ETC.
>> Ministries of Education in India elsewhere are watching closely, expressly
>> eager to help if we can point the way.  All giving our "2020 Vision"
>> questions very practical and immediate "customers" well before 2020, much
>> like Garmin GPS units fed a wonderful ecosystem of geo-specific
>> "gmapsupp.img" offline map files over the past decade:
>> http://wiki.openstreetmap.org/wiki/OSM_Map_On_Garmin/Download
>>
>> That decade's now done: what IS our framework for the coming decade?
>> Please join us if you have strategic/partnership ideas towards making
>> Offline OSM Designs happen, *Thank You !!*
>>
>>
>>
>> 10AM New York Time / 2PM UTC
>>
>> Thursday, June 11th
>>
>> RSVP with your Skype username, if we are more than 15-20 we'll use an
>> industrial conference call system instead!
>>
>>
>> CONCLUSION: Could OpenStreetMap be the very ultimate in Constructionist
>> Learning Projects, replacing OLPC in coming years, on a quickly
>> shrinking/endangered but still green-in-parts planet?  Regardless, how to
>> build the OLPC Movement's offline/civic learning successes, consciously
>> learning from its community infrastructural mistakes?  Nick Doiron (
>> http://mapmeld.com) and Anish Mangal (
>> https://in.linkedin.com/in/anishmangal) who've spent years thinking
>> about this topic will lead the discussion, agenda is entirely yours!
>> OPTIONAL: submit agenda items in advance right here:
>>
>>
>> https://docs.google.com/document/d/1o6QtzLb6e58YKWqMf_junux2XyBRLFm31un8YLcYslg
>>
>>
>>
>> *< forward invitation to Twitter/wherever as appropriate >*
>>
>>
>> --
>>
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>>
>>
>> _______________________________________________
>> support-gang mailing list
>> support-gang at lists.laptop.org
>> http://lists.laptop.org/listinfo/support-gang
>>
>>
>>
>> --
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Unleash Kids" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to unleashkids+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>>
>> --
>>
>> Jérôme Gagnon-Voyer
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Unleash Kids" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to unleashkids+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Jérôme Gagnon-Voyer
>
> --
> Unsung Heroes of OLPC, interviewed live @ http://unleashkids.org !
> ---
> You received this message because you are subscribed to the Google Groups
> "Unleash Kids" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to unleashkids+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/iaep/attachments/20150621/b9da2090/attachment-0001.html>