[Sugar-devel] Sugar-devel Digest, Vol 103, Issue 56

Tony Anderson tony at olenepal.org
Mon May 29 09:22:41 EDT 2017
Previous message (by thread): [Sugar-devel] ASLO2 review
Next message (by thread): [Sugar-devel] Mind Math Activity
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi Jatin,

This significantly reduces the development effort. So, does that mean the
developer have to specify download url in the activity.info ?
What do you think about the rest of process. Is it the best practice to use
background worker to fetch activity.info ?


Both Flask and Django use the fusion model, a user's request returns a 
page generated by local information (db or file system). A template is 
used since the form of the page is independent of the specific object 
requested. The python code generates the page by passing the information 
specific to that object to a template.

The python code finds the bundle (make a list of bundles in the 
specified directory and sort them). Using zipFile, the python code opens 
and reads activity.info.
It passes the relevant information to the template - including the 
download url.

The Django prototype shows an example of how that is done.

I hope the development effort is reduced enough that we can have a 
working prototype installed on Sunjammer by the Monday meeting, June 12. 
This prototype can then access the information directly from 
download.sugarlabs.org/activities. From this prototype, we can see what 
will be required to bring all of the bundles up to standard.

Tonyh


On 05/29/2017 03:58 PM, sugar-devel-request at lists.sugarlabs.org wrote:
> Send Sugar-devel mailing list submissions to
> 	sugar-devel at lists.sugarlabs.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://lists.sugarlabs.org/listinfo/sugar-devel
> or, via email, send a message with subject or body 'help' to
> 	sugar-devel-request at lists.sugarlabs.org
>
> You can reach the person managing the list at
> 	sugar-devel-owner at lists.sugarlabs.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Sugar-devel digest..."
>
>
> Today's Topics:
>
>     1. Re: ASLO2 review (Jatin Dhankhar)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 29 May 2017 18:28:25 +0530
> From: Jatin Dhankhar <dhankhar.jatin at gmail.com>
> To: Tony Anderson <tony_anderson at usa.net>
> Cc: Samuel Cantero <scanterog at gmail.com>, "Bender, Walter"
> 	<walter.bender at gmail.com>,  Sugar-dev Devel
> 	<sugar-devel at lists.sugarlabs.org>
> Subject: Re: [Sugar-devel] ASLO2 review
> Message-ID:
> 	<CAD+LdAE=Tt-wQi=memB6sehJV=AX-3GKA4-fRwCsWC4JuU+Rrg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Tony,
>
> When the activity maintainer and the developer agree that a new version of
>> an activity is ready for release, all that is needed is for the activity
>> maintainer to build the bundle and store it in the appropriate add-on
>> directory. ASLO.
>>
>> ASLO does not need any notification since each time a user clicks on an
>> activity, ASLO finds the latest version and extracts the activity.info to
>> prepare the detail page. If the user clicks on download, the bundle is
>> downloaded by its url.
>>
>> Note: this model makes ASLO essentially bullet-proof since the only
>> actions available are to browse activities and select one for download. The
>> access is read-only. At the same time ASLO does not need a backend
>> requiring adminstration.
>
> This significantly reduces the development effort. So, does that mean the
> developer have to specify download url in the activity.info ?
> What do you think about the rest of process. Is it the best practice to use
> background worker to fetch activity.info ?
>
> Thanks,
> Jatin Dhankhar
>
> On Mon, May 29, 2017 at 6:21 PM, Tony Anderson <tony_anderson at usa.net>
> wrote:
>
>> Hi Jatin,
>>
>>
>>
>>   Release is made into repo -> Webhook is triggered -> Response is parsed
>> -> A worker is spawned to handle building (inside docker or in a worker
>> thread) ->
>> Once bundle is available, activity.info is parsed -> Parsed activity is
>> pushed to message queue -> Message queue is used to update index of
>> available activities
>>
>> When the activity maintainer and the developer agree that a new version of
>> an activity is ready for release, all that is needed is for the activity
>> maintainer to build the bundle and store it in the appropriate add-on
>> directory. ASLO.
>>
>> ASLO does not need any notification since each time a user clicks on an
>> activity, ASLO finds the latest version and extracts the activity.info to
>> prepare the detail page. If the user clicks on download, the bundle is
>> downloaded by its url.
>>
>> Note: this model makes ASLO essentially bullet-proof since the only
>> actions available are to browse activities and select one for download. The
>> access is read-only. At the same time ASLO does not need a backend
>> requiring adminstration.
>>
>> Tony
>>
>>
>> On 05/28/2017 06:43 PM, Jatin Dhankhar wrote:
>>
>> Hi,
>>
>> I am using webhooks for notifying the server (for release events only). At
>> first I was thinking to clone the whole repo but it is possible to access a
>> single file via Github API [1] which ultimately hands you a json including
>> raw url to download from.
>> After which it can be parsed to download the file but only limitation is
>> we cannot fetch a file greater than 1 MB.
>> Also it may take a long time to download/parse and optionally generate
>> bundle, such tasks can be handled via a job management system like celery
>> and when bundle is ready, push the name of activity to pub/sub queue system
>> (Is it an effective approach ?), we can build them inside docker containers
>> as suggested by Samuel.
>> I have tested the hook and fetching, yet to test building inside Container
>> and setting up Background workers. I aslo used https://www.iron.io/ at a
>> previous project for background workers and it was good but it is expensive
>> and may cause vendor lock-in.
>>
>> It would be cool to use a CDN (either in-house or third party like
>> SourceForge or Bintray) to distribute the built activities or we can push
>> to activities.sugarlabs.org.
>>
>> So process so far would be
>>
>> Release is made into repo -> Webhook is triggered -> Response is parsed ->
>>> A worker is spawned to handle building (inside docker or in a worker
>>> thread) -> Once bundle is available, activity.info is parsed -> Parsed
>>> activity is pushed to message queue -> Message queue is used to update
>>> index of available activities
>>
>> If something is missing/wrong, kindly guide me in the right direction.
>> I have not configured the server properly yet, but it will be accessible
>> at aslo-dev.jatindhankhar.in, right now serving on port 5000.
>> Thanks,
>>
>> Jatin Dhankhar
>>
>>
>>
>>
>>
>> [1] https://developer.github.com/v3/repos/contents/
>>
>> On Sat, May 27, 2017 at 9:32 PM, Jatin Dhankhar <dhankhar.jatin at gmail.com>
>> wrote:
>>
>>> would suggest, if Walter and Tony are agree, to start preparing a proof
>>>> of concept. Basically do the following:
>>>> (1) Clone some sugar activity to your GH account like TurtleArt. Enable
>>>> webhooks for this repo and send events to your server at some URL (like
>>>> http://aslo.jatindhankhar.in/hook) when a new release has been done.
>>>> (2) Parse this payload and get all the information from the
>>>> activity.info file (you might look SamP's code). Please tell us what
>>>> might be missing for the site (comparing current ASLO and ASLOv2) so we can
>>>> think how to fill those missing info. Please check this carefully.
>>>> (3) Build the bundle inside a Docker container (no need for hacks). Just
>>>> simple docker container and no docker inside docker. Probably bundles will
>>>> be store at download.sugarlabs.org/activitities3/. But for now, just
>>>> put in same server and some URL like http://aslo.jatindhankhar
>>>> .in/download/<path_to_app>. Then we will think how to sync this.
>>>> (4) Show a web page with the listed app we can download this new
>>>> generated bundle.
>>>>
>>>>
>>> Okay, Will start work on this tomorrow. I was looking at some Flask
>>> tutorials. Is it okay if I use Flask ?
>>> Also wrote the weekly blog #3.
>>>
>>> Thanks,
>>> Jatin Dhankhar
>>>
>>> On Sat, May 27, 2017 at 7:48 PM, Samuel Cantero < <scanterog at gmail.com>
>>> scanterog at gmail.com> wrote:
>>>
>>>> On Fri, May 26, 2017 at 5:57 AM, Jatin Dhankhar <
>>>> dhankhar.jatin at gmail.com> wrote:
>>>>
>>>>> Hi Sam,
>>>>>
>>>>> Thank you for very much for the detailed explanation. Apart from what
>>>>> you have already mentioned. I checked the socialhelp module and it fails to
>>>>> build since the package* python-cairosvg* is not available in the
>>>>> Ubuntu repos anymore[1].
>>>>>
>>>>>
>>>> I think we don't have to worry about integration with socialhelp.
>>>>
>>>>
>>>>> I also agree that new proposal[2] has sane architecture and use of
>>>>> Gtihub API over raw calls is beneficial.
>>>>>
>>>> I would suggest, if Walter and Tony are agree, to start preparing a
>>>> proof of concept. Basically do the following:
>>>> (1) Clone some sugar activity to your GH account like TurtleArt. Enable
>>>> webhooks for this repo and send events to your server at some URL (like
>>>> <http://aslo.jatindhankhar.in/hook>http://aslo.jatindhankhar.in/hook)
>>>> when a new release has been done.
>>>> (2) Parse this payload and get all the information from the
>>>> activity.info file (you might look SamP's code). Please tell us what
>>>> might be missing for the site (comparing current ASLO and ASLOv2) so we can
>>>> think how to fill those missing info. Please check this carefully.
>>>> (3) Build the bundle inside a Docker container (no need for hacks). Just
>>>> simple docker container and no docker inside docker. Probably bundles will
>>>> be store at download.sugarlabs.org/activitities3/. But for now, just
>>>> put in same server and some URL like http://aslo.jatindhankhar.in/d
>>>> ownload/<path_to_app>. Then we will think how to sync this.
>>>> (4) Show a web page with the listed app we can download this new
>>>> generated bundle.
>>>>
>>>> If you need more info, just let us know.
>>>>
>>>> Best,
>>>>
>>>> Sam C.
>>>>
>>>>
>>>>> Thanks,
>>>>> Jatin Dhankhar
>>>>>
>>>>> [1] https://github.com/samdroid-apps/aslo/blob/master/social
>>>>> help/Dockerfile#L20
>>>>> [2] https://goo.gl/VEIzCr
>>>>>
>>>>> On Fri, May 26, 2017 at 6:33 AM, Tony Anderson < <tony at olenepal.org>
>>>>> tony at olenepal.org> wrote:
>>>>>
>>>>>> Hi Sam
>>>>>>
>>>>>> Great work! I agree with you completely.
>>>>>>
>>>>>> Tony
>>>>>>
>>>>>>
>>>>>> On 05/26/2017 03:02 AM, Samuel Cantero wrote:
>>>>>>
>>>>>> Hi guys!
>>>>>>
>>>>>> I have review Sam's code and this is what I've found so far:
>>>>>>
>>>>>> 1. Initially, all activities shown in the web interface are read/parse
>>>>>> from the JSON files [1][2] uploaded to:
>>>>>> <https://github.com/samdroid-apps/sugar-activities>
>>>>>> https://github.com/samdroid-apps/sugar-activities. Using a repo as a
>>>>>> activities registry has advantages and disadvantages.
>>>>>>
>>>>>> 2. The main web app spawns a thread [3] which polls a kafka message
>>>>>> queue (consumer) in order to get the new messages (GitHub payloads) [4]
>>>>>> inserted into the message queue by events sent to
>>>>>> <https://hook.sugarlabs.org/hook>https://hook.sugarlabs.org/hook.
>>>>>>
>>>>>> BUT this only looks for changes (GH push events) to sugar-activities
>>>>>> repo. This allows to update the activity information in the web interface
>>>>>> (version, download link, etc) as well as to introduce new activities [5].
>>>>>>
>>>>>> You can check how the GH payload is processed here: [6]. In short, it
>>>>>> parses the GH payload, gets the repo URL and publishes some "message" to
>>>>>> the kafka queue (or bus), i.e act as a kafka producer so it can be read
>>>>>> later by [4].
>>>>>>
>>>>>> This "message" generated at [7] is just a simple dict (json) with two
>>>>>> data: github repo URL and bundle_id. The activity *bundle_id* is
>>>>>> gotten from the *activity.info <http://activity.info>* file (it
>>>>>> doesn't use GH API, just use raw access and always master branch).
>>>>>>
>>>>>> 3. Let's review now the bot's part so we can understand the whole
>>>>>> workflow. Inside repo we can find two folders: bot and bot-master.
>>>>>>
>>>>>> *Bot-master* is a kafka consumer for the topic
>>>>>> "org.sugarlabs.aslo-changes" [8]. This topic is different for the one used
>>>>>> at [4] (org.sugarlabs.hook). Simplifying, every new topic is a new message
>>>>>> bus. As a consumer, it needs to receive some message to process first.
>>>>>>
>>>>>> This is where *bot* comes in. *Bot* is the producer for the topic
>>>>>> "org.sugarlabs.aslo-changes" (as expected) [9] but also it is a consumer
>>>>>> for the topic "org.sugarlabs.hook" [10]. This means that when a new payload
>>>>>> is sent to hook.sugarlabs.org/hook, *bot* also gets the message
>>>>>> "produced" at [7] (repo URL, bundle_id) and based on that message does the
>>>>>> following:
>>>>>> (a) clone the repo (master branch) [11];  - this is of course the
>>>>>> activity repo from some GH account - no centralized repositories because
>>>>>> devs registers the webhooks as explains here:
>>>>>> <https://github.com/samdroid-apps/sugar-activities/blob/master/README.md>
>>>>>> https://github.com/samdroid-apps/sugar-activities/blob
>>>>>> /master/README.md.
>>>>>> (b) get the json file using the bundle_id from
>>>>>> <http://raw.githubusercontent.com/SAMdroid-apps/sugar-activities/master/>
>>>>>> http://raw.githubusercontent.com/SAMdroid-apps/sugar-ac
>>>>>> tivities/master/{bundle_id}.json. Again, no GH API [12];
>>>>>> (c) Parse activity.info from cloned repo. There are many useful
>>>>>> things here like: check if activity is GTK3, if has old toolbars and check
>>>>>> minimal sugar version supported [13];
>>>>>> (d) Get the images from cloned repo: screenshots and icons [14];
>>>>>> (e) generates bundle filename (xo name) and set some extra data in
>>>>>> activity.info;
>>>>>> (f) generate (compile) the bundle (xo file) [15]. As you can see [16],
>>>>>> this code calls the docker container "samdroid/activity-build". I suppose
>>>>>> this has been generated previously using [17]. Because SamP was running
>>>>>> this bot inside Docker and was building the image using docker again, he
>>>>>> needed the wrapdocker file [18].
>>>>>>
>>>>>> Once the bundle has been generated, it produces the following message
>>>>>> [19] for bot-master, which in turns save the bundle [20] and update
>>>>>> sugar-activities repo with the new releases/version and other needed
>>>>>> information (as new translations) - example at [21]. Yes, this process also
>>>>>> manages i18n.
>>>>>>
>>>>>> *My conclusion*
>>>>>>
>>>>>> After reviewing code, I still believe the proposal I've sent before
>>>>>> [22] is the simplest way to manage this. Of course, maybe we can optimize
>>>>>> some stuff along the way. Maybe we can add the chance to upload bundles
>>>>>> file (XOs) manually as well, so we don't "remove" devs the chance to upload
>>>>>> theirs XOs.
>>>>>>
>>>>>> Reading SamP who wrote ASLO2, I'm still even more convinced [22] is
>>>>>> the right way. He basically described this proposed solution in previous
>>>>>> email:  <http://lists.sugarlabs.org/archive/iaep/2017-May/019761.html>
>>>>>> http://lists.sugarlabs.org/archive/iaep/2017-May/019761.html. Of
>>>>>> course, ASLO2 gives us a lot of code we can reuse for ASLO3.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> SamC.
>>>>>>
>>>>>> [1]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L112>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L112
>>>>>> [2]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/backend.py#L32>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/backend.py#L32
>>>>>> [3]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L114>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L114
>>>>>> [4]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L92>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L92
>>>>>> [5]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L107>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L107
>>>>>> [6]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/hookin/main.py#L12>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/hookin/main.py#L12
>>>>>> [7]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/hookin/backend.py#L82>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/hookin
>>>>>> /backend.py#L82
>>>>>> [8]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot-master/main.py#L77>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot-ma
>>>>>> ster/main.py#L77
>>>>>> [9]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L40>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L40
>>>>>> [10]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L36>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L36
>>>>>> [11]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L64>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L64
>>>>>> [12]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L31>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L31
>>>>>> [13]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/props.py#L117>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/props.py#L117
>>>>>> [14]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L55>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L55
>>>>>> [15]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L74>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L74
>>>>>> [16]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L84>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L84
>>>>>> [17]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/activity-build-docker/Dockerfile>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/activ
>>>>>> ity-build-docker/Dockerfile
>>>>>> [18]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/wrapdocker>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/wrapdocker
>>>>>> [19]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L100>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L100
>>>>>> [20]
>>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot-master/main.py#L47>
>>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot-m
>>>>>> aster/main.py#L47
>>>>>> [21]
>>>>>> <https://github.com/samdroid-apps/sugar-activities/commit/8de09150934502a845b5f33e6031e5ebd60e4d94>
>>>>>> https://github.com/samdroid-apps/sugar-activities/commi
>>>>>> t/8de09150934502a845b5f33e6031e5ebd60e4d94
>>>>>> [22]  <https://goo.gl/VEIzCr>https://goo.gl/VEIzCr
>>>>>>
>>>>>>
>>>>>>
>>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20170529/9483825d/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Sugar-devel mailing list
> Sugar-devel at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/sugar-devel
>
>
> ------------------------------
>
> End of Sugar-devel Digest, Vol 103, Issue 56
> ********************************************
Previous message (by thread): [Sugar-devel] ASLO2 review
Next message (by thread): [Sugar-devel] Mind Math Activity
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the Sugar-devel mailing list