[Sugar-devel] ASLO2 review

[Jatin Dhankhar]

Hi Tony,

When the activity maintainer and the developer agree that a new version of
> an activity is ready for release, all that is needed is for the activity
> maintainer to build the bundle and store it in the appropriate add-on
> directory. ASLO.
>
> ASLO does not need any notification since each time a user clicks on an
> activity, ASLO finds the latest version and extracts the activity.info to
> prepare the detail page. If the user clicks on download, the bundle is
> downloaded by its url.
>
> Note: this model makes ASLO essentially bullet-proof since the only
> actions available are to browse activities and select one for download. The
> access is read-only. At the same time ASLO does not need a backend
> requiring adminstration.


This significantly reduces the development effort. So, does that mean the
developer have to specify download url in the activity.info ?
What do you think about the rest of process. Is it the best practice to use
background worker to fetch activity.info ?

Thanks,
Jatin Dhankhar

On Mon, May 29, 2017 at 6:21 PM, Tony Anderson <tony_anderson at usa.net>
wrote:

> Hi Jatin,
>
>
>
>  Release is made into repo -> Webhook is triggered -> Response is parsed
> -> A worker is spawned to handle building (inside docker or in a worker
> thread) ->
> Once bundle is available, activity.info is parsed -> Parsed activity is
> pushed to message queue -> Message queue is used to update index of
> available activities
>
> When the activity maintainer and the developer agree that a new version of
> an activity is ready for release, all that is needed is for the activity
> maintainer to build the bundle and store it in the appropriate add-on
> directory. ASLO.
>
> ASLO does not need any notification since each time a user clicks on an
> activity, ASLO finds the latest version and extracts the activity.info to
> prepare the detail page. If the user clicks on download, the bundle is
> downloaded by its url.
>
> Note: this model makes ASLO essentially bullet-proof since the only
> actions available are to browse activities and select one for download. The
> access is read-only. At the same time ASLO does not need a backend
> requiring adminstration.
>
> Tony
>
>
> On 05/28/2017 06:43 PM, Jatin Dhankhar wrote:
>
> Hi,
>
> I am using webhooks for notifying the server (for release events only). At
> first I was thinking to clone the whole repo but it is possible to access a
> single file via Github API [1] which ultimately hands you a json including
> raw url to download from.
> After which it can be parsed to download the file but only limitation is
> we cannot fetch a file greater than 1 MB.
> Also it may take a long time to download/parse and optionally generate
> bundle, such tasks can be handled via a job management system like celery
> and when bundle is ready, push the name of activity to pub/sub queue system
> (Is it an effective approach ?), we can build them inside docker containers
> as suggested by Samuel.
> I have tested the hook and fetching, yet to test building inside Container
> and setting up Background workers. I aslo used https://www.iron.io/ at a
> previous project for background workers and it was good but it is expensive
> and may cause vendor lock-in.
>
> It would be cool to use a CDN (either in-house or third party like
> SourceForge or Bintray) to distribute the built activities or we can push
> to activities.sugarlabs.org.
>
> So process so far would be
>
> Release is made into repo -> Webhook is triggered -> Response is parsed ->
>> A worker is spawned to handle building (inside docker or in a worker
>> thread) -> Once bundle is available, activity.info is parsed -> Parsed
>> activity is pushed to message queue -> Message queue is used to update
>> index of available activities
>
>
> If something is missing/wrong, kindly guide me in the right direction.
> I have not configured the server properly yet, but it will be accessible
> at aslo-dev.jatindhankhar.in, right now serving on port 5000.
> Thanks,
>
> Jatin Dhankhar
>
>
>
>
>
> [1] https://developer.github.com/v3/repos/contents/
>
> On Sat, May 27, 2017 at 9:32 PM, Jatin Dhankhar <dhankhar.jatin at gmail.com>
> wrote:
>
>> would suggest, if Walter and Tony are agree, to start preparing a proof
>>> of concept. Basically do the following:
>>> (1) Clone some sugar activity to your GH account like TurtleArt. Enable
>>> webhooks for this repo and send events to your server at some URL (like
>>> http://aslo.jatindhankhar.in/hook) when a new release has been done.
>>> (2) Parse this payload and get all the information from the
>>> activity.info file (you might look SamP's code). Please tell us what
>>> might be missing for the site (comparing current ASLO and ASLOv2) so we can
>>> think how to fill those missing info. Please check this carefully.
>>> (3) Build the bundle inside a Docker container (no need for hacks). Just
>>> simple docker container and no docker inside docker. Probably bundles will
>>> be store at download.sugarlabs.org/activitities3/. But for now, just
>>> put in same server and some URL like http://aslo.jatindhankhar
>>> .in/download/<path_to_app>. Then we will think how to sync this.
>>> (4) Show a web page with the listed app we can download this new
>>> generated bundle.
>>>
>>>
>> Okay, Will start work on this tomorrow. I was looking at some Flask
>> tutorials. Is it okay if I use Flask ?
>> Also wrote the weekly blog #3.
>>
>> Thanks,
>> Jatin Dhankhar
>>
>> On Sat, May 27, 2017 at 7:48 PM, Samuel Cantero < <scanterog at gmail.com>
>> scanterog at gmail.com> wrote:
>>
>>> On Fri, May 26, 2017 at 5:57 AM, Jatin Dhankhar <
>>> dhankhar.jatin at gmail.com> wrote:
>>>
>>>> Hi Sam,
>>>>
>>>> Thank you for very much for the detailed explanation. Apart from what
>>>> you have already mentioned. I checked the socialhelp module and it fails to
>>>> build since the package* python-cairosvg* is not available in the
>>>> Ubuntu repos anymore[1].
>>>>
>>>>
>>> I think we don't have to worry about integration with socialhelp.
>>>
>>>
>>>> I also agree that new proposal[2] has sane architecture and use of
>>>> Gtihub API over raw calls is beneficial.
>>>>
>>>
>>> I would suggest, if Walter and Tony are agree, to start preparing a
>>> proof of concept. Basically do the following:
>>> (1) Clone some sugar activity to your GH account like TurtleArt. Enable
>>> webhooks for this repo and send events to your server at some URL (like
>>> <http://aslo.jatindhankhar.in/hook>http://aslo.jatindhankhar.in/hook)
>>> when a new release has been done.
>>> (2) Parse this payload and get all the information from the
>>> activity.info file (you might look SamP's code). Please tell us what
>>> might be missing for the site (comparing current ASLO and ASLOv2) so we can
>>> think how to fill those missing info. Please check this carefully.
>>> (3) Build the bundle inside a Docker container (no need for hacks). Just
>>> simple docker container and no docker inside docker. Probably bundles will
>>> be store at download.sugarlabs.org/activitities3/. But for now, just
>>> put in same server and some URL like http://aslo.jatindhankhar.in/d
>>> ownload/<path_to_app>. Then we will think how to sync this.
>>> (4) Show a web page with the listed app we can download this new
>>> generated bundle.
>>>
>>> If you need more info, just let us know.
>>>
>>> Best,
>>>
>>> Sam C.
>>>
>>>
>>>>
>>>> Thanks,
>>>> Jatin Dhankhar
>>>>
>>>> [1] https://github.com/samdroid-apps/aslo/blob/master/social
>>>> help/Dockerfile#L20
>>>> [2] https://goo.gl/VEIzCr
>>>>
>>>> On Fri, May 26, 2017 at 6:33 AM, Tony Anderson < <tony at olenepal.org>
>>>> tony at olenepal.org> wrote:
>>>>
>>>>> Hi Sam
>>>>>
>>>>> Great work! I agree with you completely.
>>>>>
>>>>> Tony
>>>>>
>>>>>
>>>>> On 05/26/2017 03:02 AM, Samuel Cantero wrote:
>>>>>
>>>>> Hi guys!
>>>>>
>>>>> I have review Sam's code and this is what I've found so far:
>>>>>
>>>>> 1. Initially, all activities shown in the web interface are read/parse
>>>>> from the JSON files [1][2] uploaded to:
>>>>> <https://github.com/samdroid-apps/sugar-activities>
>>>>> https://github.com/samdroid-apps/sugar-activities. Using a repo as a
>>>>> activities registry has advantages and disadvantages.
>>>>>
>>>>> 2. The main web app spawns a thread [3] which polls a kafka message
>>>>> queue (consumer) in order to get the new messages (GitHub payloads) [4]
>>>>> inserted into the message queue by events sent to
>>>>> <https://hook.sugarlabs.org/hook>https://hook.sugarlabs.org/hook.
>>>>>
>>>>> BUT this only looks for changes (GH push events) to sugar-activities
>>>>> repo. This allows to update the activity information in the web interface
>>>>> (version, download link, etc) as well as to introduce new activities [5].
>>>>>
>>>>> You can check how the GH payload is processed here: [6]. In short, it
>>>>> parses the GH payload, gets the repo URL and publishes some "message" to
>>>>> the kafka queue (or bus), i.e act as a kafka producer so it can be read
>>>>> later by [4].
>>>>>
>>>>> This "message" generated at [7] is just a simple dict (json) with two
>>>>> data: github repo URL and bundle_id. The activity *bundle_id* is
>>>>> gotten from the *activity.info <http://activity.info>* file (it
>>>>> doesn't use GH API, just use raw access and always master branch).
>>>>>
>>>>> 3. Let's review now the bot's part so we can understand the whole
>>>>> workflow. Inside repo we can find two folders: bot and bot-master.
>>>>>
>>>>> *Bot-master* is a kafka consumer for the topic
>>>>> "org.sugarlabs.aslo-changes" [8]. This topic is different for the one used
>>>>> at [4] (org.sugarlabs.hook). Simplifying, every new topic is a new message
>>>>> bus. As a consumer, it needs to receive some message to process first.
>>>>>
>>>>> This is where *bot* comes in. *Bot* is the producer for the topic
>>>>> "org.sugarlabs.aslo-changes" (as expected) [9] but also it is a consumer
>>>>> for the topic "org.sugarlabs.hook" [10]. This means that when a new payload
>>>>> is sent to hook.sugarlabs.org/hook, *bot* also gets the message
>>>>> "produced" at [7] (repo URL, bundle_id) and based on that message does the
>>>>> following:
>>>>> (a) clone the repo (master branch) [11];  - this is of course the
>>>>> activity repo from some GH account - no centralized repositories because
>>>>> devs registers the webhooks as explains here:
>>>>> <https://github.com/samdroid-apps/sugar-activities/blob/master/README.md>
>>>>> https://github.com/samdroid-apps/sugar-activities/blob
>>>>> /master/README.md.
>>>>> (b) get the json file using the bundle_id from
>>>>> <http://raw.githubusercontent.com/SAMdroid-apps/sugar-activities/master/>
>>>>> http://raw.githubusercontent.com/SAMdroid-apps/sugar-ac
>>>>> tivities/master/{bundle_id}.json. Again, no GH API [12];
>>>>> (c) Parse activity.info from cloned repo. There are many useful
>>>>> things here like: check if activity is GTK3, if has old toolbars and check
>>>>> minimal sugar version supported [13];
>>>>> (d) Get the images from cloned repo: screenshots and icons [14];
>>>>> (e) generates bundle filename (xo name) and set some extra data in
>>>>> activity.info;
>>>>> (f) generate (compile) the bundle (xo file) [15]. As you can see [16],
>>>>> this code calls the docker container "samdroid/activity-build". I suppose
>>>>> this has been generated previously using [17]. Because SamP was running
>>>>> this bot inside Docker and was building the image using docker again, he
>>>>> needed the wrapdocker file [18].
>>>>>
>>>>> Once the bundle has been generated, it produces the following message
>>>>> [19] for bot-master, which in turns save the bundle [20] and update
>>>>> sugar-activities repo with the new releases/version and other needed
>>>>> information (as new translations) - example at [21]. Yes, this process also
>>>>> manages i18n.
>>>>>
>>>>> *My conclusion*
>>>>>
>>>>> After reviewing code, I still believe the proposal I've sent before
>>>>> [22] is the simplest way to manage this. Of course, maybe we can optimize
>>>>> some stuff along the way. Maybe we can add the chance to upload bundles
>>>>> file (XOs) manually as well, so we don't "remove" devs the chance to upload
>>>>> theirs XOs.
>>>>>
>>>>> Reading SamP who wrote ASLO2, I'm still even more convinced [22] is
>>>>> the right way. He basically described this proposed solution in previous
>>>>> email:  <http://lists.sugarlabs.org/archive/iaep/2017-May/019761.html>
>>>>> http://lists.sugarlabs.org/archive/iaep/2017-May/019761.html. Of
>>>>> course, ASLO2 gives us a lot of code we can reuse for ASLO3.
>>>>>
>>>>> Best,
>>>>>
>>>>> SamC.
>>>>>
>>>>> [1]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L112>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L112
>>>>> [2]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/backend.py#L32>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/backend.py#L32
>>>>> [3]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L114>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L114
>>>>> [4]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L92>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L92
>>>>> [5]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L107>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L107
>>>>> [6]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/hookin/main.py#L12>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/hookin/main.py#L12
>>>>> [7]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/hookin/backend.py#L82>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/hookin
>>>>> /backend.py#L82
>>>>> [8]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot-master/main.py#L77>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot-ma
>>>>> ster/main.py#L77
>>>>> [9]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L40>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L40
>>>>> [10]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L36>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L36
>>>>> [11]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L64>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L64
>>>>> [12]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L31>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L31
>>>>> [13]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/props.py#L117>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/props.py#L117
>>>>> [14]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L55>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L55
>>>>> [15]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L74>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L74
>>>>> [16]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L84>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L84
>>>>> [17]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/activity-build-docker/Dockerfile>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/activ
>>>>> ity-build-docker/Dockerfile
>>>>> [18]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/wrapdocker>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/wrapdocker
>>>>> [19]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L100>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L100
>>>>> [20]
>>>>> <https://github.com/samdroid-apps/aslo/blob/master/bot-master/main.py#L47>
>>>>> https://github.com/samdroid-apps/aslo/blob/master/bot-m
>>>>> aster/main.py#L47
>>>>> [21]
>>>>> <https://github.com/samdroid-apps/sugar-activities/commit/8de09150934502a845b5f33e6031e5ebd60e4d94>
>>>>> https://github.com/samdroid-apps/sugar-activities/commi
>>>>> t/8de09150934502a845b5f33e6031e5ebd60e4d94
>>>>> [22]  <https://goo.gl/VEIzCr>https://goo.gl/VEIzCr
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20170529/9483825d/attachment-0001.html>