[Sugar-devel] ASLO2 review
Tony Anderson
tony_anderson at usa.net
Mon May 29 08:51:42 EDT 2017
Hi Jatin,
Release is made into repo -> Webhook is triggered -> Response is
parsed -> A worker is spawned to handle building (inside docker or in a
worker thread) ->
Once bundle is available, activity.info <http://activity.info> is parsed
-> Parsed activity is pushed to message queue -> Message queue is used
to update index of available activities
When the activity maintainer and the developer agree that a new version
of an activity is ready for release, all that is needed is for the
activity maintainer to build the bundle and store it in the appropriate
add-on directory. ASLO.
ASLO does not need any notification since each time a user clicks on an
activity, ASLO finds the latest version and extracts the activity.info
to prepare the detail page. If the user clicks on download, the bundle
is downloaded by its url.
Note: this model makes ASLO essentially bullet-proof since the only
actions available are to browse activities and select one for download.
The access is read-only. At the same time ASLO does not need a backend
requiring adminstration.
Tony
On 05/28/2017 06:43 PM, Jatin Dhankhar wrote:
> Hi,
>
> I am using webhooks for notifying the server (for release events
> only). At first I was thinking to clone the whole repo but it is
> possible to access a single file via Github API [1] which ultimately
> hands you a json including raw url to download from.
> After which it can be parsed to download the file but only limitation
> is we cannot fetch a file greater than 1 MB.
> Also it may take a long time to download/parse and optionally generate
> bundle, such tasks can be handled via a job management system like
> celery and when bundle is ready, push the name of activity to pub/sub
> queue system (Is it an effective approach ?), we can build them inside
> docker containers as suggested by Samuel.
> I have tested the hook and fetching, yet to test building inside
> Container and setting up Background workers. I aslo used
> https://www.iron.io/ at a previous project for background workers and
> it was good but it is expensive and may cause vendor lock-in.
>
> It would be cool to use a CDN (either in-house or third party like
> SourceForge or Bintray) to distribute the built activities or we can
> push to activities.sugarlabs.org <http://activities.sugarlabs.org>.
>
> So process so far would be
>
> Release is made into repo -> Webhook is triggered -> Response is
> parsed -> A worker is spawned to handle building (inside docker or
> in a worker thread) -> Once bundle is available, activity.info
> <http://activity.info> is parsed -> Parsed activity is pushed to
> message queue -> Message queue is used to update index of
> available activities
>
> If something is missing/wrong, kindly guide me in the right direction.
> I have not configured the server properly yet, but it will be
> accessible at aslo-dev.jatindhankhar.in
> <http://aslo-dev.jatindhankhar.in>, right now serving on port 5000.
> Thanks,
>
> Jatin Dhankhar
>
>
>
> [1] https://developer.github.com/v3/repos/contents/
>
> On Sat, May 27, 2017 at 9:32 PM, Jatin Dhankhar
> <dhankhar.jatin at gmail.com <mailto:dhankhar.jatin at gmail.com>> wrote:
>
> would suggest, if Walter and Tony are agree, to start
> preparing a proof of concept. Basically do the following:
> (1) Clone some sugar activity to your GH account like
> TurtleArt. Enable webhooks for this repo and send events to
> your server at some URL (like
> http://aslo.jatindhankhar.in/hook
> <http://aslo.jatindhankhar.in/hook>) when a new release has
> been done.
> (2) Parse this payload and get all the information from the
> activity.info <http://activity.info/> file (you might look
> SamP's code). Please tell us what might be missing for the
> site (comparing current ASLO and ASLOv2) so we can think how
> to fill those missing info. Please check this carefully.
> (3) Build the bundle inside a Docker container (no need for
> hacks). Just simple docker container and no docker inside
> docker. Probably bundles will be store at
> download.sugarlabs.org/activitities3/
> <http://download.sugarlabs.org/activitities3/>. But for now,
> just put in same server and some URL like
> http://aslo.jatindhankhar.in/download/
> <http://aslo.jatindhankhar.in/download/><path_to_app>. Then we
> will think how to sync this.
> (4) Show a web page with the listed app we can download this
> new generated bundle.
>
>
> Okay, Will start work on this tomorrow. I was looking at some
> Flask tutorials. Is it okay if I use Flask ?
> Also wrote the weekly blog #3.
>
> Thanks,
> Jatin Dhankhar
>
> On Sat, May 27, 2017 at 7:48 PM, Samuel Cantero
> <scanterog at gmail.com <mailto:scanterog at gmail.com>> wrote:
>
> On Fri, May 26, 2017 at 5:57 AM, Jatin Dhankhar
> <dhankhar.jatin at gmail.com <mailto:dhankhar.jatin at gmail.com>>
> wrote:
>
> Hi Sam,
>
> Thank you for very much for the detailed explanation.
> Apart from what you have already mentioned. I checked the
> socialhelp module and it fails to build since the
> package/ python-cairosvg/ is not available in the Ubuntu
> repos anymore[1].
>
>
> I think we don't have to worry about integration with socialhelp.
>
> I also agree that new proposal[2] has sane architecture
> and use of Gtihub API over raw calls is beneficial.
>
>
> I would suggest, if Walter and Tony are agree, to start
> preparing a proof of concept. Basically do the following:
> (1) Clone some sugar activity to your GH account like
> TurtleArt. Enable webhooks for this repo and send events to
> your server at some URL (like
> http://aslo.jatindhankhar.in/hook
> <http://aslo.jatindhankhar.in/hook>) when a new release has
> been done.
> (2) Parse this payload and get all the information from the
> activity.info <http://activity.info> file (you might look
> SamP's code). Please tell us what might be missing for the
> site (comparing current ASLO and ASLOv2) so we can think how
> to fill those missing info. Please check this carefully.
> (3) Build the bundle inside a Docker container (no need for
> hacks). Just simple docker container and no docker inside
> docker. Probably bundles will be store at
> download.sugarlabs.org/activitities3/
> <http://download.sugarlabs.org/activitities3/>. But for now,
> just put in same server and some URL like
> http://aslo.jatindhankhar.in/download/
> <http://aslo.jatindhankhar.in/download/><path_to_app>. Then we
> will think how to sync this.
> (4) Show a web page with the listed app we can download this
> new generated bundle.
>
> If you need more info, just let us know.
>
> Best,
>
> Sam C.
>
>
>
> Thanks,
> Jatin Dhankhar
>
> [1]
> https://github.com/samdroid-apps/aslo/blob/master/socialhelp/Dockerfile#L20
> <https://github.com/samdroid-apps/aslo/blob/master/socialhelp/Dockerfile#L20>
> [2] https://goo.gl/VEIzCr
>
> On Fri, May 26, 2017 at 6:33 AM, Tony Anderson
> <tony at olenepal.org <mailto:tony at olenepal.org>> wrote:
>
> Hi Sam
>
> Great work! I agree with you completely.
>
> Tony
>
>
> On 05/26/2017 03:02 AM, Samuel Cantero wrote:
>> Hi guys!
>>
>> I have review Sam's code and this is what I've found
>> so far:
>>
>> 1. Initially, all activities shown in the web
>> interface are read/parse from the JSON files [1][2]
>> uploaded to:
>> https://github.com/samdroid-apps/sugar-activities
>> <https://github.com/samdroid-apps/sugar-activities>.
>> Using a repo as a activities registry has advantages
>> and disadvantages.
>>
>> 2. The main web app spawns a thread [3] which polls a
>> kafka message queue (consumer) in order to get the
>> new messages (GitHub payloads) [4] inserted into the
>> message queue by events sent to
>> https://hook.sugarlabs.org/hook
>> <https://hook.sugarlabs.org/hook>.
>>
>> BUT this only looks for changes (GH push events) to
>> sugar-activities repo. This allows to update the
>> activity information in the web interface (version,
>> download link, etc) as well as to introduce new
>> activities [5].
>>
>> You can check how the GH payload is processed here:
>> [6]. In short, it parses the GH payload, gets the
>> repo URL and publishes some "message" to the kafka
>> queue (or bus), i.e act as a kafka producer so it can
>> be read later by [4].
>>
>> This "message" generated at [7] is just a simple dict
>> (json) with two data: github repo URL and bundle_id.
>> The activity /bundle_id/ is gotten from the
>> /activity.info <http://activity.info>/ file (it
>> doesn't use GH API, just use raw access and always
>> master branch).
>>
>> 3. Let's review now the bot's part so we can
>> understand the whole workflow. Inside repo we can
>> find two folders: bot and bot-master.
>>
>> /Bot-master/ is a kafka consumer for the topic
>> "org.sugarlabs.aslo-changes" [8]. This topic is
>> different for the one used at [4]
>> (org.sugarlabs.hook). Simplifying, every new topic is
>> a new message bus. As a consumer, it needs to receive
>> some message to process first.
>>
>> This is where /bot/ comes in. /Bot/ is the producer
>> for the topic "org.sugarlabs.aslo-changes" (as
>> expected) [9] but also it is a consumer for the topic
>> "org.sugarlabs.hook" [10]. This means that when a new
>> payload is sent to hook.sugarlabs.org/hook
>> <http://hook.sugarlabs.org/hook>, /bot/ also gets the
>> message "produced" at [7] (repo URL, bundle_id) and
>> based on that message does the following:
>> (a) clone the repo (master branch) [11]; - this is
>> of course the activity repo from some GH account - no
>> centralized repositories because devs registers the
>> webhooks as explains here:
>> https://github.com/samdroid-apps/sugar-activities/blob/master/README.md
>> <https://github.com/samdroid-apps/sugar-activities/blob/master/README.md>.
>> (b) get the json file using the bundle_id from
>> http://raw.githubusercontent.com/SAMdroid-apps/sugar-activities/master/
>> <http://raw.githubusercontent.com/SAMdroid-apps/sugar-activities/master/>{bundle_id}.json.
>> Again, no GH API [12];
>> (c) Parse activity.info <http://activity.info> from
>> cloned repo. There are many useful things here like:
>> check if activity is GTK3, if has old toolbars and
>> check minimal sugar version supported [13];
>> (d) Get the images from cloned repo: screenshots and
>> icons [14];
>> (e) generates bundle filename (xo name) and set some
>> extra data in activity.info <http://activity.info>;
>> (f) generate (compile) the bundle (xo file) [15]. As
>> you can see [16], this code calls the docker
>> container "samdroid/activity-build". I suppose this
>> has been generated previously using [17]. Because
>> SamP was running this bot inside Docker and was
>> building the image using docker again, he needed the
>> wrapdocker file [18].
>>
>> Once the bundle has been generated, it produces the
>> following message [19] for bot-master, which in turns
>> save the bundle [20] and update sugar-activities repo
>> with the new releases/version and other needed
>> information (as new translations) - example at [21].
>> Yes, this process also manages i18n.
>>
>> *My conclusion*
>> *
>> *
>> After reviewing code, I still believe the proposal
>> I've sent before [22] is the simplest way to manage
>> this. Of course, maybe we can optimize some stuff
>> along the way. Maybe we can add the chance to upload
>> bundles file (XOs) manually as well, so we don't
>> "remove" devs the chance to upload theirs XOs.
>>
>> Reading SamP who wrote ASLO2, I'm still even more
>> convinced [22] is the right way. He basically
>> described this proposed solution in previous email:
>> http://lists.sugarlabs.org/archive/iaep/2017-May/019761.html
>> <http://lists.sugarlabs.org/archive/iaep/2017-May/019761.html>.
>> Of course, ASLO2 gives us a lot of code we can reuse
>> for ASLO3.
>>
>> Best,
>>
>> SamC.
>>
>> [1]
>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L112
>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L112>
>> [2]
>> https://github.com/samdroid-apps/aslo/blob/master/web/backend.py#L32
>> <https://github.com/samdroid-apps/aslo/blob/master/web/backend.py#L32>
>> [3]
>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L114
>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L114>
>> [4]
>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L92
>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L92>
>> [5]
>> https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L107
>> <https://github.com/samdroid-apps/aslo/blob/master/web/main.py#L107>
>> [6]
>> https://github.com/samdroid-apps/aslo/blob/master/hookin/main.py#L12
>> <https://github.com/samdroid-apps/aslo/blob/master/hookin/main.py#L12>
>> [7]
>> https://github.com/samdroid-apps/aslo/blob/master/hookin/backend.py#L82
>> <https://github.com/samdroid-apps/aslo/blob/master/hookin/backend.py#L82>
>> [8]
>> https://github.com/samdroid-apps/aslo/blob/master/bot-master/main.py#L77
>> <https://github.com/samdroid-apps/aslo/blob/master/bot-master/main.py#L77>
>> [9]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L40
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L40>
>> [10]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L36
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L36>
>> [11]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L64
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L64>
>> [12]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L31
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L31>
>> [13]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/props.py#L117
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/props.py#L117>
>> [14]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L55
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/img.py#L55>
>> [15]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L74
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L74>
>> [16]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L84
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/build.py#L84>
>> [17]
>> https://github.com/samdroid-apps/aslo/blob/master/activity-build-docker/Dockerfile
>> <https://github.com/samdroid-apps/aslo/blob/master/activity-build-docker/Dockerfile>
>> [18]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/wrapdocker
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/wrapdocker>
>> [19]
>> https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L100
>> <https://github.com/samdroid-apps/aslo/blob/master/bot/main.py#L100>
>> [20]
>> https://github.com/samdroid-apps/aslo/blob/master/bot-master/main.py#L47
>> <https://github.com/samdroid-apps/aslo/blob/master/bot-master/main.py#L47>
>> [21]
>> https://github.com/samdroid-apps/sugar-activities/commit/8de09150934502a845b5f33e6031e5ebd60e4d94
>> <https://github.com/samdroid-apps/sugar-activities/commit/8de09150934502a845b5f33e6031e5ebd60e4d94>
>> [22] https://goo.gl/VEIzCr
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20170529/05142236/attachment-0001.html>
More information about the Sugar-devel
mailing list