[ASLO] ASLO build and deployment process (Jatin Dhankhar)

[Jatin Dhankhar]

>
> At the moment, I think you should focus on the 25 activity test set and
> getting it working with django.

Okay, on it.


>  we can arrange to host the development on a Sugar Labs server, it should
> make both the repository itself (download.sugarlabs.org) and the ASLO
> metadata directly accessible.

In the meantime we can host it on DigitalOcean, I have credits for
DigitalOcean which I got as part of Student pack.

My IRC nickname is tony37. However, I am never on #sugar (a twelve hour
> time difference so the normal participants are in bed). Last year, Utkarsh
> and I met whenever he felt the need to discuss a technical issue in detail.
> We limited the meetings to one hour. That proved quite productive.


Yes we can do the same.

Thanks,
Jatin Dhankhar

On Thu, Apr 13, 2017 at 5:50 AM, Tony Anderson <tony_anderson at usa.net>
wrote:

> Hi Jatin
>
> At the moment, I think you should focus on the 25 activity test set and
> getting it working with django. I find it is much more productive to move
> from one working version to another with added capability. If we can
> arrange to host the development on a Sugar Labs server, it should make both
> the repository itself (download.sugarlabs.org) and the ASLO metadata
> directly accessible.
>
> My IRC nickname is tony37. However, I am never on #sugar (a twelve hour
> time difference so the normal participants are in bed). Last year, Utkarsh
> and I met whenever he felt the need to discuss a technical issue in detail.
> We limited the meetings to one hour. That proved quite productive.
>
> Tony
>
>
> On 04/13/2017 01:14 AM, Jatin Dhankhar wrote:
>
> Hi,
>
> I think we need to agree on the use of IRC. If you want to communicate
>> with members of the community, you must go where they are (#sugar). If you
>> want a one-to-one meeting on IRC with me, I would suggest #sugar-newbies.
>> It is normally dormant but works well and saves a log for later review. It
>> worked well for meetings with Utkarsh Tiwari during last year's GSOC.
>
> Sure, whatever works :). What is your IRC nickname ?
>
> There are two things that you will need to have local to the django
>> project. First is the directory download.sugarlabs.org which has the
>> Sugar activity bundles
>
> Do I need to mirror the whole setup/directory ?
>
> When talking about scraping you probably meant
> <http://activities.sugarlabs.org>http://activities.sugarlabs.org instead
> of http://download.sugarlabs.org/, right ?
> Also for scraping, Scrapy <https://scrapy.org/> seems to more popular
> than beautifulsoup ?
>
> Also a big thanks for including Walter in the discussion :D
>
> - Jatin Dhankhar
>
>
> On Wed, Apr 12, 2017 at 6:30 AM, Tony Anderson <tony_anderson at usa.net>
> wrote:
>
>> Hi, Jatin
>>
>> I think we need to agree on the use of IRC. If you want to communicate
>> with members of the community, you must go where they are (#sugar). If you
>> want a one-to-one meeting on IRC with me, I would suggest #sugar-newbies.
>> It is normally dormant but works well and saves a log for later review. It
>> worked well for meetings with Utkarsh Tiwari during last year's GSOC.
>>
>> There are two things that you will need to have local to the django
>> project. First is the directory download.sugarlabs.org which has the
>> Sugar activity bundles. The second is the 'metadata' in the mysql db. For
>> scraping, I would recommend BeautifulSoup (bs4). The trick will be to
>> decide what data we want to capture and add to the json.
>>
>> The json fields in activities.json are ones I chose for the minimal
>> system. You may want to include other information such as the number of
>> downloads, which collections (should be entered as tags in a tag-field) and
>> so on. One item I have referred to as flags (I marked some as X but don't
>> remember what that meant, oh well). The intent is to record the platforms
>> where the activity works. We also should provide links to the homepage,
>> repository page, and update page (whatever that is). I think if you have a
>> working scrape tool, the data it collects can be expanded as needed
>> (assuming the tool runs in a reasonable time).
>>
>> Naturally, it would be easier if you have access to the db directly.
>>
>> Tony
>>
>>
>> On 04/12/2017 01:40 AM, Jatin Dhankhar wrote:
>>
>> One thing you could look at. On activities.sugarlabs,org, can you
>>> determine from Remora where the metadata is stored? I assume a db.
>>> Currently I am thinking to use BeautifulSoup to scrape the site to get that
>>> data, but it would be much easier to access the data directly.
>>
>> As per wiki https://wiki.mozilla.org/Update:Remora_Server_Requirements#
>> SVN.2C_DB_and_app_config data is stored in mysql database. I don't have
>> access to the production server where ASLO is currently running, following
>> file https://github.com/sugarlabs/aslo/blob/master/aslo/db-update.sh#L9 confirms
>> that data is stored in a mysql db. However it would be interesting and fun
>> to scrape the data from live site. I would do that.
>>
>> Thanks will poke around the code, looks to me it's a django app and I
>> have to mount it on my django project, thanks :)
>>
>> If you are talking about IRC as a place to meet Sugar community members,
>>> use the freenode #sugar. This is probably most active from 8-17 EST
>>> (UTC-5). I am currently in the Philippines which is UTC+ 7.
>>
>>
>>  Yes, tried that.  https://gitter.im fits in naturally with Github
>> (really sorry for suggesting a new mode of communication everyday) 😅
>>
>> - Jatin Dhankhar
>>
>>
>> On Tue, Apr 11, 2017 at 6:14 AM, Tony Anderson < <tony_anderson at usa.net>
>> tony_anderson at usa.net> wrote:
>>
>>> If you are talking about IRC as a place to meet Sugar community members,
>>> use the freenode #sugar. This is probably most active from 8-17 EST
>>> (UTC-5). I am currently in the Philippines which is UTC+ 7.
>>>
>>> Localization of Python activities is done by Pootle, when implemented by
>>> the developer. The developer does something like the following:
>>>
>>>         from gettext import gettext as _
>>>
>>>         self.copy.set_tooltip(_('Copy'))
>>>
>>> In this way, all text displayed is taken from a po file based on the
>>> locale (e.g. en.po or hi.po). This is a simplification as the actual file
>>> is compressed: en.mo, hi.mo. These files are in the activity bundle. The
>>> detail is that when a new version is released, there is a master file:
>>> Paint.pot from which the local language files are built. This needs to be
>>> submitted to translate.sugarlabs.org which maintains a copy. However,
>>> then the localized version needs to be added back to the bundle. However,
>>> the localizations can take months for 100 languages so how synchronize the
>>> po directory with the activity release is difficult.
>>>
>>> The sugar3 vs sugar issue is decided. The community wants to move to
>>> sugar3 (gtk3). The problem is that less that 20% of the activities have
>>> been converted.
>>> The ones that have been converted are low hanging fruit. The unconverted
>>> ones may require intensive work (gimp which developed gtk originally has
>>> not made the conversion).
>>>
>>> One thing you could look at. On activities.sugarlabs,org, can you
>>> determine from Remora where the metadata is stored? I assume a db.
>>> Currently I am thinking to use BeautifulSoup to scrape the site to get that
>>> data, but it would be much easier to access the data directly.
>>>
>>> Yesterday afternoon, the ISP restored service. Last time it died after
>>> two days, but I am keeping my fingers crossed. I am attaching the django
>>> stuff.
>>>
>>> Tony
>>>
>>>
>>> On 04/11/2017 01:36 AM, Jatin Dhankhar wrote:
>>>
>>> Hi Tony,
>>>
>>>
>>>> Normally, we use  <http://chat.sugarlabs.org>http://chat.sugarlabs.org or
>>>> on freenode: sugar-meeting or sugar-newbies. These are logged sites so that
>>>> there is a record. The second is more appropriate since sugar-meeting is
>>>> used for SLOB meetings and the like. The real problem with IRC is time
>>>> zones. Email has the advantage that either party can send or receive at any
>>>> time. Last year with a GSOC mentee we used sugar-newbies by arranging a
>>>> specific meeting time in advance.
>>>
>>> Yes, that is correct, main issue in communication barrier is due to
>>> timezone issues. Since most of the people are familiar and are available on
>>> IRC, it's seems to be the primary channel of communication along with
>>> mailing lists and email. But since you said we can use anything else,
>>> giving Slack a try won't hurt (if issue is about not using closed source
>>> software then IRC is fine, or we can try Mattermost
>>> <https://about.mattermost.com/>).
>>>
>>>  Another part of the process is how to update 'translate.sugarlabs.org'
>>>> with the corresponding POT file to enable localization. We can get help
>>>> from Chris Leonard on this.
>>>
>>>
>>> I am not aware on how localization works. Do we need to download
>>> relevant files and bundle them with the acitvity before making it available
>>> on ASLO ?
>>>
>>> I have my Django version available - but the internet problems here are
>>>> still unresolved. The technician is supposed to make another visit today to
>>>> see what is wrong with our connection. Let me know if and when you think
>>>> this will be useful to you.
>>>
>>> Let me know when your connection is stable and I would start.
>>>  What  are the things you need me to do in the meantime ?
>>>
>>> One open issue is sugar3 vs sugar. Currently two versions of Sugar are
>>>> released. The sugar version supports gtk while sugar3 supports gtk3.
>>>> Unfortunately, gtk3 was developed to be totally incompatible with gtk. For
>>>> example, incorporation of one gtk3 feature requires that all direct and
>>>> indirect references to gtk be removed or the activity will throw an
>>>> exception. Several of the gtk3 conversions failed to meet this test and so
>>>> fail. The issue is whether curated activities be limited to ones converted
>>>> to gtk3. The positive is that Sugar could revert to releasing and
>>>> maintaining only a single version. The downside is that 100 or more
>>>> activities will no longer be available. Specifically, in our implementation
>>>> of ASLO, we need to show which versions of an activity work on which
>>>> versions of Sugar (e.g. i86, arm, amd64, sugar or sugar3, and so on). We
>>>> also need to show which ones support localization. There are many English
>>>> activities and many Spanish activities that make no provision for
>>>> localization. Luckily there are many that have no language component.
>>>> However, for many of these, some kind of help is needed to convey the way
>>>> the activity works.
>>>
>>>
>>> Some people believe GTK3 is slightly better
>>> <https://www.reddit.com/r/linux/comments/3e3q8n/is_there_a_technical_reason_why_gtk3_is_better/> and
>>> I think GTK3 will stay but that should be asked in community and voted upon
>>> and taking in considerations cost of development and porting, only a
>>> discussion will help in this one.
>>>
>>> You are wading into a deep and vast body of water!
>>>
>>> As long as I have something to hold onto, I will not drown 😅
>>>
>>> Thanks,
>>> Jatin Dhankhar
>>>
>>> On Mon, Apr 10, 2017 at 9:11 AM, Tony Anderson < <tony_anderson at usa.net>
>>> tony_anderson at usa.net> wrote:
>>>
>>>> Hi, Jatin
>>>>
>>>> Normally, we use <http://chat.sugarlabs.org>http://chat.sugarlabs.org
>>>> or on freenode: sugar-meeting or sugar-newbies. These are logged sites so
>>>> that there is a record. The second is more appropriate since sugar-meeting
>>>> is used for SLOB meetings and the like. The real problem with IRC is time
>>>> zones. Email has the advantage that either party can send or receive at any
>>>> time. Last year with a GSOC mentee we used sugar-newbies by arranging a
>>>> specific meeting time in advance.
>>>>
>>>> I haven't heard from Walter, but my preference would be to use the
>>>> Sugarlabs server since the content is largely already there and it would be
>>>> easier to make it the official site if that were decided. So in the short
>>>> run, I think you should do whatever is best for your own development
>>>> process.
>>>>
>>>> I am not sure how CI fits into this. If the activity development is
>>>> done on GitHub, then the deployment model is to run setup.py to create an
>>>> xo bundle and then copy that bundle to the appropriate location in the
>>>> download.sugarlabs.org tree. Assuming the update results from a PR,
>>>> the deployer would need to update the activity information on ASLO
>>>> appropriately. However, that process depends on where that data (metadata)
>>>> is stored. Another part of the process is how to update '
>>>> translate.sugarlabs.org' with the corresponding POT file to enable
>>>> localization. We can get help from Chris Leonard on this.
>>>>
>>>> I have my Django version available - but the internet problems here are
>>>> still unresolved. The technician is supposed to make another visit today to
>>>> see what is wrong with our connection. Let me know if and when you think
>>>> this will be useful to you.
>>>>
>>>> I have now tested most of the activities (~400). I was optimistic in
>>>> the number that work out of the box. However, a part of this is running
>>>> them in the Ubuntu version of Sugar (amd64). There are many activities
>>>> which launch object code (mostly c) which is dependent on the architecture.
>>>> I am now trying to repeat the tests on an XO-1.75. One issue on Ubuntu is
>>>> that many activities assume a 1200x900 screen and so on a 1024X768 screen
>>>> overflow. This makes some of the games unusable since part of the controls
>>>> are off the screen. Because of the internet problems, the untested
>>>> activities tend to be new ones since I was using my local repository which
>>>> is a snapshot taken several months ago. The other group are the GCompris
>>>> activities (about 70).
>>>>
>>>> My intent is to build a 'curated' repository of activities known to
>>>> work and be usable on the XO and on Ubuntu (or such other platform that
>>>> Sugar may choose to support). Most of the currently not work activities
>>>> have software dependencies no longer included in the current Sugar release.
>>>> So the curated library will grow as activities are repaired over time.
>>>>
>>>> One open issue is sugar3 vs sugar. Currently two versions of Sugar are
>>>> released. The sugar version supports gtk while sugar3 supports gtk3.
>>>> Unfortunately, gtk3 was developed to be totally incompatible with gtk. For
>>>> example, incorporation of one gtk3 feature requires that all direct and
>>>> indirect references to gtk be removed or the activity will throw an
>>>> exception. Several of the gtk3 conversions failed to meet this test and so
>>>> fail. The issue is whether curated activities be limited to ones converted
>>>> to gtk3. The positive is that Sugar could revert to releasing and
>>>> maintaining only a single version. The downside is that 100 or more
>>>> activities will no longer be available. Specifically, in our implementation
>>>> of ASLO, we need to show which versions of an activity work on which
>>>> versions of Sugar (e.g. i86, arm, amd64, sugar or sugar3, and so on). We
>>>> also need to show which ones support localization. There are many English
>>>> activities and many Spanish activities that make no provision for
>>>> localization. Luckily there are many that have no language component.
>>>> However, for many of these, some kind of help is needed to convey the way
>>>> the activity works.
>>>>
>>>> You are wading into a deep and vast body of water!
>>>>
>>>> Tony
>>>>
>>>>
>>>> On 04/10/2017 12:00 AM, Jatin Dhankhar wrote:
>>>>
>>>> Hi,
>>>> Sorry for the delay. I went through the polls tutorials and I think I
>>>> am getting hang of Django. I have one query that is out of context, what is
>>>> your IRC setup ? IRC doesn't allow message to be delivered or stored once
>>>> either party is offline, people login through a external server for IRC's
>>>> to maintain their availability in a channel. May I suggest something like
>>>> Slack or Flock for communication. IRC is good for quick and fast connection
>>>> but Slack and alternatives allow easy communication. (Just a suggestion,
>>>> though)
>>>>
>>>> Should I  deploy the same polls app on DigitalOcean along with CI
>>>> pipeline and branching model in the meantime with code hosted on Github ?
>>>>
>>>> Thanks,
>>>> Jatin Dhankhar
>>>>
>>>> On Sat, Apr 8, 2017 at 6:28 AM, Tony Anderson < <tony_anderson at usa.net>
>>>> tony_anderson at usa.net> wrote:
>>>>
>>>>> Hi, Jatin
>>>>>
>>>>> I abbreviated this thread because I am getting complaints from the
>>>>> list that the emails are too long.
>>>>>
>>>>> Yesterday, I was able to complete a set of 25 activities to use as a
>>>>> test base. The Django app now has two views. First is a display of all of
>>>>> the
>>>>> activities with the icon and name. A link for each opens the full
>>>>> activity page. The data for the views is in a file which is a list of
>>>>> jsons, one json per line.
>>>>> The  views.py reads this file and builds the view from the jsons.
>>>>>
>>>>> So the app consists of:
>>>>>     urls.py
>>>>>     views.py
>>>>>     templates/
>>>>>         base.html
>>>>>         list.html
>>>>>         activity.html
>>>>>
>>>>> with the data:
>>>>>         fixtures/activities.json
>>>>>         icons/
>>>>>         xo/
>>>>>
>>>>> The icons directory has the activity icons (*.svg)
>>>>> The xo directory has the activity bundles (*.xo)
>>>>>
>>>>> Meanwhile, I am having more troubles with the ISP (a new one). When
>>>>> that gets resolved, I should be able to send the above to you.
>>>>>
>>>>> I have also mentioned our dialog to Walter Bender suggesting that you
>>>>> could set up your working model as activities3.sugarlabs.org. This
>>>>> would give you a permanent place to do the development and make your
>>>>> project visible to the community so that we can obtain feedback. I aslo
>>>>> suggested that the project repository could be placed on GitHub so that
>>>>> again the community can comment and potentially contribute.
>>>>>
>>>>> Tony
>>>>> On 04/07/2017 12:23 AM, Jatin Dhankhar wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am setting up Django now and going through some tutorials. I will
>>>>> setup the code with a very basic and barebones version of what we want to
>>>>> achieve and put it in a private repo. We can even host it online, since I
>>>>> have some DigitalOcean credit left. Will keep you posted.
>>>>>
>>>>> Thanks.
>>>>> Jatin Dhankhar
>>>>>
>>>>> On Thu, Apr 6, 2017 at 7:11 AM, Tony Anderson <
>>>>> <tony_anderson at usa.net>tony_anderson at usa.net> wrote:
>>>>>
>>>>>> Hi, Jatin
>>>>>>
>>>>>> In setting up Django, I think you will do it on your computer not on
>>>>>> a separate server. Django handles that through its own server and has
>>>>>> sgqlite as a database built-in. A good first start would be to set up the
>>>>>> tutorial app ( <https://www.djangoproject.com>
>>>>>> https://www.djangoproject.com/). Essentially you will need to
>>>>>> install django and then set up the tutorial app 'poll'. Working through
>>>>>> this tutorial will help a lot later on. The key point is that after initial
>>>>>> setup, there are three important elements: url.py which defines the urls to
>>>>>> access the application, views.py which is the python code that responds to
>>>>>> a request by accessing the db and delivering the relevant information to a
>>>>>> template. The template is an html file with variables of the form {{
>>>>>> activity.name }} transferring information from the view to the
>>>>>> template.
>>>>>>
>>>>>> What I have is an application: aslo. Once you have Django installed
>>>>>> and have run the tutorial app through Django's server, you'll be ready.
>>>>>> Essentially, you will only need to add aslo as a second app in the
>>>>>> settings. One possible confusion is that DJango lives in a project -
>>>>>> essentially a directory containing its manage.py admin interface. In the
>>>>>> directory is another directory of the same name with the settings.py and
>>>>>> url.py. The poll app is a directory in the top-level alongside the inner
>>>>>> directory with the project name.
>>>>>>
>>>>>> For example, my project is schoolsite. So my setup looks like:
>>>>>>
>>>>>> /library/schoolsite/schoolsite
>>>>>> /library/schoolsite/aslo
>>>>>>
>>>>>> Meanwhile my code links directly to an activity page with no index.
>>>>>> I'll add an index so the essential structure will be there. I'll also
>>>>>> include the 'fixtures' to set up a 25 activity capability. Fixtures are csv
>>>>>> files from which the database (metadata) can be loaded.
>>>>>>
>>>>>> Tony
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/aslo/attachments/20170413/69a1454f/attachment-0001.html>