[ASLO] ASLO build and deployment process (Jatin Dhankhar)

[Jatin Dhankhar]

Hi,

I think we need to agree on the use of IRC. If you want to communicate with
> members of the community, you must go where they are (#sugar). If you want
> a one-to-one meeting on IRC with me, I would suggest #sugar-newbies. It is
> normally dormant but works well and saves a log for later review. It worked
> well for meetings with Utkarsh Tiwari during last year's GSOC.

Sure, whatever works :). What is your IRC nickname ?

There are two things that you will need to have local to the django
> project. First is the directory download.sugarlabs.org which has the
> Sugar activity bundles

Do I need to mirror the whole setup/directory ?

When talking about scraping you probably meant
http://activities.sugarlabs.org instead of http://download.sugarlabs.org/,
right ?
Also for scraping, Scrapy <https://scrapy.org/> seems to more popular than
beautifulsoup ?

Also a big thanks for including Walter in the discussion :D

- Jatin Dhankhar


On Wed, Apr 12, 2017 at 6:30 AM, Tony Anderson <tony_anderson at usa.net>
wrote:

> Hi, Jatin
>
> I think we need to agree on the use of IRC. If you want to communicate
> with members of the community, you must go where they are (#sugar). If you
> want a one-to-one meeting on IRC with me, I would suggest #sugar-newbies.
> It is normally dormant but works well and saves a log for later review. It
> worked well for meetings with Utkarsh Tiwari during last year's GSOC.
>
> There are two things that you will need to have local to the django
> project. First is the directory download.sugarlabs.org which has the
> Sugar activity bundles. The second is the 'metadata' in the mysql db. For
> scraping, I would recommend BeautifulSoup (bs4). The trick will be to
> decide what data we want to capture and add to the json.
>
> The json fields in activities.json are ones I chose for the minimal
> system. You may want to include other information such as the number of
> downloads, which collections (should be entered as tags in a tag-field) and
> so on. One item I have referred to as flags (I marked some as X but don't
> remember what that meant, oh well). The intent is to record the platforms
> where the activity works. We also should provide links to the homepage,
> repository page, and update page (whatever that is). I think if you have a
> working scrape tool, the data it collects can be expanded as needed
> (assuming the tool runs in a reasonable time).
>
> Naturally, it would be easier if you have access to the db directly.
>
> Tony
>
>
> On 04/12/2017 01:40 AM, Jatin Dhankhar wrote:
>
> One thing you could look at. On activities.sugarlabs,org, can you
>> determine from Remora where the metadata is stored? I assume a db.
>> Currently I am thinking to use BeautifulSoup to scrape the site to get that
>> data, but it would be much easier to access the data directly.
>
> As per wiki https://wiki.mozilla.org/Update:Remora_Server_
> Requirements#SVN.2C_DB_and_app_config data is stored in mysql database. I
> don't have access to the production server where ASLO is currently running,
> following file https://github.com/sugarlabs/aslo/blob/master/
> aslo/db-update.sh#L9 confirms that data is stored in a mysql db. However
> it would be interesting and fun to scrape the data from live site. I would
> do that.
>
> Thanks will poke around the code, looks to me it's a django app and I have
> to mount it on my django project, thanks :)
>
> If you are talking about IRC as a place to meet Sugar community members,
>> use the freenode #sugar. This is probably most active from 8-17 EST
>> (UTC-5). I am currently in the Philippines which is UTC+ 7.
>
>
>  Yes, tried that.  https://gitter.im fits in naturally with Github
> (really sorry for suggesting a new mode of communication everyday) 😅
>
> - Jatin Dhankhar
>
>
> On Tue, Apr 11, 2017 at 6:14 AM, Tony Anderson <tony_anderson at usa.net>
> wrote:
>
>> If you are talking about IRC as a place to meet Sugar community members,
>> use the freenode #sugar. This is probably most active from 8-17 EST
>> (UTC-5). I am currently in the Philippines which is UTC+ 7.
>>
>> Localization of Python activities is done by Pootle, when implemented by
>> the developer. The developer does something like the following:
>>
>>         from gettext import gettext as _
>>
>>         self.copy.set_tooltip(_('Copy'))
>>
>> In this way, all text displayed is taken from a po file based on the
>> locale (e.g. en.po or hi.po). This is a simplification as the actual file
>> is compressed: en.mo, hi.mo. These files are in the activity bundle. The
>> detail is that when a new version is released, there is a master file:
>> Paint.pot from which the local language files are built. This needs to be
>> submitted to translate.sugarlabs.org which maintains a copy. However,
>> then the localized version needs to be added back to the bundle. However,
>> the localizations can take months for 100 languages so how synchronize the
>> po directory with the activity release is difficult.
>>
>> The sugar3 vs sugar issue is decided. The community wants to move to
>> sugar3 (gtk3). The problem is that less that 20% of the activities have
>> been converted.
>> The ones that have been converted are low hanging fruit. The unconverted
>> ones may require intensive work (gimp which developed gtk originally has
>> not made the conversion).
>>
>> One thing you could look at. On activities.sugarlabs,org, can you
>> determine from Remora where the metadata is stored? I assume a db.
>> Currently I am thinking to use BeautifulSoup to scrape the site to get that
>> data, but it would be much easier to access the data directly.
>>
>> Yesterday afternoon, the ISP restored service. Last time it died after
>> two days, but I am keeping my fingers crossed. I am attaching the django
>> stuff.
>>
>> Tony
>>
>>
>> On 04/11/2017 01:36 AM, Jatin Dhankhar wrote:
>>
>> Hi Tony,
>>
>>
>>> Normally, we use  <http://chat.sugarlabs.org>http://chat.sugarlabs.org or
>>> on freenode: sugar-meeting or sugar-newbies. These are logged sites so that
>>> there is a record. The second is more appropriate since sugar-meeting is
>>> used for SLOB meetings and the like. The real problem with IRC is time
>>> zones. Email has the advantage that either party can send or receive at any
>>> time. Last year with a GSOC mentee we used sugar-newbies by arranging a
>>> specific meeting time in advance.
>>
>> Yes, that is correct, main issue in communication barrier is due to
>> timezone issues. Since most of the people are familiar and are available on
>> IRC, it's seems to be the primary channel of communication along with
>> mailing lists and email. But since you said we can use anything else,
>> giving Slack a try won't hurt (if issue is about not using closed source
>> software then IRC is fine, or we can try Mattermost
>> <https://about.mattermost.com/>).
>>
>>  Another part of the process is how to update 'translate.sugarlabs.org'
>>> with the corresponding POT file to enable localization. We can get help
>>> from Chris Leonard on this.
>>
>>
>> I am not aware on how localization works. Do we need to download relevant
>> files and bundle them with the acitvity before making it available on ASLO
>> ?
>>
>> I have my Django version available - but the internet problems here are
>>> still unresolved. The technician is supposed to make another visit today to
>>> see what is wrong with our connection. Let me know if and when you think
>>> this will be useful to you.
>>
>> Let me know when your connection is stable and I would start.
>>  What  are the things you need me to do in the meantime ?
>>
>> One open issue is sugar3 vs sugar. Currently two versions of Sugar are
>>> released. The sugar version supports gtk while sugar3 supports gtk3.
>>> Unfortunately, gtk3 was developed to be totally incompatible with gtk. For
>>> example, incorporation of one gtk3 feature requires that all direct and
>>> indirect references to gtk be removed or the activity will throw an
>>> exception. Several of the gtk3 conversions failed to meet this test and so
>>> fail. The issue is whether curated activities be limited to ones converted
>>> to gtk3. The positive is that Sugar could revert to releasing and
>>> maintaining only a single version. The downside is that 100 or more
>>> activities will no longer be available. Specifically, in our implementation
>>> of ASLO, we need to show which versions of an activity work on which
>>> versions of Sugar (e.g. i86, arm, amd64, sugar or sugar3, and so on). We
>>> also need to show which ones support localization. There are many English
>>> activities and many Spanish activities that make no provision for
>>> localization. Luckily there are many that have no language component.
>>> However, for many of these, some kind of help is needed to convey the way
>>> the activity works.
>>
>>
>> Some people believe GTK3 is slightly better
>> <https://www.reddit.com/r/linux/comments/3e3q8n/is_there_a_technical_reason_why_gtk3_is_better/> and
>> I think GTK3 will stay but that should be asked in community and voted upon
>> and taking in considerations cost of development and porting, only a
>> discussion will help in this one.
>>
>> You are wading into a deep and vast body of water!
>>
>> As long as I have something to hold onto, I will not drown 😅
>>
>> Thanks,
>> Jatin Dhankhar
>>
>> On Mon, Apr 10, 2017 at 9:11 AM, Tony Anderson < <tony_anderson at usa.net>
>> tony_anderson at usa.net> wrote:
>>
>>> Hi, Jatin
>>>
>>> Normally, we use http://chat.sugarlabs.org or on freenode:
>>> sugar-meeting or sugar-newbies. These are logged sites so that there is a
>>> record. The second is more appropriate since sugar-meeting is used for SLOB
>>> meetings and the like. The real problem with IRC is time zones. Email has
>>> the advantage that either party can send or receive at any time. Last year
>>> with a GSOC mentee we used sugar-newbies by arranging a specific meeting
>>> time in advance.
>>>
>>> I haven't heard from Walter, but my preference would be to use the
>>> Sugarlabs server since the content is largely already there and it would be
>>> easier to make it the official site if that were decided. So in the short
>>> run, I think you should do whatever is best for your own development
>>> process.
>>>
>>> I am not sure how CI fits into this. If the activity development is done
>>> on GitHub, then the deployment model is to run setup.py to create an xo
>>> bundle and then copy that bundle to the appropriate location in the
>>> download.sugarlabs.org tree. Assuming the update results from a PR, the
>>> deployer would need to update the activity information on ASLO
>>> appropriately. However, that process depends on where that data (metadata)
>>> is stored. Another part of the process is how to update '
>>> translate.sugarlabs.org' with the corresponding POT file to enable
>>> localization. We can get help from Chris Leonard on this.
>>>
>>> I have my Django version available - but the internet problems here are
>>> still unresolved. The technician is supposed to make another visit today to
>>> see what is wrong with our connection. Let me know if and when you think
>>> this will be useful to you.
>>>
>>> I have now tested most of the activities (~400). I was optimistic in the
>>> number that work out of the box. However, a part of this is running them in
>>> the Ubuntu version of Sugar (amd64). There are many activities which launch
>>> object code (mostly c) which is dependent on the architecture. I am now
>>> trying to repeat the tests on an XO-1.75. One issue on Ubuntu is that many
>>> activities assume a 1200x900 screen and so on a 1024X768 screen overflow.
>>> This makes some of the games unusable since part of the controls are off
>>> the screen. Because of the internet problems, the untested activities tend
>>> to be new ones since I was using my local repository which is a snapshot
>>> taken several months ago. The other group are the GCompris activities
>>> (about 70).
>>>
>>> My intent is to build a 'curated' repository of activities known to work
>>> and be usable on the XO and on Ubuntu (or such other platform that Sugar
>>> may choose to support). Most of the currently not work activities have
>>> software dependencies no longer included in the current Sugar release. So
>>> the curated library will grow as activities are repaired over time.
>>>
>>> One open issue is sugar3 vs sugar. Currently two versions of Sugar are
>>> released. The sugar version supports gtk while sugar3 supports gtk3.
>>> Unfortunately, gtk3 was developed to be totally incompatible with gtk. For
>>> example, incorporation of one gtk3 feature requires that all direct and
>>> indirect references to gtk be removed or the activity will throw an
>>> exception. Several of the gtk3 conversions failed to meet this test and so
>>> fail. The issue is whether curated activities be limited to ones converted
>>> to gtk3. The positive is that Sugar could revert to releasing and
>>> maintaining only a single version. The downside is that 100 or more
>>> activities will no longer be available. Specifically, in our implementation
>>> of ASLO, we need to show which versions of an activity work on which
>>> versions of Sugar (e.g. i86, arm, amd64, sugar or sugar3, and so on). We
>>> also need to show which ones support localization. There are many English
>>> activities and many Spanish activities that make no provision for
>>> localization. Luckily there are many that have no language component.
>>> However, for many of these, some kind of help is needed to convey the way
>>> the activity works.
>>>
>>> You are wading into a deep and vast body of water!
>>>
>>> Tony
>>>
>>>
>>> On 04/10/2017 12:00 AM, Jatin Dhankhar wrote:
>>>
>>> Hi,
>>> Sorry for the delay. I went through the polls tutorials and I think I am
>>> getting hang of Django. I have one query that is out of context, what is
>>> your IRC setup ? IRC doesn't allow message to be delivered or stored once
>>> either party is offline, people login through a external server for IRC's
>>> to maintain their availability in a channel. May I suggest something like
>>> Slack or Flock for communication. IRC is good for quick and fast connection
>>> but Slack and alternatives allow easy communication. (Just a suggestion,
>>> though)
>>>
>>> Should I  deploy the same polls app on DigitalOcean along with CI
>>> pipeline and branching model in the meantime with code hosted on Github ?
>>>
>>> Thanks,
>>> Jatin Dhankhar
>>>
>>> On Sat, Apr 8, 2017 at 6:28 AM, Tony Anderson < <tony_anderson at usa.net>
>>> tony_anderson at usa.net> wrote:
>>>
>>>> Hi, Jatin
>>>>
>>>> I abbreviated this thread because I am getting complaints from the list
>>>> that the emails are too long.
>>>>
>>>> Yesterday, I was able to complete a set of 25 activities to use as a
>>>> test base. The Django app now has two views. First is a display of all of
>>>> the
>>>> activities with the icon and name. A link for each opens the full
>>>> activity page. The data for the views is in a file which is a list of
>>>> jsons, one json per line.
>>>> The  views.py reads this file and builds the view from the jsons.
>>>>
>>>> So the app consists of:
>>>>     urls.py
>>>>     views.py
>>>>     templates/
>>>>         base.html
>>>>         list.html
>>>>         activity.html
>>>>
>>>> with the data:
>>>>         fixtures/activities.json
>>>>         icons/
>>>>         xo/
>>>>
>>>> The icons directory has the activity icons (*.svg)
>>>> The xo directory has the activity bundles (*.xo)
>>>>
>>>> Meanwhile, I am having more troubles with the ISP (a new one). When
>>>> that gets resolved, I should be able to send the above to you.
>>>>
>>>> I have also mentioned our dialog to Walter Bender suggesting that you
>>>> could set up your working model as activities3.sugarlabs.org. This
>>>> would give you a permanent place to do the development and make your
>>>> project visible to the community so that we can obtain feedback. I aslo
>>>> suggested that the project repository could be placed on GitHub so that
>>>> again the community can comment and potentially contribute.
>>>>
>>>> Tony
>>>> On 04/07/2017 12:23 AM, Jatin Dhankhar wrote:
>>>>
>>>> Hi,
>>>>
>>>> I am setting up Django now and going through some tutorials. I will
>>>> setup the code with a very basic and barebones version of what we want to
>>>> achieve and put it in a private repo. We can even host it online, since I
>>>> have some DigitalOcean credit left. Will keep you posted.
>>>>
>>>> Thanks.
>>>> Jatin Dhankhar
>>>>
>>>> On Thu, Apr 6, 2017 at 7:11 AM, Tony Anderson < <tony_anderson at usa.net>
>>>> tony_anderson at usa.net> wrote:
>>>>
>>>>> Hi, Jatin
>>>>>
>>>>> In setting up Django, I think you will do it on your computer not on a
>>>>> separate server. Django handles that through its own server and has sgqlite
>>>>> as a database built-in. A good first start would be to set up the tutorial
>>>>> app ( <https://www.djangoproject.com>https://www.djangoproject.com/).
>>>>> Essentially you will need to install django and then set up the tutorial
>>>>> app 'poll'. Working through this tutorial will help a lot later on. The key
>>>>> point is that after initial setup, there are three important elements:
>>>>> url.py which defines the urls to access the application, views.py which is
>>>>> the python code that responds to a request by accessing the db and
>>>>> delivering the relevant information to a template. The template is an html
>>>>> file with variables of the form {{ activity.name }} transferring
>>>>> information from the view to the template.
>>>>>
>>>>> What I have is an application: aslo. Once you have Django installed
>>>>> and have run the tutorial app through Django's server, you'll be ready.
>>>>> Essentially, you will only need to add aslo as a second app in the
>>>>> settings. One possible confusion is that DJango lives in a project -
>>>>> essentially a directory containing its manage.py admin interface. In the
>>>>> directory is another directory of the same name with the settings.py and
>>>>> url.py. The poll app is a directory in the top-level alongside the inner
>>>>> directory with the project name.
>>>>>
>>>>> For example, my project is schoolsite. So my setup looks like:
>>>>>
>>>>> /library/schoolsite/schoolsite
>>>>> /library/schoolsite/aslo
>>>>>
>>>>> Meanwhile my code links directly to an activity page with no index.
>>>>> I'll add an index so the essential structure will be there. I'll also
>>>>> include the 'fixtures' to set up a 25 activity capability. Fixtures are csv
>>>>> files from which the database (metadata) can be loaded.
>>>>>
>>>>> Tony
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/aslo/attachments/20170412/0bcb59af/attachment-0001.html>