[ASLO] ASLO build and deployment process (Jatin Dhankhar)

Tony Anderson tony_anderson at usa.net
Wed Apr 12 20:20:10 EDT 2017


Hi Jatin

At the moment, I think you should focus on the 25 activity test set and 
getting it working with django. I find it is much more productive to 
move from one working version to another with added capability. If we 
can arrange to host the development on a Sugar Labs server, it should 
make both the repository itself (download.sugarlabs.org) and the ASLO 
metadata directly accessible.

My IRC nickname is tony37. However, I am never on #sugar (a twelve hour 
time difference so the normal participants are in bed). Last year, 
Utkarsh and I met whenever he felt the need to discuss a technical issue 
in detail. We limited the meetings to one hour. That proved quite 
productive.

Tony

On 04/13/2017 01:14 AM, Jatin Dhankhar wrote:
> Hi,
>
>     I think we need to agree on the use of IRC. If you want to
>     communicate with members of the community, you must go where they
>     are (#sugar). If you want a one-to-one meeting on IRC with me, I
>     would suggest #sugar-newbies. It is normally dormant but works
>     well and saves a log for later review. It worked well for meetings
>     with Utkarsh Tiwari during last year's GSOC.
>
> Sure, whatever works :). What is your IRC nickname ?
>
>     There are two things that you will need to have local to the
>     django project. First is the directory download.sugarlabs.org
>     <http://download.sugarlabs.org/> which has the Sugar activity bundles
>
> Do I need to mirror the whole setup/directory ?
>
> When talking about scraping you probably meant 
> http://activities.sugarlabs.org instead of 
> http://download.sugarlabs.org/, right ?
> Also for scraping, Scrapy <https://scrapy.org/> seems to more popular 
> than beautifulsoup ?
>
> Also a big thanks for including Walter in the discussion :D
>
> - Jatin Dhankhar
>
> On Wed, Apr 12, 2017 at 6:30 AM, Tony Anderson <tony_anderson at usa.net 
> <mailto:tony_anderson at usa.net>> wrote:
>
>     Hi, Jatin
>
>     I think we need to agree on the use of IRC. If you want to
>     communicate with members of the community, you must go where they
>     are (#sugar). If you want a one-to-one meeting on IRC with me, I
>     would suggest #sugar-newbies. It is normally dormant but works
>     well and saves a log for later review. It worked well for meetings
>     with Utkarsh Tiwari during last year's GSOC.
>
>     There are two things that you will need to have local to the
>     django project. First is the directory download.sugarlabs.org
>     <http://download.sugarlabs.org> which has the Sugar activity
>     bundles. The second is the 'metadata' in the mysql db. For
>     scraping, I would recommend BeautifulSoup (bs4). The trick will be
>     to decide what data we want to capture and add to the json.
>
>     The json fields in activities.json are ones I chose for the
>     minimal system. You may want to include other information such as
>     the number of downloads, which collections (should be entered as
>     tags in a tag-field) and so on. One item I have referred to as
>     flags (I marked some as X but don't remember what that meant, oh
>     well). The intent is to record the platforms where the activity
>     works. We also should provide links to the homepage, repository
>     page, and update page (whatever that is). I think if you have a
>     working scrape tool, the data it collects can be expanded as
>     needed (assuming the tool runs in a reasonable time).
>
>     Naturally, it would be easier if you have access to the db directly.
>
>     Tony
>
>
>     On 04/12/2017 01:40 AM, Jatin Dhankhar wrote:
>>
>>         One thing you could look at. On activities.sugarlabs,org, can
>>         you determine from Remora where the metadata is stored? I
>>         assume a db. Currently I am thinking to use BeautifulSoup to
>>         scrape the site to get that data, but it would be much easier
>>         to access the data directly.
>>
>>     As per wiki
>>     https://wiki.mozilla.org/Update:Remora_Server_Requirements#SVN.2C_DB_and_app_config
>>     <https://wiki.mozilla.org/Update:Remora_Server_Requirements#SVN.2C_DB_and_app_config> data
>>     is stored in mysql database. I don't have access to the
>>     production server where ASLO is currently running, following file
>>     https://github.com/sugarlabs/aslo/blob/master/aslo/db-update.sh#L9 <https://github.com/sugarlabs/aslo/blob/master/aslo/db-update.sh#L9> confirms
>>     that data is stored in a mysql db. However it would be
>>     interesting and fun to scrape the data from live site. I would do
>>     that.
>>
>>     Thanks will poke around the code, looks to me it's a django app
>>     and I have to mount it on my django project, thanks :)
>>
>>         If you are talking about IRC as a place to meet Sugar
>>         community members, use the freenode #sugar. This is probably
>>         most active from 8-17 EST (UTC-5). I am currently in the
>>         Philippines which is UTC+ 7. 
>>
>>
>>      Yes, tried that. https://gitter.im fits in naturally with Github
>>     (really sorry for suggesting a new mode of communication everyday) 😅
>>
>>     - Jatin Dhankhar
>>
>>
>>     On Tue, Apr 11, 2017 at 6:14 AM, Tony Anderson
>>     <tony_anderson at usa.net <mailto:tony_anderson at usa.net>> wrote:
>>
>>         If you are talking about IRC as a place to meet Sugar
>>         community members, use the freenode #sugar. This is probably
>>         most active from 8-17 EST (UTC-5). I am currently in the
>>         Philippines which is UTC+ 7.
>>
>>         Localization of Python activities is done by Pootle, when
>>         implemented by the developer. The developer does something
>>         like the following:
>>
>>                 from gettext import gettext as _
>>
>>                 self.copy.set_tooltip(_('Copy'))
>>
>>         In this way, all text displayed is taken from a po file based
>>         on the locale (e.g. en.po or hi.po). This is a simplification
>>         as the actual file is compressed: en.mo, hi.mo. These files
>>         are in the activity bundle. The detail is that when a new
>>         version is released, there is a master file: Paint.pot from
>>         which the local language files are built. This needs to be
>>         submitted to translate.sugarlabs.org
>>         <http://translate.sugarlabs.org> which maintains a copy.
>>         However, then the localized version needs to be added back to
>>         the bundle. However, the localizations can take months for
>>         100 languages so how synchronize the po directory with the
>>         activity release is difficult.
>>
>>         The sugar3 vs sugar issue is decided. The community wants to
>>         move to sugar3 (gtk3). The problem is that less that 20% of
>>         the activities have been converted.
>>         The ones that have been converted are low hanging fruit. The
>>         unconverted ones may require intensive work (gimp which
>>         developed gtk originally has not made the conversion).
>>
>>         One thing you could look at. On activities.sugarlabs,org, can
>>         you determine from Remora where the metadata is stored? I
>>         assume a db. Currently I am thinking to use BeautifulSoup to
>>         scrape the site to get that data, but it would be much easier
>>         to access the data directly.
>>
>>         Yesterday afternoon, the ISP restored service. Last time it
>>         died after two days, but I am keeping my fingers crossed. I
>>         am attaching the django stuff.
>>
>>         Tony
>>
>>
>>         On 04/11/2017 01:36 AM, Jatin Dhankhar wrote:
>>>         Hi Tony,
>>>
>>>             Normally, we use http://chat.sugarlabs.orgor on
>>>             freenode: sugar-meeting or sugar-newbies. These are
>>>             logged sites so that there is a record. The second is
>>>             more appropriate since sugar-meeting is used for SLOB
>>>             meetings and the like. The real problem with IRC is time
>>>             zones. Email has the advantage that either party can
>>>             send or receive at any time. Last year with a GSOC
>>>             mentee we used sugar-newbies by arranging a specific
>>>             meeting time in advance. 
>>>
>>>         Yes, that is correct, main issue in communication barrier is
>>>         due to timezone issues. Since most of the people are
>>>         familiar and are available on IRC, it's seems to be the
>>>         primary channel of communication along with mailing lists
>>>         and email. But since you said we can use anything else,
>>>         giving Slack a try won't hurt (if issue is about not using
>>>         closed source software then IRC is fine, or we can try
>>>         Mattermost <https://about.mattermost.com/>).
>>>
>>>              Another part of the process is how to update
>>>             'translate.sugarlabs.org
>>>             <http://translate.sugarlabs.org/>' with the
>>>             corresponding POT file to enable localization. We can
>>>             get help from Chris Leonard on this. 
>>>
>>>
>>>         I am not aware on how localization works. Do we need to
>>>         download relevant files and bundle them with the acitvity
>>>         before making it available on ASLO ?
>>>
>>>             I have my Django version available - but the internet
>>>             problems here are still unresolved. The technician is
>>>             supposed to make another visit today to see what is
>>>             wrong with our connection. Let me know if and when you
>>>             think this will be useful to you.
>>>
>>>         Let me know when your connection is stable and I would start.
>>>          What  are the things you need me to do in the meantime ?
>>>
>>>             One open issue is sugar3 vs sugar. Currently two
>>>             versions of Sugar are released. The sugar version
>>>             supports gtk while sugar3 supports gtk3. Unfortunately,
>>>             gtk3 was developed to be totally incompatible with gtk.
>>>             For example, incorporation of one gtk3 feature requires
>>>             that all direct and indirect references to gtk be
>>>             removed or the activity will throw an exception. Several
>>>             of the gtk3 conversions failed to meet this test and so
>>>             fail. The issue is whether curated activities be limited
>>>             to ones converted to gtk3. The positive is that Sugar
>>>             could revert to releasing and maintaining only a single
>>>             version. The downside is that 100 or more activities
>>>             will no longer be available. Specifically, in our
>>>             implementation of ASLO, we need to show which versions
>>>             of an activity work on which versions of Sugar (e.g.
>>>             i86, arm, amd64, sugar or sugar3, and so on). We also
>>>             need to show which ones support localization. There are
>>>             many English activities and many Spanish activities that
>>>             make no provision for localization. Luckily there are
>>>             many that have no language component. However, for many
>>>             of these, some kind of help is needed to convey the way
>>>             the activity works. 
>>>
>>>
>>>         Some people believe GTK3 is slightly better
>>>         <https://www.reddit.com/r/linux/comments/3e3q8n/is_there_a_technical_reason_why_gtk3_is_better/> and
>>>         I think GTK3 will stay but that should be asked in community
>>>         and voted upon and taking in considerations cost of
>>>         development and porting, only a discussion will help in this
>>>         one.
>>>
>>>             You are wading into a deep and vast body of water!
>>>
>>>         As long as I have something to hold onto, I will not drown 😅
>>>
>>>         Thanks,
>>>         Jatin Dhankhar
>>>
>>>         On Mon, Apr 10, 2017 at 9:11 AM, Tony Anderson
>>>         <tony_anderson at usa.net <mailto:tony_anderson at usa.net>> wrote:
>>>
>>>             Hi, Jatin
>>>
>>>             Normally, we use http://chat.sugarlabs.org or on
>>>             freenode: sugar-meeting or sugar-newbies. These are
>>>             logged sites so that there is a record. The second is
>>>             more appropriate since sugar-meeting is used for SLOB
>>>             meetings and the like. The real problem with IRC is time
>>>             zones. Email has the advantage that either party can
>>>             send or receive at any time. Last year with a GSOC
>>>             mentee we used sugar-newbies by arranging a specific
>>>             meeting time in advance.
>>>
>>>             I haven't heard from Walter, but my preference would be
>>>             to use the Sugarlabs server since the content is largely
>>>             already there and it would be easier to make it the
>>>             official site if that were decided. So in the short run,
>>>             I think you should do whatever is best for your own
>>>             development process.
>>>
>>>             I am not sure how CI fits into this. If the activity
>>>             development is done on GitHub, then the deployment model
>>>             is to run setup.py to create an xo bundle and then copy
>>>             that bundle to the appropriate location in the
>>>             download.sugarlabs.org <http://download.sugarlabs.org>
>>>             tree. Assuming the update results from a PR, the
>>>             deployer would need to update the activity information
>>>             on ASLO appropriately. However, that process depends on
>>>             where that data (metadata) is stored. Another part of
>>>             the process is how to update 'translate.sugarlabs.org
>>>             <http://translate.sugarlabs.org>' with the corresponding
>>>             POT file to enable localization. We can get help from
>>>             Chris Leonard on this.
>>>
>>>             I have my Django version available - but the internet
>>>             problems here are still unresolved. The technician is
>>>             supposed to make another visit today to see what is
>>>             wrong with our connection. Let me know if and when you
>>>             think this will be useful to you.
>>>
>>>             I have now tested most of the activities (~400). I was
>>>             optimistic in the number that work out of the box.
>>>             However, a part of this is running them in the Ubuntu
>>>             version of Sugar (amd64). There are many activities
>>>             which launch object code (mostly c) which is dependent
>>>             on the architecture. I am now trying to repeat the tests
>>>             on an XO-1.75. One issue on Ubuntu is that many
>>>             activities assume a 1200x900 screen and so on a 1024X768
>>>             screen overflow. This makes some of the games unusable
>>>             since part of the controls are off the screen. Because
>>>             of the internet problems, the untested activities tend
>>>             to be new ones since I was using my local repository
>>>             which is a snapshot taken several months ago. The other
>>>             group are the GCompris activities (about 70).
>>>
>>>             My intent is to build a 'curated' repository of
>>>             activities known to work and be usable on the XO and on
>>>             Ubuntu (or such other platform that Sugar may choose to
>>>             support). Most of the currently not work activities have
>>>             software dependencies no longer included in the current
>>>             Sugar release. So the curated library will grow as
>>>             activities are repaired over time.
>>>
>>>             One open issue is sugar3 vs sugar. Currently two
>>>             versions of Sugar are released. The sugar version
>>>             supports gtk while sugar3 supports gtk3. Unfortunately,
>>>             gtk3 was developed to be totally incompatible with gtk.
>>>             For example, incorporation of one gtk3 feature requires
>>>             that all direct and indirect references to gtk be
>>>             removed or the activity will throw an exception. Several
>>>             of the gtk3 conversions failed to meet this test and so
>>>             fail. The issue is whether curated activities be limited
>>>             to ones converted to gtk3. The positive is that Sugar
>>>             could revert to releasing and maintaining only a single
>>>             version. The downside is that 100 or more activities
>>>             will no longer be available. Specifically, in our
>>>             implementation of ASLO, we need to show which versions
>>>             of an activity work on which versions of Sugar (e.g.
>>>             i86, arm, amd64, sugar or sugar3, and so on). We also
>>>             need to show which ones support localization. There are
>>>             many English activities and many Spanish activities that
>>>             make no provision for localization. Luckily there are
>>>             many that have no language component. However, for many
>>>             of these, some kind of help is needed to convey the way
>>>             the activity works.
>>>
>>>             You are wading into a deep and vast body of water!
>>>
>>>             Tony
>>>
>>>
>>>             On 04/10/2017 12:00 AM, Jatin Dhankhar wrote:
>>>>             Hi,
>>>>             Sorry for the delay. I went through the polls tutorials
>>>>             and I think I am getting hang of Django. I have one
>>>>             query that is out of context, what is your IRC setup ?
>>>>             IRC doesn't allow message to be delivered or stored
>>>>             once either party is offline, people login through a
>>>>             external server for IRC's to maintain their
>>>>             availability in a channel. May I suggest something like
>>>>             Slack or Flock for communication. IRC is good for quick
>>>>             and fast connection but Slack and alternatives allow
>>>>             easy communication. (Just a suggestion, though)
>>>>
>>>>             Should I  deploy the same polls app on DigitalOcean
>>>>             along with CI pipeline and branching model in the
>>>>             meantime with code hosted on Github ?
>>>>
>>>>             Thanks,
>>>>             Jatin Dhankhar
>>>>
>>>>             On Sat, Apr 8, 2017 at 6:28 AM, Tony Anderson
>>>>             <tony_anderson at usa.net <mailto:tony_anderson at usa.net>>
>>>>             wrote:
>>>>
>>>>                 Hi, Jatin
>>>>
>>>>                 I abbreviated this thread because I am getting
>>>>                 complaints from the list that the emails are too long.
>>>>
>>>>                 Yesterday, I was able to complete a set of 25
>>>>                 activities to use as a test base. The Django app
>>>>                 now has two views. First is a display of all of the
>>>>                 activities with the icon and name. A link for each
>>>>                 opens the full activity page. The data for the
>>>>                 views is in a file which is a list of jsons, one
>>>>                 json per line.
>>>>                 The  views.py reads this file and builds the view
>>>>                 from the jsons.
>>>>
>>>>                 So the app consists of:
>>>>                     urls.py
>>>>                     views.py
>>>>                     templates/
>>>>                 base.html
>>>>                 list.html
>>>>                 activity.html
>>>>
>>>>                 with the data:
>>>>                 fixtures/activities.json
>>>>                         icons/
>>>>                         xo/
>>>>
>>>>                 The icons directory has the activity icons (*.svg)
>>>>                 The xo directory has the activity bundles (*.xo)
>>>>
>>>>                 Meanwhile, I am having more troubles with the ISP
>>>>                 (a new one). When that gets resolved, I should be
>>>>                 able to send the above to you.
>>>>
>>>>                 I have also mentioned our dialog to Walter Bender
>>>>                 suggesting that you could set up your working model
>>>>                 as activities3.sugarlabs.org
>>>>                 <http://activities3.sugarlabs.org>. This would give
>>>>                 you a permanent place to do the development and
>>>>                 make your project visible to the community so that
>>>>                 we can obtain feedback. I aslo suggested that the
>>>>                 project repository could be placed on GitHub so
>>>>                 that again the community can comment and
>>>>                 potentially contribute.
>>>>
>>>>                 Tony
>>>>                 On 04/07/2017 12:23 AM, Jatin Dhankhar wrote:
>>>>>                 Hi,
>>>>>
>>>>>                 I am setting up Django now and going through some
>>>>>                 tutorials. I will setup the code with a very basic
>>>>>                 and barebones version of what we want to achieve
>>>>>                 and put it in a private repo. We can even host it
>>>>>                 online, since I have some DigitalOcean credit
>>>>>                 left. Will keep you posted.
>>>>>
>>>>>                 Thanks.
>>>>>                 Jatin Dhankhar
>>>>>
>>>>>                 On Thu, Apr 6, 2017 at 7:11 AM, Tony Anderson
>>>>>                 <tony_anderson at usa.net
>>>>>                 <mailto:tony_anderson at usa.net>> wrote:
>>>>>
>>>>>                     Hi, Jatin
>>>>>
>>>>>                     In setting up Django, I think you will do it
>>>>>                     on your computer not on a separate server.
>>>>>                     Django handles that through its own server and
>>>>>                     has sgqlite as a database built-in. A good
>>>>>                     first start would be to set up the tutorial
>>>>>                     app (https://www.djangoproject.com/).
>>>>>                     Essentially you will need to install django
>>>>>                     and then set up the tutorial app 'poll'.
>>>>>                     Working through this tutorial will help a lot
>>>>>                     later on. The key point is that after initial
>>>>>                     setup, there are three important elements:
>>>>>                     url.py which defines the urls to access the
>>>>>                     application, views.py which is the python code
>>>>>                     that responds to a request by accessing the db
>>>>>                     and delivering the relevant information to a
>>>>>                     template. The template is an html file with
>>>>>                     variables of the form {{ activity.name
>>>>>                     <http://activity.name> }} transferring
>>>>>                     information from the view to the template.
>>>>>
>>>>>                     What I have is an application: aslo. Once you
>>>>>                     have Django installed and have run the
>>>>>                     tutorial app through Django's server, you'll
>>>>>                     be ready. Essentially, you will only need to
>>>>>                     add aslo as a second app in the settings. One
>>>>>                     possible confusion is that DJango lives in a
>>>>>                     project - essentially a directory containing
>>>>>                     its manage.py admin interface. In the
>>>>>                     directory is another directory of the same
>>>>>                     name with the settings.py and url.py. The poll
>>>>>                     app is a directory in the top-level alongside
>>>>>                     the inner directory with the project name.
>>>>>
>>>>>                     For example, my project is schoolsite. So my
>>>>>                     setup looks like:
>>>>>
>>>>>                     /library/schoolsite/schoolsite
>>>>>                     /library/schoolsite/aslo
>>>>>
>>>>>                     Meanwhile my code links directly to an
>>>>>                     activity page with no index. I'll add an index
>>>>>                     so the essential structure will be there. I'll
>>>>>                     also include the 'fixtures' to set up a 25
>>>>>                     activity capability. Fixtures are csv files
>>>>>                     from which the database (metadata) can be loaded.
>>>>>
>>>>>                     Tony
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/aslo/attachments/20170413/0849d2d6/attachment-0001.html>


More information about the ASLO mailing list