[Sugar-devel] Updates: Python3 app store (aslov4) : Sugar App Store

[shaansubbaiah.cs18 at bmsce.ac.in]

 Hey Srevin,
This looks really good! Page load times, search was fast. Haven't tried downloading any Activities though.
It may just be me but the page has a little too much contrast with the colors.Pagination might also be a good addition. Cool lava lamp toggle :P

Will be waiting for a full release 😃
Best,Shaan


    On Tuesday, June 16, 2020, 9:24:53 PM GMT+5:30, Srevin Saju <srevinsaju at sugarlabs.org> wrote:  
 
   
Hello, 
 
 
Hope all are having a safe week amid nCoV.
 
I and Manish (sugar at radii.dev) had been working on the GSoC project which did not get a slot (Python3 app store aslov4), and I guess we have made good progress in it. We decided to collaborate on the project because a frontend and backend developer combo helps to get the best features of both ends :D.
 
To avoid misconceptions and misinterpretations, please read the entire email before replying. Thanks!   :P
 
 
Need Statement:
 
 
In short, the most important points
    
   - Create the simplest possible app store for Sugar activities, where each activity included has been (a) ported to Python 3 and released, and (b) tested on Sugar Live Build.
   - We used to have an app store for Sugar activities, but because we can't seem to attract any PHP developers the app store has failed to keep up with development.
   - We now use activities.sugarlabs.org for Python 2 activities only.
   - We have tried to make a replacement for  activities.sugarlabs.org three times, and each time the features we need were not finished. These projects have been too ambitious and have not been supported collectively by the Sugar Labs community.
 
~ from https://github.com/sugarlabs/GSoC/blob/master/Ideas-2020.md#sugar-app-store-for-python-3-activities-aslov4
 
This made us rethink the idea of a redesigned app store for sugarlabs. This, would be a collection of all the activities in sugarlabs organization which can be successfully built without errors.
 
The new app store is special in a lot of ways, let me bring out the most important points:
 
    
   - The appstore is built in pure static HTML, Vanilla Javascript and JQuery. Anyone with basic Javascript knowledge can easily add features. Theming and customization are provided by Bootstrap4. The theme, scripts are completely independent of each other. Changes to the will not affect the generation of the appstore.    
 
   - Search function is provided by a powerful Javascript library called minisearch which is an open source search indexer. All processing is therefore done on client side. The benefits of using an external library are many, as you know,    
 
   - The App Store is based on JSON (Javascript Object Notation). an "index.json" is produced by the generator helper script so that search indexing, activity filtering can be easily done on the client side.   
 
   - Python scripts are used to generate static HTML files of the app bundles and index.json. The activity's unique HTML document is produced without style / css, and are externally loaded, so that it can be customized / removed whenever necessary (for example on very low bandwidth connections, or for adding another theme)
   - Sugar Appstore depends on static HTMLs also because, it can be useful for SEOs, and to provide an external link to the activity from a third party website. Dynamically filtered search results might not be static. Providing a correct link to the bundle helps another user to share a specific activity bundle with another
   - All the activities on https://github.com/sugarlabs are indexed automatically <see (9)> and built using Continuous Integration (GitHub Actions) over here. Out of 300+ bundles, 199 and still counting have successfully built a *.xo bundle and are continuously uploaded to CDN.    
 
   - The appstore could be downloaded for Offline use, or can be hosted on a local server if a user would not like to use the online one. This is specifically useful, if a school would host, the sugar appstore on its local server, so students could download activities without accessing the internet. The entire app store (batteries and bundles included) weighs approximately 613 MB. See this for a detailed idea.
   - In case a user would like to add another bundle to the sugar appstore, previously he/she had to have a SugarLabs shell account (AFAIK, and as far as I understand). This might limit the accessibility to some users. This is hence solved by a new system of adding bundles to the activity. Pull Request to Add Applications  is system which I adopted from the AppImage Hub (https://github.com/appimage/appimage.github.io) where a similar app store is built based on the same concept. Every PR is tested by continuous integration, and then all successfully merged PRs are added to the website, and the static website rebuild is triggered on each commit. A similar idea is implemented in Sugarlabs Appstore. This ensures that a .xo can be built on another host also using the same commands. Improving the quality of the bundles. For information regarding how to add a bundle, see this. For a sample PR which intends to add a bundle, see PR#6 which intends to add Jupyter Activity. All urls which can be cloned using git are accepted. You may look into how the build script works here https://github.com/srevinsaju/sugar-activity-build/blob/master/.github/workflows/bundle_test.yml. For those users who are not interested in GitHub, its recommended if Sugarlabs would host a GitLab which also supports CI; As this issue has been discussed previously, I am not putting too much insight into the same; :D
   - Builds are refreshed every 5th minute of a day; (cron: "5 * * * *"). There is no specific reason why 5th minute is chosen, mostly because I can finish pushing changes between 0th minute and 5th minute (normally because my school classes finish at the 0th minute; LOL; )
   - Builds are hosted to netlify CDN. Netlify CDN is chosen because its free for open source, and it also has a capability to get form submissions without a server. This is a cheap implementation without a server. There is no other reason why netlify chosen. Netlify has a very permissive quota. Other static site hosting as such as github.io will make the repo huge; or will require payment. Moreover. build on netlify takes only 1minute and 5 seconds to complete, with older fast network speed.
   - Responsive HTML design. Makes it adaptable on all screen sizes, on small android phones to 1080p screens and much more. This is better for SEO, as well as accessibility.   
 
   - Lighthouse speed testing shows better cumulative speed score than activities.sugarlabs.org. On Desktop, the score for ASLOv4 is 94/100; on mobile its 76/100 for ASLOv4. For activities.sugarlabs.org, its 89/100 on Desktop, 76/100 on mobile. Results may vary slightly from time to time; Repeat tests to get concurrent values. Results provided by Lighthouse powered by Google. Result details for sugarstore.netlify.app : https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fsugarstore.netlify.app%2F&tab=desktop; Result details for activities.sugarlabs.org: https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Factivities.sugarlabs.org&tab=desktop
   - Its possible to embed (keeping a local copy of) JQuery and Bootstrap for offline use, but I personally prefer to keep the cdn (i.e <script>), this is because; modern browsers automatically cache commonly used script files. So this would increase the performance with caching. This will not require the user to download the libs again. Moreover, if the user has visited any other site which uses the same library, it would be actually better.
   - It looks sophisticated to keep a download count on static websites, but that does not mean its impossible. Its nevertheless, not possible (afaik) to get download files with only static files (i.e without a server to depend on). I could
   
   - send a POST request to another server, which would increment the download count on every click of the download button; along with a GET request from the server to get the download count. This would imply, for every activity; a GET request will be emitted, or preferably, a single GET request to get all the download counts and update each of the activity shown for display with the new    
 
   - Use Firebase Library to create a firebase server where I could use Cloud Firestore and send and receive download stats. However, this has a few cons: i.e increase the page load time from 1.1s to 2.3 seconds; as well as make it complex to protect the server from user manipulation of the download count. I have tried this, I will enable it, if necessary
   - Add a download analytics with a one way download count (send only - no receive). By using Google Analytics, I can send custom event trigger, and administrators can analyze and filter data / sort data by download counts. I, however, did not find a direct method to get data from Google Analytics.
     
   - Future Additions:
   
   - Add Featured Activities: Activities which are most worked on: i.e MusicBlocks, TurtleArt, Physics Activities, Fructose Set of Activities could get a wide card along with a feature image. This will act as a showcase to the developer and the activity; mostly based on GitHub Stargazers. I would like to follow https://snapcraft.io as a reference from which Ubuntu Software is built.   
 
   - Add a link to Flatpak Package: this aslov4 could be the final activity store for sugarlabs. Recently many contributions have diverged. Its important to unify them together. Some users might prefer flatpak over a .xo file. Activities that are packaged as Flatpaks are good because some users, follow a modern desktop system, or might not be interested to install Sugar OS to run sugar desktop. For those corner cases, providing Flatpak as an alternative would not let users turn away from using Sugar Activities. Primarily Sugar Activities are meant for improving a child's skills or understanding; which ever mode, an activity is distributed; i.e., the most accessible package.
   - Add information to other packages of sugar activity bundle. This includes Ubuntu Packages. Fedora RPMs, and the Arch Linux Community Packages; I will need a way to identify each of them; But at the present, I would like to leave it as is. If any developers at sugar is interested to change some structure of Sugar Repository, this would make things faster.    
 
   - Add a screenshot, description tag in activity.info: Screenshots of the working activity on the appstore will probably give a clearer idea on how the activity works. The https://help.sugarlabs.org is not frequently updated with changes on activity's repository. If we could add another tag called screenshots (following the convention given by Martin on https://github.com/tchx84/sugarapp. See https://github.com/tchx84/sugarapp/blob/master/flatpak-guide.md#porting-an-application-with-sugarapp (point 7) on how an ideal "activity.info" could be created. This would help me out in creating a better appstore (faster). The new "activity.info" could include 'screenshot' and 'description' tag which is not currently used. Martin has also mentioned of the source of the new data in the link given below.   
 
   - Create a Python3 minimal activity which does not need to load the entire website for downloading an activity. The activity will just pull index.json, and parse it. As the extra HTML, CSS, and JS are unnecessary for a GTK3 based sugar-activity, it would help to save a lot of data, especially on countries where internet connection is extremely slow (50kbps ?)   
 
  
I would like to convey my thanks to Manish for providing me frequent suggestions and always pointing out important flaws ( High CPU usage bug in Chromium/Chrome Browser: 65a7b8f3ea2593976ef21df7a2592b030501c6d8 , Accessibility ), and also the base repository, i.e., https://github.com/free-libre-software/sugarappstore as a template and idea provider for the sugarlabs app store to take this shape. 
 
 
________________________________________________
 
 
Logo:
 
The logo is designed under Creative Commons License Attribution Share Alike 4.0. You may get the source code of the logo here: 
 
 
source: https://github.com/sugarlabs-appstore/appstore-assets
 
The logo has been designed with modern conventions of app design - symmetry and vibrance. Maybe its similar to the colorful icon design proposal for GSoC : https://github.com/sugarlabs/GSoC/blob/master/Ideas-2020.md#colored-desktop-and-activity-icons
 
 
________________________________________________
 
Generator (tool used to generate static webpages):
 
The generator is script which automates build process of bundles and writes static HTML files. Primarily written on Python Programming Language, can be customized with alternative script files which are supported on Linux. Generator, aka (internally called), SaaSBuild (Build tool for Sugarlabs Activity App Store), is designed to be multi platform supported, although tested on Linux only. Its possible to run it on other OSes too.
 
source: https://github.com/sugarlabs-appstore/sugarappstore
 
NOTE [IMPORTANT]: Current development goes on in PR#4 (https://github.com/sugarlabs-appstore/sugarappstore/pull/4). You might need to checkout oop-ss if you want to test out the latest features. The master branch contains the work by Manish before the official start of GSoC. The PR is waiting for a review.
 
 
________________________________________________
    
 
Static HTML, CSS, JS
 
These are files which can be run without the need of the generated index.json. Using JQuery, the data is dynamically read, and updated on the index.html.
 
Q) Why was Static parts of the website not merged with the generator (SaaSBuild)?
 
Its because of yet another few reasons
    
   - Another developer could manually add his own themes / html styleset with, maybe another JS Framework, for example React, Angular or Vue.js later. He/She would not have to rewrite the Generator class again for changing the frontend part.
   - Developers who are willing to contribute to HTML / Web design / Js can only work on the "static" repository. This will help to (probably) increase contributors. A developer who is only good at JS would not be hence be discouraged to contribute to the current repository, if Python is also involved.
 
source: https://github.com/sugarlabs-appstore/sugarappstore-static
 
________________________________________________
 
Continuous Integration
 
Continuous Integration Downloads and sets up Sugar desktop, i.e sugar-toolkit-gtk3, sugar, sugar-datastore, sugar-artwork and creates bundles and packages it for publish to CDN
 
 
source: https://github.com/srevinsaju/sugar-activity-build
 
________________________________________________
 
The DEPLOYED WEBSITE:
 
The best part, check it out here: https://sugarstore.netlify.app
 
For a list of indexed apps in a portable format: 
 
    
   - https://sugarstore.netlify.app/index.json
   - https://github.com/srevinsaju/sugar-activity-build/releases/download/latest/index.json
   - https://github.com/srevinsaju/sugar-activity-build/releases/download/latest/sitemap.xml
 
For bundles app store (portable - except cdn), see
 
https://github.com/srevinsaju/sugar-activity-build/releases
 
________________________________________________
 
Q) How to run the appstore?
 
Use any static hosting server; for testing I use:
    
   - python3 -m http.server 3000
     
   - gunicorn
     
   - php
     
   - npm install -g http-server; http-server
     
   - apache
     
   - ngrok
     
   - serveo   
 
 
Regarding whom to approach:
 
We are working on two different things. Manish knows a way lot more than me regarding HTTP Headers, Sphinx, CORS, User-Agent, and other web server related stuff. I am working on design (css, html, js) and the generator script (python), and those related to sugar GTK shell, sugar activities. We both collaborate on working with the generator script.
 
 
=====================================
 Hope everyone like this. All questions are welcome. Please note that the App Store has not yet released v0.1, so its basically in alpha.  
PS: If anyone is interested to contribute, PRs are welcome. Please reply me on this thread if you require access to https://github.com/sugarlabs-appstore/ organization. It was created so that Manish and I could collaborate. And also so that, development won't freeze in the absence of one. 
 
 
PS: If you have some other sweet suggestions for Sugarlabs App Store (name), please let me know. Just as jarabe, carquinyol, and sucrose are used within sugar, it might be interesting for kids to see their favorite sweet on the screen. We have stuck with sugarlabs-appstore, as we did not get any other name!
 
Thanks if you read all the way till here!!
 
 
Stay safe! Take care!!
 
      
   V/r

Srevin Saju
https://github.com/srevinsaju

 
 _______________________________________________
Sugar-devel mailing list
Sugar-devel at lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20200616/94af6c22/attachment-0001.htm>