[Sugar-devel] Sugar-Server enhancement
Manash Raja
mpdmanash at gmail.com
Fri Apr 15 17:18:59 EDT 2016
Hi all,
I fixed the bug. I haven't created a PR just yet.
https://github.com/ManashRaja/sugar/commit/660985d2183416cd3ed758095e92adf82f87a10c
As of now I am parsing HTML data for json data to look for registered
laptops at an XS from the webaddress http://schoolserver:5000 that serves a
liist of registered laptops.
I will modify it if there is another better method that can be implemented
from the Sugar side (we don't intend to modify the server side I guess).
And here is the feature page as suggested by James.
https://wiki.sugarlabs.org/go/Features/Multi_XS-server_registration
Please have a look.
I am very excited to be a part of the discussions that go into the making
of great features that can affect people down there in deployment. :) There
is a scope to do so much.
Thanks
On Sat, Apr 16, 2016 at 2:32 AM, James Cameron <quozl at laptop.org> wrote:
> On Fri, Apr 15, 2016 at 04:02:39PM +0800, Tony Anderson wrote:
> > Hi, James
> >
> > This thread was getting long so I replied only to the most recent
> > communication. I am sure you have the full thread which shows the
> > scope of the discussion.
>
> You're making it longer, yes, by hijacking it. You can find the full
> thread here:
>
> http://lists.sugarlabs.org/archive/sugar-devel/2016-April/thread.html
>
> > According to trac bug #362 was opened seven years ago against 0.82
> > and last looked at three years ago. Several competent people looked
> > at it and left comments. I see none that signify consensus.
>
> I take it that you don't want Manash to fix this bug, thanks.
>
> I'd like it fixed. I think there are others who want it fixed. It
> would probably help with XS testing as well.
>
> > To have a design discussion, it is valuable to have a proposed design.
> >
> > I have tried to explain my proposal in detail. If there are
> > questions, I would be happy to try to respond. Fixing the 'Journal
> > is Full' dialog is a major help. However, what do you recommend to
> > deployments when this happens?
>
> 1. upgrade to Sugar 0.108 (by RPM or images), or backport the patch
> [#9623, #1720] into your custom builds,
>
> 2. transcode content to play in browser not Journal,
>
> 3. delete any activities that are not needed,
>
> 4. deploy Sugar Network to use the network activity cache,
>
> Also delete the Browse temporary directories, you reported this on
> 19th January, it remains a problem, and you refused to test my fix, so
> I lost interest very rapidly. [#4931]
>
> > The bottom line is that a reasonably active user is likely to need
> > more room to store her Journal than is available on the XO.
>
> No, because it's no longer a significant problem. XO-1 are in the
> minority and getting rarer as they die. Those that haven't died have
> SD cards.
>
> > In the Journal code a filled star sets the 'keep' flag in the
> > metadata. The cleared star clears the 'keep' flag in the metadata.
> > Using this feature greatly simplifies the coding and the Journal
> > view. As far as I know, the only use of this at the moment is to
> > support the Portfolio activity.
>
> You are using an implementation detail in describing the flag. The
> name given to the flag in documentation and user interface is
> "favorite" (sic).
>
> > I think the detail view is inappropriate exactly as it would be to
> > move the multiple selection checkbox there. These controls need to
> > be immediately available.
>
> I disagree; they won't get used, and so it would be a waste of
> valuable vertical space. The reason the checkbox won't get used is;
>
> - most laptops don't have a server,
>
> - an LRU algorithm can maintain the cache effectively.
>
> > The 'backup/sync' script is a good place to do check storage quotas
> > because the script needs to touch the datastore on a regular basis.
> > It has access to the amount of store in use and the LRU information.
> > For example, if the user wants a document downloaded, the script
> > knows its size and whether some other local copies need to be
> > deleted to make room.
>
> I disagree. This script runs infrequently. The LRU must be
> implemented inside the datastore for it to function properly.
> Otherwise the lag between user action and response by the script would
> be too long.
>
> > While an implementation detail, so far no change has been necessary
> > to the datastore class.
>
> That's no reason not to change it.
>
> > Actually, since the 'keep' or favorite star sets the metadata,
> > so far there has been no need to change the Journal.
>
> That's no reason not to change it.
>
> > "The multiple user feature is supported by Fedora and Sugar, but we
> > removed it for OLPC OS."
> >
> > I think I am beginning to understand. OLPC OS is your generic name
> > for the images to be installed on each model of the XO.
>
> Really, you are out of touch! OLPC OS is our name for the operating
> system releases on the XO laptop. We've been using the term at OLPC
> for a very long time, and use it in each release announcement.
>
> > I am deploying build 13.2.5 with Sugar 0.106 on all models.
>
> That's so sad. It was released in July 2015. It has the journal full
> bug you mentioned. I'm not interested in supporting that release,
> because I've already released two others. Upgrade.
>
> > So you are saying that we, users of Sugar or ' OLPC OS' could have a
> > multiple user version of Sugar if 'you', as developers, didn't
> > remove it.
>
> Well done.
>
> > As I understand it, you propose to generate unique serial-numbers
> > per user.
>
> No. I was describing what happens _now_ during registration, by
> reference to the code:
>
> 1. for XO laptops the serial number of the laptop is used,
>
> 2. for non-XO laptops a serial number is generated randomly,
>
> 3. there is no attempt to ensure the random serial number is
> unique, but the width of the random string is sufficient to make it
> unlikely,
>
> > So SSO would be guaranteed since no two users could have
> > the same serial-number. This would certainly work and probably
> > involve very little change to the existing code. What will be needed
> > is a 'dns' to map serial-numbers to usernames.
>
> No, I wasn't proposing that. It's your idea. I don't think it's
> guaranteed to be unique though.
>
> > Every school I have worked with keeps a careful record of students
> > (often in paper ledgers). Currently I provide a name record in a
> > Django database on the server (along with an XO inventory by serial
> > number).
>
> Fail to see relevance. Not all schools will or can do this.
>
> > Agreed that determining which Journal objects need to be saved to
> > the school server is not a difficult problem. However, datastore is
> > a class so each user's datastore and the common datastore would be
> > instances. So this seemed like a simple thing to implement.
> >
> > Actually, The deletion of Journal objects without an associated
> > document works amazingly well. The number of objects in the Journal
> > view goes from hundreds to only a few (often less than 20).
> > Moreover, these 20 are the obviously interesting ones. Nothing is
> > lost as the metadata is saved to the school server. It becomes much
> > easier to 'reflect' when you are only looking at the documents you
> > created. Meanwhile the myriad of objects can be subject to
> > statistical analysis.
> >
> > For many activities, such as the Terminal, the document saved is
> > actually 'state' information. This allows the Terminal activity to
> > be restored with tabs and pwd. There are many game activities such
> > as Memorize that also store state. It would seem more appropriate to
> > save this state information in the metadata. For example, a json
> > could be created in the metadata to hold state information. The
> > script could keep these objects to enable the user to resume.
>
> Your concept of metadata is not of interest to me; journal objects
> must continue reflect the learner's use of the laptop if the journal
> is to meet the designed style of reflection.
>
> Unreflecting adults are not the target user.
>
> >
> > Tony
> >
> > On 04/15/2016 01:36 PM, James Cameron wrote:
> > >On Fri, Apr 15, 2016 at 10:25:59AM +0800, Tony Anderson wrote:
> > >>Hi Manash
> > >>
> > >>The registration process is awkward but not the problem.
> > >This is unfair scope creep. Manash began by asking about bug #362 and
> > >has been working to fix that. Now you're asking him to consider a
> > >much larger task; not a coding task, but a redesign of Sugar Journal
> > >and Backup interaction. This is huge.
> > >
> > >And as far as I can tell, students aren't even accepted yet [1].
> > >
> > >1. https://en.wikipedia.org/wiki/Google_Summer_of_Code#2016
> > >
> > >What you propose is from a set of tasks [2] you added to the Wiki,
> > >which have not undergone any design review according to Sugar Labs
> > >design practice and feature policy. I do not see any consensus on
> > >these; we're yet to build a consensus.
> > >
> > >2.
> https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_on_the_Ground
> > >
> > >Or, it looks like you're trying to make your own fork of Sugar, which
> > >I'm fine with, it's open source after all, but to push that on others
> > >without their input is wrong.
> > >
> > >If you proceed without consensus as a sole designer, then OLPC will
> > >fork Sugar (as we already have so that XO-1s will go faster), and
> > >you'll be making your own builds.
> > >
> > >>The problem is that rsync is used to create backups of the Journal
> > >>and no effective means is offered to restore.
> > >Agreed. We have no restore from server feature in Sugar 0.108, along
> > >with no way to start a backup to server, and no selective restore.
> > >
> > >(We have backup to media, restore from media, but no selective
> > >restore from media. Also, restore from media replaces Journal!)
> > >
> > >>However, the ultimate problem is thinking of the problem as one of
> > >>backup. If you try to solve the wrong problem, often the result is a
> > >>wasted effort.
> > >>
> > >>The Journal is single place where Sugar users save their documents.
> > >>This is done by the Sugar activities when they close. The majority
> > >>of XOs are still XO-1s with a 1GB store.
> > >This point in your argument is void, because XO-1 are 45% of the XO
> > >laptops manufactured so far. I have the numbers.
> > >
> > >Also, many XO-1 have been upgraded with an SD card.
> > >
> > >>If the available store is less than 50GB,
> > >No, that's 50 MB, not 50 GB. See _SPACE_TRESHOLD (sic) in
> > >sugar:src/jarabe/journal/journalactivity.py [3].
> > >
> > >3.
> https://github.com/sugarlabs/sugar/blob/master/src/jarabe/journal/journalactivity.py#L56
> > >
> > >>Sugar effectively shuts down.
> > >This point in your argument is void, because this has been fixed [4,
> > >5, 6], please upgrade to Sugar 0.108 which is in OLPC OS 13.2.7 [6].
> > >
> > >4. http://dev.laptop.org/ticket/9623
> > >5. https://bugs.sugarlabs.org/ticket/1720
> > >6. http://wiki.laptop.org/go/Release_notes/13.2.7#Fixes
> > >
> > >>This typically results in the deployment reflashing the XO erasing
> > >>all of the documents created by that user - a tragedy.
> > >It was a known bug, so that's a training issue. You previously
> > >proposed to train a teacher to use "rm -rf" to delete a known_hosts
> > >file instead of Manash coding up an "ssh-keygen -R" command. It is
> > >inconsistent to be able to do one and not the other.
> > >
> > >>What I am proposing is to use the school server as the primary store
> > >>for the Journal with its effectively unlimited storage capacity. The
> > >>ds_backup script needs to read the datastore uploading any new or
> > >>modified documents. The local datastore can then be viewed as a
> > >>cache for current working documents.
> > >I'm favour of this ideal in principle, but it remains a huge design
> > >and consensus challenge, not a coding challenge.
> > >
> > >However, with the XO-1, XO-1.5 and XO-1.75 using IEEE 802.11g the
> > >local wireless network will collapse sooner due to this new load.
> > >
> > >>On the XO, the datastore is shown in the Journal. The 'keep' star
> > >There's no such thing. There's a favorite star [7]. It has a defined
> > >purpose. Are you proposing to destroy that purpose, or add another
> > >column to the journal? There's even less room now that the multiple
> > >selection checkbox was added.
> > >
> > >7. https://help.sugarlabs.org/en/journal.html#journal-features
> > >
> > >>could be used to show whether there is a local copy of that document
> > >>or not. If the document is not needed locally, the user can clear
> > >>the star. In this case, the backup script could delete the local
> > >>copy. If there is no local copy of the document, then the user could
> > >>set the star. In this case the backup script could download the
> > >>document.
> > >My preference would be for the flag to be in the Journal detail
> > >view [8], where there is available display space.
> > >
> > >8. https://help.sugarlabs.org/en/journal.html#journal-detail-view
> > >
> > >>This capability could be used to set a quota on the amount of space
> > >>used by the Journal. If the space is exceeded, the 'backup' script
> > >>could delete local copies of document by LRU until the quota is met.
> > >>Similarly, there should be a quota on Sugar activities which could
> > >>also automatically be pruned back LRU. Managing the store
> > >>automatically is consistent with keeping the Sugar UI as simple as
> > >>possible.
> > >This should be built into Sugar rather than in the non-Sugar backup
> > >script. They should be maintained together.
> > >
> > >This would be a code change to git repository sugar-datastore and the
> > >Journal activity in repository sugar.
> > >
> > >>As always, there are complications. The original OLPC concept was
> > >>that there would be one XO per user. As a result the software was
> > >>designed for a single user identified by the XO serial number.
> > >The multiple user feature is supported by Fedora and Sugar, but we
> > >removed it for OLPC OS.
> > >
> > >>Today, many XO deployments provide enough XOs for a classroom.
> > >>During the day, different students use the XO as their class goes to
> > >>the computer lab or as the computers are distributed from classroom
> > >>to classroom. However, all of the documents created are in a single
> > >>Journal with only the user's memory to indicate which document goes
> > >>with which user.
> > >OLPC did not design OLPC OS to be used in this scenario, so no
> > >surprise you've hit that. But it's not a Sugar problem. Don't
> > >conflate Sugar with OLPC OS.
> > >
> > >>The OLPC Ubuntu Sugar 14.04 Trusty LTS (to use its official name)
> > >>solves this problem at the laptop side by using standard gnu/linux
> > >>logins.
> > >The multiple user feature is supported by Ubuntu and Sugar, and I
> > >haven't removed it yet. I know how to; small configuration change to
> > >lightdm package.
> > >
> > >Don't forget SoaS. The Fedora 23 SoaS is easily installed to disk and
> > >has multiple user capability. The Fedora 24 SoaS is shaping up to be
> > >just as good or better, since it is based on Sugar 0.108.
> > >
> > >>Each user has her own username and password. The Sugar activities
> > >>have been moved to common space in the file system so only one copy
> > >>is needed to support multiple users. Users are not 'olpc' but
> > >>identified by their username. However, the datastore is part of the
> > >>user space (one datastore per user).
> > >Yes. ODPU.
> > >
> > >>This is problematic since the backup script uploads to
> > >>/library/user/serial-number on the school server.
> > >No, you're wrong. In the Ubuntu scenario, the register_laptop
> > >function will invent a serial number because it won't find Open
> > >Firmware [1]. So it wouldn't be a problem. It doesn't sound like
> > >you've tested this.
> > >
> > >1.
> https://github.com/sugarlabs/sugar/blob/master/src/jarabe/desktop/schoolserver.py#L110
> > >
> > >>So, one strategy would be to upload to /library/user/username. This
> > >>requires that usernames be unique across all laptops using a given
> > >>schoolserver. This could be enforced at registration on the school
> > >>server.
> > >Starting to sound very complicated. Single-sign-on (SSO) across a
> > >school. These are truly amazing teachers with lots of free
> > >administration time.
> > >
> > >(There are deployments using Sugar with SSO already, but as it's
> > >outside the scope of Sugar we don't hear about them at Sugar Labs, and
> > >we don't provide the facility in OLPC OS, but that doesn't stop them.)
> > >
> > >>However, the Sugar releases for the XO
> > >We call that OLPC OS, which includes Sugar and Gnome desktops.
> > >
> > >>still maintains Sugar activities in /home/olpc/Activities. So, one
> > >>requirement is to restructure Sugar as was done for OLPC Ubuntu
> > >>Sugar 14.04 Trusty LTS.
> > >That would not block implementing a server datastore, since the
> > >implementation would not care what $HOME is set to.
> > >
> > >(And besides, it's already done for SoaS, so the Fedora activity
> > >packages can be used immediately.)
> > >
> > >>Another approach might be to create directories for each user of a
> > >>single XO (e.g. /library/user/serial-number/user1).
> > >That would require authentication service by the server datastore.
> > >
> > >>Another complication is that the Browse activity downloads files
> > >>from the school server to the Journal (e.g. pdfs, mp3). These
> > >>documents do not need to be saved to the users Journal backup on the
> > >>school server since they can be restored from the school server
> > >>'library'. Also, such documents when downloaded should be stored in
> > >>a common space available to all users of that laptop. Fortunately,
> > >>the source of a document is provided in the metadata.
> > >What you describe here can also be solved by deduplication.
> > >
> > >The Journal Git backend proposed by Martin and Walter could help with
> > >deduplication of journal objects across multiple journals.
> > >
> > >https://wiki.sugarlabs.org/go/Summer_of_Code/2016#Sugar_Core
> > >
> > >>One approach would be to divide the datastore into two directories
> > >>on the laptop, one in common space and the other local to the user.
> > >>The Journal could show both sets of objects.
> > >Or the server datastore would recognise content hashes of server
> > >artefacts and know it need not send the content from the client to the
> > >server before LRU local deletion. It could hard link it.
> > >
> > >>Finally, each Journal object consists of a metadata file and an
> > >>optional document. The metadata files tend to clutter the Journal
> > >>display (mine has hundreds of Terminal activity and Log activity
> > >>entries). I would propose that the Journal show only objects which
> > >>have a document with a user-supplied name (a metadata flag). The
> > >>script should backup the metadata files for those objects without a
> > >>document to a 'log' on the school server for statistical analysis
> > >>but delete them from the local datastore. Journal objects saved
> > >>without a user-supplied name (but something like Write.activity)
> > >>should have their document deleted. As part of GSOC there is an
> > >>initiative to require users to supply a name for documents they wish
> > >>to save - so this problem may not be part of the 'backup' scheme.
> > >>Whether a document is saved or deleted, the metadata can be saved to
> > >>the log and displayed by the existing statistical tools.
> > >I'm against any classification of journal objects in this way. We
> > >cannot know how useful a Terminal and Log activity object is to the
> > >learner.
> > >
> > >However, I would like a way for expert users to terminate an activity
> > >without saving a journal object.
> > >
> > >>As an old crumudgeon, I still believe design precedes coding.
> > >>
> > >>Reading the existing code is always a good idea:
> > >>
> > >>Sugar
> > >>
> > >> *
> > >>/usr/lib/python2.7/site-packages/jarabe/desktop/schoolserver.py
> > >>#registers server - notice transition from gconf to gsettings
> > >> * /usr/bin/ds_backup.sh #primarily decides if backup can be run
> > >> #backup logic is needed
> > >>because an rsync can use a lot of bandwidth in a local network
> > >> * /usr/bin/ds_backup.py #actually does the backup using rsync
> > >>(note: -d option AFAIK deletes an object from the backup if it is
> > >>deleted in the source,
> > >> #this has the effect of
> > >>limiting the size of the datastore to the available space on the XO
> > >>not on the school server).
> > >>
> > >>Server (xsce6)
> > >>
> > >> * /usr/libexec/idmgr #contains a number of utilities
> > >>used in registration
> > >> * /library/users #contains a directory per
> > >>serial-number of registered user
> > >> #use ls -a to see files
> > >>created. The idmgr creates a public/private key pair which is used
> > >>by sftp to authenticate - avoiding password
> > >>
> > >>Note: if you look at the server code, you can see why registering
> > >>the laptop on each connection works (and can avoid any need for a
> > >>registration menu item).
> > >>
> > >>When you get to know your way around the existing process, I'll send
> > >>you a copy of the ds_backup.py code I use to implement the item by
> > >>item backup.
> > >You should start using GitHub like the rest of us.
> > >
> >
> > _______________________________________________
> > Sugar-devel mailing list
> > Sugar-devel at lists.sugarlabs.org
> > http://lists.sugarlabs.org/listinfo/sugar-devel
>
> --
> James Cameron
> http://quozl.netrek.org/
> _______________________________________________
> Sugar-devel mailing list
> Sugar-devel at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/sugar-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/sugar-devel/attachments/20160416/872511b0/attachment-0001.html>
More information about the Sugar-devel
mailing list