[Sugar-devel] [ANNOUNCE] packagekit-backend-presolve as an attempt to provide installing activity dependencies on demand in order to support GNU/Linux distribution agnostic launch from app stores

[Aleksey Lim]

Hi all!


This is an announce of new project that is intended to solve, as a part
of more complex work, issue that prevents reliable launching of Sugar
Activities from app stores like Activity Library or Sugar Network.

The design document (mentioned below) url is:

    http://wiki.sugarlabs.org/go/Platform_Team/packagekit-backend-presolve

This is only design document, but the first implemlementation will be
pretty trivial and will be ready for the next Sugar Network development
release (later this month). Any ideas/suggestions are welcome.


== Summary ==

This is a [http://www.packagekit.org/ PackageKit] backend which is
intended to be used on restricted systems like XO-1 laptops where
regular package management routines are too heavy for limited amount of
memory or CPU resources. 

== The reason ==

The known, for the author of these lines, current practice is:

* [http://wiki.laptop.org/go/Olpc-update
  olpc-update]
  [[wikipedia:Rsync|Rsync]] the full system.

* [[Platform_Team/Server_Kit/sugar-client|sugar-client]]'s
  [[Platform_Team/Server_Kit/sugar-client#Unattended_system_update|unattended
  update]] in [[Dextrose]]
  Regular update but only for particular,
  much smaller than main ones, repository(ies).

Both ways are useful for usecases they were created for. But there is a
situation when they both become less useful. Having [[Activity Library]]
(or its distributed version on regional or school server), XO users
might download activities directly from the Internet (or distributed
server). So, it is becoming hard to use:

* olpc-update,
  [[Activity Library]] might have non-standard
  dependencies and it is either hard to keep olpc-update server
  up-to-date, or, useless to install all possible dependencies on every
  XO.

* sugar-client,
  activity dependencies are, in most cases, from
  official repositories when sugar-client's 3rd party repository are
  useless.

In other words, there is a need to install particular packages from
official Fedora repository and using existing methods is either
impractical (olpc-update) or impossible (sugar-client). Fallback to
using regular (like yum) ways is also not useful because they take too
much memory (at the first) and CPU resources, it is a special problem on
XO-1 laptops.

The packagekit-backend-presolve is intended to solve this issue by
providing standard way (it is PackageKit backend, i.e., sugar launching
code will be common for all platforms, it will call PackageKit all time)
to install activity dependencies and consuming as less as possible
resources on XO laptops. The backend was designed to support reliable
launch from resources like [[Activity Library]] and/or [[Sugar Network]]
when there are activities that might have non-standard dependencies.
This backend might be treated as a temporal solution, i.e., when either
XO will be powerful enough or regular installation procedures will take
less resources, it will be possible to switch to default PackageKit
backend (without making changes in activity launching code).

== Concept ==

The general idea is to pre-resolve dependencies for a limited set of
top-level packages and store this information on a server. Clients need
to download only small amount of information (compared with entire list
of available packages) to reuse it as-is (and avoid reading database
into memory during solving process).

Another important point is that pre-resolved dependency graphs don't
include packages from the base system. In other words, a particular
dependency tree doesn't end with the {{Code|glibc}} package. Instead, it
ends with the {{Code|sugar}} package. This expedient should drastically
reduce the size of trees.

Finally, missing packages will be installed directly by using the
{{Code|rpm -i}} command.

=== Upsides ===

* Users download metadata on a package by package basis, thus avoiding
  the need to download the full packages database;

* After downloading, metadata will be reused as-is, i.e., after checking
  what packages were already installed, system will download only missed
  RPMs and install them directly by calling {{Code|rpm -i}} command;

* Activities on resources like [[Activity Library]] or [[Sugar Network]]
  will be distro-agnostic. That is, they will work unmodified on all
  platforms that provide PackageKit (which means all mainstream
  GNU/Linux distributions).

=== Downsides ===

* The project is intended to install only limited number of top-level
  packages, i.e., packages that have pre-resolved dependency trees on a
  server; for example, it is not capable for full system update;

* Need to support server-side generation of dependency trees.

== Implementation ==

=== Premisses ===

The implementation is following the premises:

* Being targeted only to XO laptops (assuming that other platforms are
  more powerful where standard PackageKit backend can be used);

* Keep less work on XO side and more work on server side to make
  installation process on client side as fast and reliable as possible;

* Implement only first-needed functionality and provide more complex
  features as required, e.g., current implementation doesn't support
  updating already installed packages leave it to hard (and rare, e.g.,
  yearly based) full system updates (re-flashes or calling olpc-update)
  and incremental regular updates (sugar-client).

=== Overview ===

* Clients keep a list of packages installed by the back-end;

* After getting install request, the back-end will check if the
  requested package is already installed;

* If not, it downloads a file from know HTTP location with the requested
  package's name;

* Downloaded file contains RPM url for requested package and a list of
  package-name/url pairs for all dependencies (starting from some known
  point);

* All missing packages will be downloaded and installed by directly
  calling {{Code|rpm -i}};

* Installed packages list will be updated correspondingly.

== Usage examples ==

[[Sugar Network]], in attempt to provide GNU/Linux distribution agnostic
launches of activities it provides, supports packagekit-backend-presolve
users.

API server provides the following kinds of urls:

* {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages</nowiki>}}
  list all supported platforms;

* {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages/</nowiki><PLATFORM>}}
  list all supported packages for particular platform;

* {{Code|<nowiki>http://api-devel.network.sugarlabs.org/packages/</nowiki><PLATFORM>/<PACKAGE>}}
  pre-solved dependency graph for particular package.

Where:

* {{Code|PLATFORM}}
  one of supported packagekit-backend-presolve
  platforms, e.g., OLPC-13.0.1;

* {{Code|PACKAGE}}
  a name of the particular package.

In order to support distribution agnostic launches, [[Sugar Network]]
supports metadata ''database'' for all packages that are being used as
dependencies for activities it provides (see ''Packages'' project using
one of existing [[Sugar_Network#Try_it|Sugar Network clients]]). Every
entry in the ''database'' contains a map of native package names for
particular GNU/Linux distributions. So, activities mention only
''database'' entry name as a dependency. [[Sugar Network]] will
automatically generate dependency graphs for packagekit-backend-presolve
using ''metadata'' database. Afterwards, on launch side, ''database''
entry name will be resolved to a package name according to local
distribution and will be passed to the PackageKit (on XO laptops,
packagekit-backend-presolve will be used; on regular platforms, default
PackageKit back-end).

== Sources ==

Clone source repository from [http://git.sugarlabs.org/desktop
Gitorious] project:

 git clone
 git://git.sugarlabs.org/desktop/packagekit-backend-presolve.git

-- 
Aleksey