[Sugar-devel] getting the Sugar Toolkit working with python 3

D. Joe sugarlabs at etrumeus.com
Tue Jan 28 12:23:31 EST 2020
Previous message (by thread): [Sugar-devel] getting the Sugar Toolkit working with python 3
Next message (by thread): [Sugar-devel] getting the Sugar Toolkit working with python 3
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
OK, in the face of my unease with seeing someone doing that (modifying files in directories under package-management control) anywhere but in an isolated dev environment (container or chroot), I think I've finally grasped a second dimension to the Python path situation, particularly as it applies to Debian (and by extension, to Ubuntu).

There are two ways, generally speaking, to address this sort of problem: Either a) put the files where your installation of Python is looking for them (what's been done in the example upthread), or b) tell your installation of Python any new locations in which it should look for the files.

What we've been trying to do is put the files in a directory set aside by Debian policy as a place for Debian packages to put their modules. [10][10], [20][20]

I *had* thought Debian (with, presumably, Ubuntu following suit) uses paths containing `dist-packages` so as not to conflict with the default from the Python project upstream, which is to use `site-packages`. [30][30] 

We've been discussing how best to handle these differences for a little while now. [40][40], [50][50]

For instance, `sugar-live-build` currently transfers files from one `site-packages` directory to the parallel `dist-packages` directories for `sugar3`, `carquinyol`, and `jarabe` using `rsync` or `mv`. [60]

James pointed out Debian bug report 765022 in earlier discussion, but I only just now absorbed the lesson from the last response of the Debian developer there.[70]

There are two dimensions to consider: 

The first is whether or not Python module files are installed under control of the distribution's package manager using the distribution's .deb package, or built and installed by the system administrator from source. I had misattributed the `dist-packages` vs `site-packages` difference as addressing this dimension, but I was wrong. The file system distinction between `/usr/lib` versus `/usr/local/lib` is meant to address this dimension, and continues long-standing practice.

The second dimension is the attempt to make sure that the Python files match the python executable being used. Debian tries to accomodate the possibility that the sysadmin or user would try to use a python executable other than the one installed by the distribution. We use the distribution-supplied python interpreters (eg, `/usr/bin/python` and `/usr/bin/python3`), so this question hasn't arisen for us and may be why I missed seeing the distinction. 

It is to address this difference that the `dist-packages` versus `site-packages` distinction has been made by Debian (and inherited by Ubuntu). The first is meant for use by the python interpreter(s) that have been installed from the **distribution**'s packages, whereas the latter is meant to be used by those installed at the local **site** from upstream Python project sources, using their defaults, by the sysadmin (presumably this site-specific binary would also be installed in /usr/local, which is where a further opportunity for confusion arises)

So, I wonder if the right place to put packages when built locally not for distribution as .deb files is to put them in `/usr/local/lib/python3/dist-packages` or `/usr/local/lib/python2.7/dist-packages` The Debian reference says the interpreter will look in `/usr/local/lib/python3/site-packages`[10][10] so I'm not sure how to reconcile that with the discussion in the Debian bug [70][70] other than to note that the bug report is old or that the documentation may be stale. Even so, putting things in `/usr/local/` still seems the way to go? 

This is all still consistent with observations that `/usr/lib/python3/site-packages` did not work, because why would any distribution-supplied packages (the `/usr/lib` part) ever be made to work with a python binary version installed by the site sysadmin (the `site-packages` part)?

I'm just now getting to the Fedora docs about this.[80][80] They bring up the further question of architecture-depended paths (eg, `lib` vs `lib64`) versus arch-independent things (going in `share` rather than `lib`). 

Fedora is also out in front on the attempt to merge things from the `/usr` hierarchy (eg, everything mentioned above) up into the corresponding top-level filesystem directores (eg, into `/lib` instead of `/usr/lib`, `/bin` instead of `/usr/bin`). So who knows how much attention has withered in the face of these added dimensions.[90][90]

A quick check on the machine at hand:

```
$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
$ python
Python 2.7.17 (default, Nov  7 2019, 10:07:09) 
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/home/joe/.local/lib/python2.7/site-packages/osbuild-0.33-py2.7.egg', '/home/joe/.local/lib/python2.7/site-packages/json_format-0.1-py2.7.egg', '/home/joe/.local/lib/python2.7/site-packages/plog-0.16-py2.7.egg', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/joe/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/gtk-2.0']
>>> 
$ python3
Python 3.6.9 (default, Nov  7 2019, 10:44:02) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']
>>> 
$
```

I don't know that I have anything configured in a local-specific way that would put these paths in my sys.path so I think if we could figure out how to put things in these paths it might work, without mucking around in distribution-managed locations like `/usr/lib`.

I've been taking a quick look into the Debian packages `python3`, `python3-defaults` and `python3-minimal` to try to figure out where and how Debian puts `dist-packages` in all these paths since autogen allows us to change the prefix but not this internal path component (I think). So, I'm wondering what we can use from Debian in order to try to support Debian better. My best guess so far is that the trick is somewhere in `/usr/share/python3/runtime.d/public_modules.rtinstall` which seems a thin shell around `/usr/bin/py3compile`.

Anyway, I don't know if that helps anyone else at all but this has been bugging me so I thought I'd take a stab at it. 


[10]: https://www.debian.org/doc/packaging-manuals/python-policy/python.html#paths
[20]: https://stackoverflow.com/questions/9387928/whats-the-difference-between-dist-packages-and-site-packages#9388115
[30]: https://stackoverflow.com/questions/31384639/what-is-pythons-site-packages-directory
[40]: http://lists.sugarlabs.org/archive/sugar-devel/2019-May/056895.html
[50]: https://github.com/sugarlabs/sugar-toolkit-gtk3/pull/383#issuecomment-425647453
[60]: https://github.com/sugarlabs/sugar-live-build/blob/master/src/config/hooks/normal/0900-sugar.hook.chroot
[70]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=765022#30
[80]: https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
[90]: https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/


On Sat, Jan 25, 2020 at 12:22:42AM +0530, soham bhattacharya wrote:
> Thanks, moving the sugar3 directory from /usr/local/lib/python3.6/site-packages/ to /usr/local/lib/python3.6/dist-packages/ did the trick. 
> 
> ----- Original Message -----
> From: "James Cameron" <quozl at laptop.org>
> To: "soham bhattacharya" <116cs0171 at nitrkl.ac.in>
> Cc: "sugar-devel" <sugar-devel at lists.sugarlabs.org>
> Sent: Friday, January 24, 2020 11:35:59 PM
> Subject: Re: [Sugar-devel] getting the Sugar Toolkit working with python 3
> 
> Not without knowing why the files are not found by Python.  Please
> diagnose that.
> 
> See also
> https://github.com/sugarlabs/sugar/blob/master/docs/development-environment.md
> and in particular the note about how Debian and Ubuntu systems are
> preconfigured not to discover any Python package installed by the
> user, because the files are installed to site-packages instead of
> dist-packages.
> 
> On Fri, Jan 24, 2020 at 08:13:17PM +0530, soham bhattacharya wrote:
> > 
> > Okay I tried building the sugar toolkit from https://github.com/sugarlabs/sugar-toolkit-gtk3. As per documentation I used autogen.sh, make, and make install. But still importing sugar3 in python3 does not work. Can you please help me to install sugar toolkit version 0.116 to python3.
> > 
> > Thanks
> > ----- Original Message -----
> > From: "James Cameron" <quozl at laptop.org>
> > To: "soham bhattacharya" <116cs0171 at nitrkl.ac.in>
> > Cc: "sugar-devel" <sugar-devel at lists.sugarlabs.org>
> > Sent: Friday, January 24, 2020 6:34:54 AM
> > Subject: Re: [Sugar-devel] getting the Sugar Toolkit working with python 3
> > 
> > What ppa?  I don't know of any ppa.  Perhaps you mean the Ubuntu
> > package archive?
> > 
> > The Python 3 package name will be python3-sugar3 once Ubuntu accepts it.
> > Unlikely to happen unless someone wants it.
> > 
> > The Python 2 package name is python-sugar3, and you can see the
> > version history in Ubuntu here;
> > 
> > https://packages.ubuntu.com/search?keywords=python-sugar3
> > 
> > 0.112 was packaged in Ubuntu 18.04 and 19.04.
> > 
> > Ubuntu take these packages from Debian.  Debian is yet to move
> > python3-sugar3 from experimental to testing release;
> > 
> > https://packages.debian.org/search?keywords=python3-sugar3
> > 
> > I've proposed a "Debian advocacy for Sugar" project idea for Google
> > Summer of Code this year;
> > 
> > https://github.com/sugarlabs/GSoC/blob/5ac9c642ff8c307e9348c2d357d0241c51d2cab4/Ideas-2020.md#debian-advocacy-for-sugar
> > 
> > On Fri, Jan 24, 2020 at 06:12:39AM +0530, soham bhattacharya wrote:
> > > 
> > > Yes, the installed version was 0.112. I guess the Ubuntu ppa was outdated. I will build and install. Thanks for the help.
> > > ----- Original Message -----
> > > From: "James Cameron" <quozl at laptop.org>
> > > To: "soham bhattacharya" <116cs0171 at nitrkl.ac.in>
> > > Cc: "sugar-devel" <sugar-devel at lists.sugarlabs.org>
> > > Sent: Friday, January 24, 2020 3:10:07 AM
> > > Subject: Re: [Sugar-devel] getting the Sugar Toolkit working with python 3
> > > 
> > > What version of the Sugar Toolkit did you install?
> > > 
> > > When you choose "apt install", you are relying on the Debian or Ubuntu
> > > projects to have packaged Sugar Toolkit.
> > > 
> > > Support for Python 3 was added in version 0.116.
> > > 
> > > Make sure you have version 0.116 at least.
> > > 
> > > On Fri, Jan 24, 2020 at 02:40:54AM +0530, soham bhattacharya wrote:
> > > > Hi,
> > > > I am Soham. I am new to the community and would like to contribute to Sugar Labs. 
> > > > 
> > > > The turtleart-activity(https://github.com/sugarlabs/turtleart-activity) repository mentions a dependency being the Sugar Toolkit. So I went to https://github.com/sugarlabs/sugar-toolkit-gtk3 and installed it using apt install, like it is mentioned in the readme. But whenever I open python3 and try import sugar3, it gives me a ModuleNotFoundError. 
> > > > 
> > > > So can anybody help me with getting the sugar toolkit running with python3. I am sorry if this is a silly question.
> > > > 
> > > > Thank You
> > > > _______________________________________________
> > > > Sugar-devel mailing list
> > > > Sugar-devel at lists.sugarlabs.org
> > > > http://lists.sugarlabs.org/listinfo/sugar-devel
> > > 
> > > -- 
> > > James Cameron
> > > http://quozl.netrek.org/
> > > _______________________________________________
> > > Sugar-devel mailing list
> > > Sugar-devel at lists.sugarlabs.org
> > > http://lists.sugarlabs.org/listinfo/sugar-devel
> > 
> > -- 
> > James Cameron
> > http://quozl.netrek.org/
> 
> -- 
> James Cameron
> http://quozl.netrek.org/
> _______________________________________________
> Sugar-devel mailing list
> Sugar-devel at lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/sugar-devel

-- 
-- 
Joe   On ceding power to tech companies: http://xkcd.com/1118/
man screen | grep -A2 weird
  A weird imagination is most useful to gain full advantage of
  all the features.
Previous message (by thread): [Sugar-devel] getting the Sugar Toolkit working with python 3
Next message (by thread): [Sugar-devel] getting the Sugar Toolkit working with python 3
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the Sugar-devel mailing list