[Sugar-devel] Language section debugging, use ICU?

[Chris Leonard]

On Tue, Mar 26, 2013 at 3:17 PM, Manuel Quiñones <manuq at laptop.org> wrote:
> I have ongoing work on polishing the Language section of the Sugar
> settings panel.  I'm sharing my findings to open discussion, to start
> bringing back discussions to the mailing list, and to encourage
> testing of my patches.
>
> First, an introduction of the issues, ordered by priority as I understand:
>
> 1. list of languages: how could I select my language if it is
> displayed in another language?
>    http://bugs.sugarlabs.org/ticket/51
>
> 2. list of languages: if there are two languages and English is the
> first one, the list is displayed as the second one
>    http://bugs.sugarlabs.org/ticket/4327
>
> 3. many language names are not translated
>    http://bugs.sugarlabs.org/ticket/4449#comment
>
> 4. the section takes a lot of time to start, we should display a
> watch/busy cursor while it is loading
>    https://bugs.sugarlabs.org/ticket/245
>
> The issues are interconnected as you will find if you try to read the
> many comments, which mix them all through the years :)
>
> A brief of the current implementation:
>
> A. we parse the output of 'locale -av' command and create a list of
> (language name, territory name, locale code)
>    https://git.sugarlabs.org/sugar/mainline/blobs/master/extensions/cpsection/language/model.py#line33
>
> B. we call gettext with ISO-639 to translate the language name
>    https://git.sugarlabs.org/sugar/mainline/blobs/master/extensions/cpsection/language/view.py#line28
>
> C. we call gettext with ISO-3166 to translate the territory/country name
>    https://git.sugarlabs.org/sugar/mainline/blobs/master/extensions/cpsection/language/view.py#line29
>
> Number 2 gets obsolete if we solve no. 1.  The problem in 2. is that
> we should not use gettext to translate strings to English, as they are
> already in English.  It is iliustrated in this comment:
> http://bugs.sugarlabs.org/ticket/4327#comment:6
>
> Number 3 is a flaw of the current implementation: the output of
> 'locale -av' does not match 100% the strings in the po files for the
> given gettext domains.  See comment by Chris Leonard on this:
> http://bugs.sugarlabs.org/ticket/4449#comment:9
>
> Number 4 has a general patch that works for all sections, except for
> Modem section.  We actually have code that displays a busy cursor, but
> it is shown only for an instant, because (fortunatly) the UI doesn't
> block the program.  The patch wraps the initialization of the section
> in a GObject.idle_add call, to make it work.  As said before, this
> conflicts with the current implementation of the Modem section.  So..
> still work to be done on this one.
>
> By the way, number 4. gets less relevant if we speed up the section.
> That leads me to talk about my findings on issue number 1:  I have
> investigated alternatives to our current gettext implementation.
>
> - using gettext domain ISO 639-3 instead of ISO 639 for translating
> the language name
> - using external libraries Babel and PyICU
>
> Here is a table of my results:
>
> https://docs.google.com/spreadsheet/ccc?key=0Auk55vVISSpndEdnbDA4OHJCelRwUHlKbmFNQVBsSkE#gid=0
>
> And here is the result of profiling them:
>
> http://bugs.sugarlabs.org/ticket/4449#comment:12
>
> So, it looks like the ICU project is very fast and provides good
> output.  And reading the project homepage it looks on shape too.  Can
> we consider a switch to it?
>
> --
> .. manuq ..

Manuel,

First, thank you for exploring the question of improving the selection
of languages from the control panel and the several issues involved.
As you correctly note in your message, there are actually a series of
intertwined issues and it is, of course, important to be clear about
which issue is being targeted by any given approach proposed and the
impact of a given approach on the related issues.

Starting this line of investigation from bug #4449:

"Spanish" and other language names are not translated in My Settings
language section

bugs.sugarlabs.org/ticket/4449

The questions that you have most directly addressed with your testing
so far are:

1) How does Sugar currently populate the list of language names (and
territory names) in the language selection Control Panel?

For which you describe a series of lookups from 1) the glibc locale
itself 2) ISO-639 for language name, 3) ISO-3166 for territory name.

2) Is this the most efficient (fastest) method of retrieving a
localized language name?

You decribe your experimentation with ICU.  It is important to note
that the source of information for ICU on localized language and
territory names is the CLDR locale, and so, at least in part, this
comes down to a discussion about glibc locales versus CLDR locales.

I would like to suggest that there is another question (issue) in
play, which is

3) What is the most authoritative and complete source of localized
language names available (without regard to the performance cost of
using it)?

My concern with using CLDR locales via ICU is that although they may
have some nice features, like containing chunks of ISO-639 and
ISO-3166 (which probably accounts for some of the observed speed-up),
the coverage from locale-to-locale on what language and territory
names is translated can be highly variable.

Using glibc locales is the "Linux" standard, whereas CLDR locales are
more of a Mozilla or Android approach.  Although the amount of L10n on
the ISO-639 and ISO-3166 projects can also be variable by language, at
least the untranslated values are there in the PO files, pointing out
the need for more work.

glibc locales versus CLDR locales were the subject of a very brief
discussion on libc-alpha (the main glibc list) [1-5]

[1] http://sourceware.org/ml/libc-alpha/2013-03/msg00406.html
[2] http://sourceware.org/ml/libc-alpha/2013-03/msg00411.html
[3] http://sourceware.org/ml/libc-alpha/2013-03/msg00430.html
[4] http://sourceware.org/ml/libc-alpha/2013-03/msg00435.html
[5] http://sourceware.org/ml/libc-alpha/2013-03/msg00481.html

cjl