[IAEP] Improving Multi-touch on Sugar

[C. Scott Ananian]

(Changed subject line to separate technical discussion from other discussion
on original thread.)

On Thu, Feb 17, 2011 at 12:12 PM, Gary Martin <garycmartin at googlemail.com>wrote:

> On 17 Feb 2011, at 14:46, "C. Scott Ananian" <cscott at cscott.net> wrote:
>
> On Thu, Feb 17, 2011 at 8:18 AM, Gary Martin <<garycmartin at googlemail.com>
> garycmartin at googlemail.com> wrote:
>
>> On 16 Feb 2011, at 20:35, "C. Scott Ananian" < <cscott at cscott.net>
>> cscott at cscott.net> wrote:
>>
>> I would also be interested in seeing an thorough experience report from
>> someone who has attempted to use Sugar on a touchscreen device.  We already
>> know that several major features (such as the frame and hover menus) fail
>> completely.
>>
>>
>> FWIW neither of those are particularly challenging design cases. The frame
>> could be triggered by a hardware button, and the 'touch & hold' interaction
>> will work just great for the hover menu case.
>>
>
> This is an encouraging report.  "Touch and hold" isn't discoverable, though
> (you end up clicking on the button when all you wanted to do was see the
> drop down) and in general sugar uses a lot of hover interaction to help kids
> find what the active parts of the UI are.  This doesn't work on
> touchscreens.
>
>
> If 'touch and hold' becomes the UI interaction that's currently covered by
> right clicking or cursor hover and loiter, it would be useful to introduce
> this technique at an early stage, much like Apple did with their 'swipe to
> open' UI switches. Imagine when the tablet is first powered up, or unlocked*
> from a screen powered off state — the user is presented with a large central
> button UI 'touch and hold to unlock' that when touched rotates to an
> unlocked position with a satisfying unlock click feedback. If released too
> early the button rotates back to its locked state and a 5-10 sec before a
> screen off power saving sleep is triggered.
>

This isn't entirely what I meant (although it's one component).  My main
objection was that there's no way to discover which elements might have a
"touch and hold" action without actually attempting to touch and hold on
every location on the screen.  If the thing you just touched *doesn't* have
an "and hold" menu, then you've just clicked the button.  Oops!  Hope you
didn't mind the effect!

There are also basic touch interactions missing, just as swipe-to-scroll.


> OT there are current Sugar button UI cases I've been hoping to see cleaned
> up (SL#2517, #2518). We have a number of buttons that only have a hover or
> right click interaction that should be functioning as well with a left click
> (pop-up palette where button has no primary action). These do cause
> confusion as lack of left clicking interaction leads you to believe there
> are hidden bits of the UI that look like buttons but don't act like them.
>

Those are exactly the 'safe' buttons to touch-and-hold just to see what will
happen! ;-)  So 'fixing' the bug will cause touch interaction/discovery to
actually get worse.


> Also, integration of the keyboard is a huge open question.  As detailed at:
>    <http://wiki.sugarlabs.org/go/Features/Onscreen_Keyboard>
> http://wiki.sugarlabs.org/go/Features/Onscreen_Keyboard
> "One major drawback of an on-screen keyboard is that in its current design,
> it blocks out a part of the Sugar UI. There's no immediate answer on how to
> handle this problem."
>
> Activities would need to be resized to accommodate the keyboard, at the
> very least.
>
>
> The trick here is not to resize/reflow anything, just translate the
> activity window vertically (and partially off screen) so that the text input
> area cursor always appears in the letterbox space still visible above the
> keyboard.
>

I do not think this is possible without a lot of changes to Sugar and to the
activity.

And even after you've done this, the experience tends to be suboptimal.
 (Even my iPhone often botches the keyboard scroll and/or blocks the thing I
need to see while I'm typing.)


> I think there's a big difference between "I tried it, and it sorta worked"
> and "I tried it, and it was a great experience".  Is it realistic to think
> that Sugar can be a "great experience" on a touch device with only minor
> tweaks here and there?  Or are we going to miss out on the exhilarating
> "direct interaction" feeling possible with a tablet and just produce a
> device that will frustrate and confuse kids?
>
>
> I don't think there are insurmountable Sugar UI issues, though I do worry
> some of the upstream components we rely on may be bottlenecks, but that's
> just because I'm not up to speed with where Fedora, GTK, X, et al are with
> regard to multi touch. Also worth mentioning that smooth, fast visual
> feedback is critical, especially in scroll views, so good HW accelerated gfx
> drivers are even more important (fingers crossed for alpha compositing and
> maybe some 3d for simple transitions — wouldn't it be great to flip over an
> Activity window to edit its datastore details/tags view on the back).
>

The short list of changes for a multitouch sugar are:

a) pygtk to gobject-introspection migration
b) hippocanvas migration (maybe unnecessary if hippocanvas can be ported to
gtk3)
c) GTK2 to GTK3 migration
d) (gconf to gsettings, python2 to python3) <- probably not actually on
critical path
e) fix all the multitouch-hostile UI elements!

That's a lot of work.

The continuation of this line of work would:
f) modularize sugar so that sugar services like collaboration and journal
are separate from sugar ui elements and separate from sugar miscellany
g) investigate running sugar/GTK3/cairo with different backends (ie, in a
browser)
h) move xlib/keyboard/display lang conf to gnome-settings-daemon plugins
i) replace journal with http://zeitgeist-project.com/ backend
j) integrate cjb's multi-pointer VNC work for pervasive collaboration

These aren't all my ideas; I've borrowed heavily from conversations with
cjb, erikos, jnettlet, and others.

This is *also* an aggressive development plan which will break many things
and require changes to support documentation, etc.  Which returns to my
original question: it is better to concentrate on porting *just the
activities*, and just adopt a better (multi-touch-aware from the beginning)
underlying OS/windowing system/system configuration tools from
ChromeOS/Android/MeeGo/etc.

But let's ignore that last question (in this thread; we can continue to
discuss it in the 'Google-ize' thread), and concentrate on "what needs to be
done to pull Sugar kicking and screaming onto HEAD of its various dependent
libraries/languages"?
  --scott

-- 
                         ( http://cscott.net/ )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sugarlabs.org/archive/iaep/attachments/20110217/5383f40c/attachment-0001.html>