[sugar] Pie menus for pygtk and OLPC Sugar!

[Eben Eliason]

We discussed the possibility of using pie menus early in the design
stages; I'm not sure why they went out.  After considering many of the
interaction design issues we've been facing lately, I'm now thinking
that they might have their place on the laptops.

The following may require some familiarity with the terms discussed in
the OLPC HIG: http://wiki.laptop.org/go/OLPC_Human_Interface_Guidelines

HOW

I think that pie menus are a nice design concept, though their
idiosyncrasies can be a bit tricky.  I've spent the past day going
over every aspect of the interface with pie menus in mind, and have
come up with a plan for their integration with the current system
which should improve usability.  There are many ways to configure pie
menus, but I think the following will handle most cases with the
simplest interaction.

1. Primary/Secondary/Tertiary

We can create a format for describing the menus that includes keys
which indicate an item's level of importance as either primary,
secondary, or tertiary.  There will be 4 primaries, 4 secondaries, and
any number of tertiary slices (space allowing, we'll probably set some
max, perhaps 8).  The idea here is that each menu could have 3 levels
levels of detail: in primary mode you would see N, S, W, E; in
secondary mode you would gain NW, NE, SE, SW; and in tertiary mode you
would see all options.  The advantages of this will be seen later in
the "WHERE" section.

2. Activation methods

There are a few ways that pie menus can be activated.  Generally
speaking, the right mouse button will be used to activate pie menus,
though some may appear on rollovers of Objects after short delay.

When the menu is explicitly triggered, click and hold is the best
method of interaction.  When the mouse remains pressed, the selection
region of the slice can extend beyond the drawn graphic of the menu
right to the edges of the screen.  This means that the hit area is
virtually infinite, and lends itself well to gestural selection for
more advanced users.  In order to prevent selection, the child could
release the mouse within the interior of the ring, or could press the
esc key to cancel the operation and hide the menu before releasing the
mouse.

Unfortunately, when the mouse button is not held, extending the hit
area doesn't work as well, since the selection will have to happen on
a click event instead of a release.  When clicking beyond the rendered
region of the pie, users will expect the menu to simply disappear,
with no selection.  Therefore, in cases where the pie menu appears on
instantaneous click or on rollover, the hit area of the slice will be
limited to the graphical region only.

3. Visual design

Everywhere in our interface icons are exposed in the first layer and
text is hidden, exposed on rollover as the secondary layer.  This
makes things easier for pies, since the biggest problem with them is
dynamically scaling them to fit strings of text.  Instead, we can
simply show icons of a fixed size equally spaced around the menu.
This is of great benefit to us, since dynamic rendering of the curves
could be costly.

We can therefore reveal the text for a menu item in one of two ways.
It could appear in the center of the pie, adjusting to show the text
for the currently highlighted slice, or it could appear as a tooltip
next to the mouse.  The latter might actually be better, since it will
appear next to the cursor, which is the child's point of focus, and
because it won't obscure the object in the center that the menu
relates to.  If we do this, the rollover does need to be instantaneous
so that the menus are quick at revealing their options.

At any point of interaction, the slice that will be selected upon
release of the mouse, or on click (depending on the way the menu was
invoked) will be highlighted.

4. (Lack of) Hierarchy

Everywhere in the laptop we are striving to reduce hierarchy, and make
everything that's important available with as little "digging" as
possible.  As such, the pie menus won't be hierarchical.  Instead,
each slice will either be an instantaneous action, or will have a
single secondary panel.  When a slice has a secondary panel, it will
have a small arrow which indicates this.

The secondary panel can function in one of two ways.  First, it can be
a popup list, which is useful when the action needs a target (eg. send
to...(my friend) or open with...(activity)).  In these cases, a list
will appear adjacent to the wedge allowing the selection of the
target.

A secondary panel can also function as a control window.  This means
that a panel will appear, containing any number of necessary controls,
much like a popup window might.  This is useful for things like a
preference pane, or a color chooser, etc, where more options and
controls might be needed.

WHERE

Pie menus are always tricky around screen edges, and I'm not a fan of
any of the design solutions that aim to fix this.  As such, I'm
proposing two very specific yet useful cases where these menus will be
used.  They are not a general purpose replacement for all menus in my
mind, but their benefits can be maximized in some cases.  Here is a
description of where I think these fit in the Sugar UI.

1. Contextual Menus/Rollovers

The current laptop design includes the notion of rollovers for
Objects, which typically exist in two states.  A primary rollover
appears to provide a text title for the Object, which is otherwise
rendered as an icon only, and a secondary extension of this appears
after a short delay to provide a preview, summary, and a list of
actions associated with the Object.

All contextual rollovers should be replaced with pie menus.  The menu
itself would appear, of course, surrounding the object so the icon is
still visible in the center.  The text description of the object the
menu relates to can appear with the icon, in the center of the menu
(eg. the child's name when rolling over an XO).

This menu could contain a list of primary actions associated with the
object.  For instance, for an XO you could "send message", "make
friend", "invite to activity", etc.  Each object type might have a
slightly different list of actions, but there will certainly be
overlap.  Where possible, we will define the arrangement of these
actions such that they always appear in the same place, regardless of
the object itself.  This is important for spatial recognition, making
gestural activation natural everywhere in the interface.  Slices for
actions that don't apply to an object will be left empty.

The only drawback to this over the previous rollover design is that
the actual preview image or summary won't appear automatically after
delay.  Instead, the preview would appear as the secondary panel of a
"preview" action slice in the menu, which would of course be
consistently positioned for every object.  The advantage, of course,
is that taking actions on objects is actually quickly exposed, whereas
before they were confined to a secondary rollover state.

2. Activity Menu

Until now, we didn't really have a menu system per se for Activities
at all.  Instead, we have a toolbar, where buttons/controls in the
toolbar can have rollover states which provide panels with additional
controls contextual to the button itself.  There is a file menu of
sorts in the frame, providing basic features like keep (save), share,
etc, but nothing that the activity developer could extend in any way.

I propose that we allow each activity to define its own activity pie
menu.  This menu could be invoked by right clicking anywhere on the
screen where there isn't an object with its own contextual menu.  It
could contain any number of items, ranked by the three levels
mentioned above.  There could perhaps be a method of selecting which
level of detail you want to see through the interface, but I have
other ideas for this as well.

This activity menu can function in two powerful ways.  The first is
that it provides a place for activities to define additional
functionality.  Until now, with no menu system, everything had to be
exposed on screen.  This is a goal we still want to shoot for, and it
should be known to developers that their activities should function
completely without need for the activity menu at all.  However, when
additional advanced controls or options would benefit experienced
users, this menu could provide an outlet for some of them without
cluttering the toolbar itself.  By design, I would recommend that such
advanced features be ranked as tertiary items.

The second and actually more powerful way to use these menus in my
mind is, rather than for advanced functionality, for limited basic
control of the interface.  As an example, consider the web browser.
The basic set of controls might be "forward" (right), "back" (left),
"reload" (up), "view source" (down).  By defining these as the
primaries, I can instantly invoke these frequently used commands with
gestural ease, without even needing to mess about with toolbar
buttons.  This kind of gestural activation of frequent actions could
be fantastic.  Similarly, a drawing activity might define "circle",
"square", "line", "brush" as it's basic set of tools, making selection
of the needed tool a breeze without needing to leave the region of the
drawing being worked on.

3. Handheld Mode

Finally, a fantastic side effect of the activity menu is its
application in handheld mode.  Previously, we had stated that
activities would have to explicitly implement handheld controls, since
there would be no mouse and keyboard.  However, when the activity
menus are designed such that the primary items are the basic key
controls for the activity, any activity can run in handheld mode, at
least in some form.

We could make the circle button on the right the menu button.  Press
and hold this button to display the pie menu, and use the directional
pad to select the option you want.  This is where the ranks for items
also comes into play.  Since we can read 8 directions from the d-pad,
we can show either primary or secondary levels of the menu.  Of
course, activities could choose to use this button for something other
than the pie menu if they wish - in games, for example - but this
would be a really nice default that would make most activities play
nice with handheld mode "out of the box."


I look forward to hearing what everyone thinks.  Also, Marco, I'd like
to hear your thoughts on all of these ideas from an implementation
standpoint.

- Eben




On 2/23/07, Marco Pesenti Gritti <mpg at redhat.com> wrote:
> On Thu, 2007-02-22 at 01:55 -0800, Don Hopkins wrote:
> > I've translated my fancy OpenLaszlo pie menu component to
> > Python/gtk/cairo, so they work with the OLPC Sugar user interface!
> >
> > These pie menus have an adaptive layout algorithm, that support linear
> > menus which can lay out in all four directions, pie menus, and hybrid
> > pie/linear menus (automatically limiting the number of pie items to 8,
> > and formatting overflow items in as linear menu items below the pie).
> >
> > I'm just learning pygtk, so they aren't as integrated or as fleshed out
> > as they should be yet, and they don't actually do anything but print
> > debugging messages so far, but this is a good first cut that's demoable.
> >
> > I like the cairo drawing module! It's easy to use, and the graphics and
> > feedback highlighting looks nice.
> >
> > It's all written in one file of Python code.
> > You should be able to run the demo, which is included in the file, just
> > by going "python piemenu.py" on an OLPC or Python system with pytgk and
> > cairo installed.
> >
> > I'd really appreciate it if anyone who knows pygtk would look at the
> > code and give me some feedback about what I'm doing wrong and how to
> > improve them, please.
> >
> > Next I have to flesh out input tracking and notification, and figure out
> > how to send events during tracking when they pop up, the mouse moves,
> > item changes, when items are selected, sub menus are selected, etc.
> > And then of course there's a some frosting to implement like keyboard
> > navigation, submenu browsing, screen edge handling, accessibility, popup
> > animation, titles, tooltips, icons, shaped windows, disabled items, etc.
> > Also I'd like to integrate them with glade and the persistence stuff so
> > you can load them in from xml files, edit them with a property sheet,
> > and even edit live menus with direct manipulation, etc.
>
> Hello,
>
> this looks pretty cool!
>
> I just wanted to mention Sugar is not going to use gtk widgets (and
> glade) as the official/supported way to build activities UI. We are
> developing a set of controls based on HippoCanvas.
>
> Compared to gtk widgets it's lighter, it integrates better with cairo,
> it has more powerful layout support and can also embed gtk widgets. On
> the down side it's far from gtk maturity, the only documentation is code
> at the moment and it miss basic features we will have to implement down
> the road (mainly focus).
>
> It looks like adapting your pie menus implementation to HippoCanvas
> would be trivial.
>
> Marco
>
> _______________________________________________
> Sugar mailing list
> Sugar at laptop.org
> http://mailman.laptop.org/mailman/listinfo/sugar
>