[sugar] Pie menus for pygtk and OLPC Sugar!
Sun Feb 25 02:00:26 EST 2007
These sound like great ideas!
The best thing to do is to code up some of these ideas and see how they
look and feel, since it's a visceral thing that is hard to think through
in your head, without having something concrete to really play with.
I'll comment on Eban's message and Marco's reply at the same time.
Marco Pesenti Gritti wrote:
> On Fri, 2007-02-23 at 15:39 -0500, Eben Eliason wrote:
>> Pie menus are always tricky around screen edges, and I'm not a fan of
>> any of the design solutions that aim to fix this. As such, I'm
>> proposing two very specific yet useful cases where these menus will be
>> used. They are not a general purpose replacement for all menus in my
>> mind, but their benefits can be maximized in some cases.
> I fully agree on this, so I'm going to comment on the specific cases.
One solution to the screen-edge problem is to move the menu so it
completely fits on the screen, and then to warp the mouse by the same
offset. But that has to be done very carefully and synchronously to
avoid any problems that would interfere with mouse-ahead. You can defer
popping up the menu until ALL events have been processed and the mouse
is actually paused, and only then move the cursor instantly after
popping up the menu, so the user can mouse ahead without popping up the
menu or warping the mouse near the screen edge. I agree that there are a
lot of undesirable things about that solution, which is quite complex,
violates the principle of least astonishment, and is even impossible to
implement in web browsers and other environments that don't let you warp
the mouse (and on hardware like absolute position input devices like
touch screens that do not have robot arms to reach out and move the
user's finger on the screen, instead of warping the mouse.)
Another approach to the screen edge problem, which I've been using on my
Pocket PC implementation of touch screen pie menus, is simply to not use
the directions that are off the edge of the screen, for pie menus that
pop up along the screen edge or in corners. You can put 1/2 pie menus
along the edges (calzone menus?), and 1/4 pie menus in the corners
There is of course the disadvantage that fewer directions can fit in
partial menus, but there is also the slight advantage that when the user
starts the gesture on the edge or in the corner of the screen, they have
the whole rest of the screen to move out into, to increase the angular
precision of the selection. Edge an corner menus allow you to move a
greater distance, for more "leverage". (This applies more in the case of
a small finger-operated touch screen with full-screen pie menus, since
most of the screen is very near a corner or edge!) So you might be able
to get away with putting a few more items-per-degree on edge and corner
menus, than in centrally triggered menus (which only allow you half a
screen of leverage).
You can also use the distance independent of the direction, to "pull
out" sub-items, so you could have a corner pie menu with three different
directions (right, down, and diagonal), or an edge pie with five
different directions (left, down/left, down, down/right, right), but
each of those distinct directions could "pull out" many different
sub-items, because you have the whole width or height or diagonal of the
screen to pull out.
So one feature I'd like pie menus to support is limiting the start and
end direction of the layout, to support fractional pie menus for the
screen edges and corners.
A couple of complementary (but mutually incompatible) features for
exploiting the distance of motion are "pull out" slices with sub-items
selected by the linear distance from the menu center, and also "layered
pies" with rings that have different numbers of items in them, so the
inner-most ring can have four items, the next ring 8 items, the next
ring 16, and so on.
>> 1. Contextual Menus/Rollovers
>> The current laptop design includes the notion of rollovers for
>> Objects, which typically exist in two states. A primary rollover
>> appears to provide a text title for the Object, which is otherwise
>> rendered as an icon only, and a secondary extension of this appears
>> after a short delay to provide a preview, summary, and a list of
>> actions associated with the Object.
>> All contextual rollovers should be replaced with pie menus. The menu
>> itself would appear, of course, surrounding the object so the icon is
>> still visible in the center. The text description of the object the
>> menu relates to can appear with the icon, in the center of the menu
>> (eg. the child's name when rolling over an XO).
I think it's important to trigger pie menus on a mouse click (and
control them by the instantaneous direction between clicks, but NOT the
path taken, in order to allow re-selection and browsing), and to center
them on the exact position of the mouse click. The user should have a
crisp consistent mental model of how pie menus work (which is NOT the
case for gesture recognition). Pie menus should completely cover all
possible "gesture space" with well defined behavior (by basing the
selection on the angle between clicks, and not the path taken). In
contrast, gesture recognition does NOT cover all gesture space (because
most gestures are syntax errors, and gestures should be far apart and
distinct in gesture space to prevent errors), and they do not allow
in-flight re-selection, and they are not "self revealing" like pie menus.
Pie menus are more predictable, reliable, forgiving, simpler and easier
to learn than gesture recognition, because it's impossible to make a
syntax error, always possible to recover from a mistaken direction
before releasing the button, and "self reveal" their directions by
popping up a window with labels, and they "train" you to mouse ahead by
Roll-overs that pop up pie menus before you click the mouse could be
kind of weird, and could violate the principle of least astonishment.
The way some web sites implement roll-over pop up linear menus on web
pages is pretty annoying, because the menus often cover up something
else that you meant to select, and that is frustrating if you just meant
to move the mouse over the menu trigger but not pop up the menu.
So whatever design you use for roll-over feedback, it should be sure to
re-center the pie menu on the current cursor position (or god forbid
warp the cursor to the menu center where it's currently displayed), and
not get in the way of reliable mouse-ahead. I can imagine a kind of
light weight roll-over feedback that previews the pie menu and reveals
its directions without actually popping up the menu itself or occluding
the view of the object you're pointing to.
There is an issue (known as the "nulling problem") that I've had to deal
with for small touch screen pie menus, because you can't see through
your finger, and the computer can't forcefully reposition it
(hopefully). One solution I've developed for my Pocket PC pie menus is
for the pie menus to take over the entire screen, and to lay out the
items around the edges of the screen (instead of centering them on the
place where your finger touched the screen). Then it draws some
light-weight feedback that looks like cracks in the glass centered on
your finger touch location, that do not occlude the menu labels (the
crack feedback is drawn underneath the labels). The cracks delimit the
edges of the target regions, and show you where on the screen to touch
to select the different items, while the labels are highlighted
according to the direction selected by your finger motion, but the
labels are always justified in the same place around the screen edge no
matter where you touch. It also draws some animated "grippy" lines in
the selected slice orthogonal to the slide direction, that slide in and
out to show you how it's tracking the distance inside of the slice.
That's one solution to the nulling problem for small touch screens
(de-couple the touch feedback from the menu label position), which might
work with the OLPC's small screen, or as roll-over feedback.
Some interesting info on the "nulling problem" and other issues, by Bill
Buxon: There's More to Interaction than Meets the Eye: Some Issues in
The stuff he writes about chunking and phrasing of gestural input is
also very important for pie menus!
>> This menu could contain a list of primary actions associated with the
>> object. For instance, for an XO you could "send message", "make
>> friend", "invite to activity", etc. Each object type might have a
>> slightly different list of actions, but there will certainly be
>> overlap. Where possible, we will define the arrangement of these
>> actions such that they always appear in the same place, regardless of
>> the object itself. This is important for spatial recognition, making
>> gestural activation natural everywhere in the interface. Slices for
>> actions that don't apply to an object will be left empty.
That's a good idea! I made a visual PostScript interface that had a
standard pie menu that you could pop up on any object, but which had a
submenu in a standard direction that always popped up an
object-type-specific submenu. So the top level menu was always the same
and had common commands, and the object specific menu was always close
by in the same direction.
Here's a paper about it that discusses the pie menu design, and has some
And here are some videos:
>> The only drawback to this over the previous rollover design is that
>> the actual preview image or summary won't appear automatically after
>> delay. Instead, the preview would appear as the secondary panel of a
>> "preview" action slice in the menu, which would of course be
>> consistently positioned for every object. The advantage, of course,
>> is that taking actions on objects is actually quickly exposed, whereas
>> before they were confined to a secondary rollover state.
This is definitely something that's worth implementing in a few
different ways and playing around with to see what feels good.
> I'm not fully convinced about this one.
> * As we discussed it would probably not be possible to use it on the
> frame (since icons are near the edges).
> * It would make sense in the Friends view. Though we allow to reposition
> Xo's there, so again, they could be near the edges.
> * What are the advantages to use them in the mesh? The two primary use
> cases seem 1 explore and 2 join activities. Pie menus doesn't seem to
> make 1 easier (you can just click on the activities) and probably makes
> 2 a little harder (more indirect preview).
> * In the current mesh design you can easily browse through participant
> names with the little animation you sketched out in the flash demo.
> Would that be as easy with pie menus?
Pie menus are best when the number of items is constant and the menus
don't change and aren't based on dynamic data. That doesn't mean you
shouldn't ever do that, just that they're best when they're predictable,
and a real human being has thoughtfully planned them out (which is
impossible for dynamically generated menus). The Sims uses pie menus
whose items are dynamically generated, so they aren't able to fully
exploit the advantages of muscle memory. The game controls the order of
the pie menu items which are layed out in a circle clockwise starting
from the top, but it does not have any explicit control over the
direction of each item. Some objects have stable pie menus that are
predictable (like the fridge and the toilet), where an item in one mode
will be replaced by another item in another mode (the fridge lets you
fix breakfast/lunch/dinner depending on the time, and the toilet has
use, seat up/down, followed by optional flush at the end of the menu).
The pie menus automatically add dummy items to force the number of items
to be even (2, 4, 8, and more than 8 go into the linear overflow items).
We discussed how to enable the game to control the directions of menu
items, but there was no easy solution that didn't place a burden on the
game designers defining the actions on the objects, so we just went with
layout of each item with parameters like linear (true for linear, false
for pie), "max_pie_items" to limit the maximum number of pie items
before making items linear, linear_direction (a string specifying the
north/up/south/down/east/left/west/right/nw/ne/sw/se linear layout
direction for the item) (menus can have mixed pie menus and linear items
in different directions), "linear_order_reversed" to reverse the linear
layout order (to flip "up" layout so upside-down pull-up menus come out
in top to bottom order).
There is some rudimentary stuff in there for paging pie menus, to
support arbitrarily large numbers of items in pages of 8 (or so) at a
time, but it's not hooked up to the user interface or well tested. The
idea is that there should be some kind of paging user interface (page
up/page down keys, as well as arrow buttons somewhere on the screen, or
an inner ring of four pie menu paging items: first/last/next/previous).
I've tried scrolling menus that let you wind the cursor around the
center to "spiral" through a long list of items (demonstrated on Brad
Meyer's "All the Widgets" video he made for ACM SIGCHI), but that turned
out to be kind of weird... (But maybe it's good for scrolling through
This clip from All the Widgets demonstrates the scrolling spiral pie
menu for selecting country flags, a color wheel pie for changing the
text and background color in Emacs, pull-out font family/style/size pie
menus, and a "precision pie menu" that lets you precisely select any
number from 0 to 360 (by "poking" out through the menu bubble with a
flexible rubber band), and also a pie menu window manager supporting
>> 2. Activity Menu
>> Until now, we didn't really have a menu system per se for Activities
>> at all. Instead, we have a toolbar, where buttons/controls in the
>> toolbar can have rollover states which provide panels with additional
>> controls contextual to the button itself. There is a file menu of
>> sorts in the frame, providing basic features like keep (save), share,
>> etc, but nothing that the activity developer could extend in any way.
>> I propose that we allow each activity to define its own activity pie
>> menu. This menu could be invoked by right clicking anywhere on the
>> screen where there isn't an object with its own contextual menu. It
>> could contain any number of items, ranked by the three levels
>> mentioned above. There could perhaps be a method of selecting which
>> level of detail you want to see through the interface, but I have
>> other ideas for this as well.
That would be cool! Note that the Python pie menus also support linear
layout as well as hybrid pie/linear layout, and the application
developer can control which items go into the pie and linear parts, and
what directions they're laid out in.
Of course it would be nice to have even more control of the pie layout,
so you can tag pie items with preferred directions and priorities, and a
smart layout algorithm could resolve the items into appropriate
directions, keep the number of items even by inserting inactive items in
unused slots, and apply hints like preferred direction or axis, keeping
opposite pairs in opposite directions, etc.
For example, you could add "on" and "off" as a pair, preferring up for
on and down for off with a high priority, then add "next" and "previous"
as a pair with no preference for direction, and four other items with no
pairing or directional preference but a lower priority, so the "next"
and "previous" would be placed left and right because they were more
important than the unpaired items, which would go along the diagonals.
If you limited the number of pie items to 8, and added some even lower
priority items, they would be relegated to linear items, or to an outer
There should be some way to set up the maximum number of pie items in
each ring of a multi-layered pie menu, like the number off electrons
allowed in each ring of an atom, and a limit on the maximum number of
rings, and also a way to control the linear layout of items that don't
fit in the rings.
> Sort of OT but I really dislike the idea of selecting the level of the
> detail of the interface. It's just a question without a good answer. If
> we decide to go this way we should at least analyze the previous
> attempts to this kind of interface and figure out why they failed.
>> This activity menu can function in two powerful ways. The first is
>> that it provides a place for activities to define additional
>> functionality. Until now, with no menu system, everything had to be
>> exposed on screen. This is a goal we still want to shoot for, and it
>> should be known to developers that their activities should function
>> completely without need for the activity menu at all. However, when
>> additional advanced controls or options would benefit experienced
>> users, this menu could provide an outlet for some of them without
>> cluttering the toolbar itself. By design, I would recommend that such
>> advanced features be ranked as tertiary items.
> This make sense to me but we have to define clearly the use cases. I
> don't want activities to come with dumb interfaces and then put all sort
> of "advanced" features in the pie to work around the design
In the ideal world, all application designers would think carefully
about the common use cases, make the interface obvious, self revealing,
and usable without menus, and design nice static or smart dynamic pie
menus that were easy for novices to learn yet work well as accelerators
for expert users (like the X11 SimCity pie menus that are shortcuts for
the tool palette).
Here's a video of the pie menus in the X11 version of SimCity, which
shows how they support mouse-ahead, and work as an optional accelerator
for the tool palette, which can be tucked away to save screen space once
you're used to the pie menus. Notice the funky "totem pole" layout of
the tool palette icons, which are different shapes and sizes according
to how they're used and how much they cost. And how they tool palette is
arranged in the same pattern as the pie menus, serving as a map to help
you learn the pie menus.
In another other ideal world (which is ideally the same ideal world as
the first ideal world, but it's too idealistic to expect that), the user
should be able to edit the user interface to customize it for their own
particular usage patterns (like Maya's marking menus that the 3D artist
can tailor to their workflow).
In a game like World of Warcraft (a real-world virtual universe, but
less-than-ideal world), each user has a very particular set of commands
they want to use at any particular time (because characters play
different roles and have different abilities, spells and items, which
changes over time and with different activities), so it's absolutely
necessary to support user-defined interfaces constructed by dragging and
dropping command icons around to create your own custom toolbars and pie
menus. (I've been thinking about how design user-editable pie menus for
World of Warcraft, but I've been too busy trying to get my character to
level 70 to actually implement WOW pie menus yet. But it would be easy,
given a good design, since WOW's user interface is fully programmable in
Lua, and already supports dragging and dropping action icons into user
>> The second and actually more powerful way to use these menus in my
>> mind is, rather than for advanced functionality, for limited basic
>> control of the interface. As an example, consider the web browser.
>> The basic set of controls might be "forward" (right), "back" (left),
>> "reload" (up), "view source" (down). By defining these as the
>> primaries, I can instantly invoke these frequently used commands with
>> gestural ease, without even needing to mess about with toolbar
>> buttons. This kind of gestural activation of frequent actions could
>> be fantastic. Similarly, a drawing activity might define "circle",
>> "square", "line", "brush" as it's basic set of tools, making selection
>> of the needed tool a breeze without needing to leave the region of the
>> drawing being worked on.
> I really like this use case.
Me too! We used pie menus for paging and navigation in HyperTIES
hypermedia browser, and the Emacs authoring tool, which I developed for
Ben Shneiderman at the University of Maryland Human Computer Interaction
Designing to Facilitate Browsing: A Look Back at the Hyperties
*3. PIE MENUS TO PERMIT LOW COGNITIVE LOAD ACTION*
The Hyperties browser uses pie menus as accelerators, to make commonly
used commands quickly and easily available. A pie menu is a type of pop
up menu whose selections are laid out in a circle around the menu center
(18). The menu pops up centered on the cursor, so that each selection is
adjacent to the cursor but in a different direction (Figure 1 and 6). A
selection is made by moving the cursor in the direction of the desired
selection, and clicking. Experienced pie menu users can make selections
from familiar menus quickly and reliably without even having to look at
the menu, because the menu selection depends on the direction between
the two mouse clicks that invoke and select from the menu. The distance
of cursor motion does not effect the selection, but the further away
from the center the cursor is, the more precise the control of the
The browser has a control panel at the bottom of the screen, with
buttons showing the names of available commands, to turn the page,
return to the previous article, show the index, etc. When users are
browsing a document by pointing and clicking on highlighted text links
in the main contents window, they move the cursor down to the bottom of
the screen to press buttons in the control panel, and back up to
continue browsing. The permanent display of those controls is important
for the novice and occasional users. On the other hand, pop up menus
reduce the distraction of moving the cursor by making these commands
available wherever the cursor currently is. This reduces perceptual and
motor load. Pie menus are arranged with their items in easy to remember
directions. For example the BACK page turning commands are to the left
(and the NEXT page is to the right) (Figure 6). This arrangement
facilitates gestural input and encourages development of muscle memory.
Experienced users can make gestural selections from these menus so
comfortably and rapidly that it is often unnecessary to display the
menu. This is called "mouse ahead display suppression", and its point is
to reduce the perceptual distraction.
The more recent NeWS version of Hyperties on the SUN workstation uses
two large windows that partition the screen vertically. Each window can
have links and users can decide whether to put the destination article
on top of the current window or on the other window. The pie menus made
it rapid and easy to permit such a selection. When users click on a
selectable target a pie menu appears (Figure 1) and allows users to
specify in which window the destination article should be displayed
(practically users merely click then move the mouse in direction of the
desire window) . This strategy is easy to explain to visitors and
satisfying to use. An early pilot test with four subjects was run, but
the appeal of this strategy is very strong and we have not conducted
more rigorous usability tests.
In the author tool, we employ a more elaborate window strategy to manage
the 15-25 articles that an author may want to keep close at hand. We
assume that authors on the SUN/Hyperties will be quite knowledgeable in
UNIX and Emacs and therefore would be eager to have a richer set of
window management features, even if the perceptual, cognitive, and motor
load were greater. Tab windows have their title bars extending to the
right of the window, instead of at the top. When even 15 or 20 windows
are open, the tabs may still all be visible for reference or selection,
even thought the contents of the windows are obscured. This is a
convenient strategy for many authoring tasks, and it may be effective in
other applications as well.
> I have a couple of general concerns about the activity menu.
> One is conceptual. The actions are global but both the visual
> representation and the controller (mouse button) are contextual. I tried
> the RadialContext extension in firefox and this aspect really feel odd
> to me.
> The other is about interaction. We conflict with context menus so the
> user would have to be careful to hit a part of the screen where there
> are not other context menus to activate them.
>> 3. Handheld Mode
>> Finally, a fantastic side effect of the activity menu is its
>> application in handheld mode. Previously, we had stated that
>> activities would have to explicitly implement handheld controls, since
>> there would be no mouse and keyboard. However, when the activity
>> menus are designed such that the primary items are the basic key
>> controls for the activity, any activity can run in handheld mode, at
>> least in some form.
>> We could make the circle button on the right the menu button. Press
>> and hold this button to display the pie menu, and use the directional
>> pad to select the option you want. This is where the ranks for items
>> also comes into play. Since we can read 8 directions from the d-pad,
>> we can show either primary or secondary levels of the menu. Of
>> course, activities could choose to use this button for something other
>> than the pie menu if they wish - in games, for example - but this
>> would be a really nice default that would make most activities play
>> nice with handheld mode "out of the box."
> I really like this.
Absolutely! The Sugar pie menus could display graphics and icons
corresponding to the hardware buttons to reinforce the correspondence. A
great side-effect of limiting the number of pie menu items to four or
eight, is that they map so nicely to other input devices like game
controllers, joysticks, numeric keypads, arrow keys, etc.
>> I look forward to hearing what everyone thinks. Also, Marco, I'd like
>> to hear your thoughts on all of these ideas from an implementation
> I think it's very unlikely we will have a composite manager for
> generation 1. That means we will have to use shaped windows to implement
> it, which is possible theoretically, but I think no one actually tested
> the performance of this on the OLPC. Before putting a lot of design work
> on this I think it would be wise to try out a prototype on the olpc
> (especially since we have already Don implementation and we can easily
> add a shaped window to it).
I like your suggestion to factor the code so it uses Cairo but is not
directly tied to GTK, and is easy to hook into a pure Cairo based
rendering environment and whatever input tracking system it uses. That
will make it easy to port the GTK based pie menus to the official Cairo
based compositing renderer, when it's ready, while in the mean time
developing window system independent features and drawing pretty
scalable graphics in Python/Cairo.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Sugar-devel