[sugar] Pie menus for pygtk and OLPC Sugar!

[Don Hopkins]

These sound like great ideas!

The best thing to do is to code up some of these ideas and see how they 
look and feel, since it's a visceral thing that is hard to think through 
in your head, without having something concrete to really play with.

I'll comment on Eban's message and Marco's reply at the same time.

Marco Pesenti Gritti wrote:
> On Fri, 2007-02-23 at 15:39 -0500, Eben Eliason wrote:
>   
>> WHERE
>>
>> Pie menus are always tricky around screen edges, and I'm not a fan of
>> any of the design solutions that aim to fix this.  As such, I'm
>> proposing two very specific yet useful cases where these menus will be
>> used.  They are not a general purpose replacement for all menus in my
>> mind, but their benefits can be maximized in some cases.
>>     
>
> I fully agree on this, so I'm going to comment on the specific cases.
>
>   
One solution to the screen-edge problem is to move the menu so it 
completely fits on the screen, and then to warp the mouse by the same 
offset. But that has to be done very carefully and synchronously to 
avoid any problems that would interfere with mouse-ahead. You can defer 
popping up the menu until ALL events have been processed and the mouse 
is actually paused, and only then move the cursor instantly after 
popping up the menu, so the user can mouse ahead without popping up the 
menu or warping the mouse near the screen edge. I agree that there are a 
lot of undesirable things about that solution, which is quite complex, 
violates the principle of least astonishment, and is even impossible to 
implement in web browsers and other environments that don't let you warp 
the mouse (and on hardware like absolute position input devices like 
touch screens that do not have robot arms to reach out and move the 
user's finger on the screen, instead of warping the mouse.)

Another approach to the screen edge problem, which I've been using on my 
Pocket PC implementation of touch screen pie menus, is simply to not use 
the directions that are off the edge of the screen, for pie menus that 
pop up along the screen edge or in corners. You can put 1/2 pie menus 
along the edges (calzone menus?), and 1/4 pie menus in the corners 
(turnover menus?).

There is of course the disadvantage that fewer directions can fit in 
partial menus, but there is also the slight advantage that when the user 
starts the gesture on the edge or in the corner of the screen, they have 
the whole rest of the screen to move out into, to increase the angular 
precision of the selection. Edge an corner menus allow you to move a 
greater distance, for more "leverage". (This applies more in the case of 
a small finger-operated touch screen with full-screen pie menus, since 
most of the screen is very near a corner or edge!) So you might be able 
to get away with putting a few more items-per-degree on edge and corner 
menus, than in centrally triggered menus (which only allow you half a 
screen of leverage).

You can also use the distance independent of the direction, to "pull 
out" sub-items, so you could have a corner pie menu with three different 
directions (right, down, and diagonal), or an edge pie with five 
different directions (left, down/left, down, down/right, right), but 
each of those distinct directions could "pull out" many different 
sub-items, because you have the whole width or height or diagonal of the 
screen to pull out.

So one feature I'd like pie menus to support is limiting the start and 
end direction of the layout, to support fractional pie menus for the 
screen edges and corners.

A couple of complementary (but mutually incompatible) features for 
exploiting the distance of motion are "pull out" slices with sub-items 
selected by the linear distance from the menu center, and also "layered 
pies" with rings that have different numbers of items in them, so the 
inner-most ring can have four items, the next ring 8 items, the next 
ring 16, and so on.

>> 1. Contextual Menus/Rollovers
>>
>> The current laptop design includes the notion of rollovers for
>> Objects, which typically exist in two states.  A primary rollover
>> appears to provide a text title for the Object, which is otherwise
>> rendered as an icon only, and a secondary extension of this appears
>> after a short delay to provide a preview, summary, and a list of
>> actions associated with the Object.
>>
>> All contextual rollovers should be replaced with pie menus.  The menu
>> itself would appear, of course, surrounding the object so the icon is
>> still visible in the center.  The text description of the object the
>> menu relates to can appear with the icon, in the center of the menu
>> (eg. the child's name when rolling over an XO).
>>
>>     
I think it's important to trigger pie menus on a mouse click (and 
control them by the instantaneous direction between clicks, but NOT the 
path taken, in order to allow re-selection and browsing), and to center 
them on the exact position of the mouse click. The user should have a 
crisp consistent mental model of how pie menus work (which is NOT the 
case for gesture recognition). Pie menus should completely cover all 
possible "gesture space" with well defined behavior (by basing the 
selection on the angle between clicks, and not the path taken). In 
contrast, gesture recognition does NOT cover all gesture space (because 
most gestures are syntax errors, and gestures should be far apart and 
distinct in gesture space to prevent errors), and they do not allow 
in-flight re-selection, and they are not "self revealing" like pie menus.

Pie menus are more predictable, reliable, forgiving, simpler and easier 
to learn than gesture recognition, because it's impossible to make a 
syntax error, always possible to recover from a mistaken direction 
before releasing the button, and "self reveal" their directions by 
popping up a window with labels, and they "train" you to mouse ahead by 
"rehearsal". 

Roll-overs that pop up pie menus before you click the mouse could be 
kind of weird, and could violate the principle of least astonishment. 
The way some web sites implement roll-over pop up linear menus on web 
pages is pretty annoying, because the menus often cover up something 
else that you meant to select, and that is frustrating if you just meant 
to move the mouse over the menu trigger but not pop up the menu.

http://en.wikipedia.org/wiki/Principle_of_least_astonishment

So whatever design you use for roll-over feedback, it should be sure to 
re-center the pie menu on the current cursor position (or god forbid 
warp the cursor to the menu center where it's currently displayed), and 
not get in the way of reliable mouse-ahead. I can imagine a kind of 
light weight roll-over feedback that previews the pie menu and reveals 
its directions without actually popping up the menu itself or occluding 
the view of the object you're pointing to.

There is an issue (known as the "nulling problem") that I've had to deal 
with for small touch screen pie menus, because you can't see through 
your finger, and the computer can't forcefully reposition it 
(hopefully). One solution I've developed for my Pocket PC pie menus is 
for the pie menus to take over the entire screen, and to lay out the 
items around the edges of the screen (instead of centering them on the 
place where your finger touched the screen). Then it draws some 
light-weight feedback that looks like cracks in the glass centered on 
your finger touch location, that do not occlude the menu labels (the 
crack feedback is drawn underneath the labels). The cracks delimit the 
edges of the target regions, and show you where on the screen to touch 
to select the different items, while the labels are highlighted 
according to the direction selected by your finger motion, but the 
labels are always justified in the same place around the screen edge no 
matter where you touch. It also draws some animated "grippy" lines in 
the selected slice orthogonal to the slide direction, that slide in and 
out to show you how it's tracking the distance inside of the slice.

That's one solution to the nulling problem for small touch screens 
(de-couple the touch feedback from the menu label position), which might 
work with the OLPC's small screen, or as roll-over feedback.

Some interesting info on the "nulling problem" and other issues, by Bill 
Buxon: There's More to Interaction than Meets the Eye: Some Issues in 
Manual Input
http://www.billbuxton.com/eye.html

The stuff he writes about chunking and phrasing of gestural input is 
also very important for pie menus!

>> This menu could contain a list of primary actions associated with the
>> object.  For instance, for an XO you could "send message", "make
>> friend", "invite to activity", etc.  Each object type might have a
>> slightly different list of actions, but there will certainly be
>> overlap.  Where possible, we will define the arrangement of these
>> actions such that they always appear in the same place, regardless of
>> the object itself.  This is important for spatial recognition, making
>> gestural activation natural everywhere in the interface.  Slices for
>> actions that don't apply to an object will be left empty.
>>
>>     
That's a good idea! I made a visual PostScript interface that had a 
standard pie menu that you could pop up on any object, but which had a 
submenu in a standard direction that always popped up an 
object-type-specific submenu. So the top level menu was always the same 
and had common commands, and the object specific menu was always close 
by in the same direction.

Here's a paper about it that discusses the pie menu design, and has some 
pictures:

http://www.donhopkins.com/drupal/node/97

And here are some videos:

http://www.donhopkins.com/home/movies/PSIBERDemo.mov
http://www.donhopkins.com/home/movies/NeWSHacking.mov
http://www.donhopkins.com/home/movies/PseudoScientific.mov
http://www.donhopkins.com/home/movies/VisualizerDemo.mov

>> The only drawback to this over the previous rollover design is that
>> the actual preview image or summary won't appear automatically after
>> delay.  Instead, the preview would appear as the secondary panel of a
>> "preview" action slice in the menu, which would of course be
>> consistently positioned for every object.  The advantage, of course,
>> is that taking actions on objects is actually quickly exposed, whereas
>> before they were confined to a secondary rollover state.
>>     
This is definitely something that's worth implementing in a few 
different ways and playing around with to see what feels good.
> I'm not fully convinced about this one.
>
> * As we discussed it would probably not be possible to use it on the
> frame (since icons are near the edges).
> * It would make sense in the Friends view. Though we allow to reposition
> Xo's there, so again, they could be near the edges.
> * What are the advantages to use them in the mesh? The two primary use
> cases seem 1 explore and 2 join activities. Pie menus doesn't seem to
> make 1 easier (you can just click on the activities) and probably makes
> 2 a little harder (more indirect preview).
> * In the current mesh design you can easily browse through participant
> names with the little animation you sketched out in the flash demo.
> Would that be as easy with pie menus?
>
>   
Pie menus are best when the number of items is constant and the menus 
don't change and aren't based on dynamic data. That doesn't mean you 
shouldn't ever do that, just that they're best when they're predictable, 
and a real human being has thoughtfully planned them out (which is 
impossible for dynamically generated menus). The Sims uses pie menus 
whose items are dynamically generated, so they aren't able to fully 
exploit the advantages of muscle memory. The game controls the order of 
the pie menu items which are layed out in a circle clockwise starting 
from the top, but it does not have any explicit control over the 
direction of each item. Some objects have stable pie menus that are 
predictable (like the fridge and the toilet), where an item in one mode 
will be replaced by another item in another mode (the fridge lets you 
fix breakfast/lunch/dinner depending on the time, and the toilet has 
use, seat up/down, followed by optional flush at the end of the menu). 
The pie menus automatically add dummy items to force the number of items 
to be even (2, 4, 8, and more than 8 go into the linear overflow items). 
We discussed how to enable the game to control the directions of menu 
items, but there was no easy solution that didn't place a burden on the 
game designers defining the actions on the objects, so we just went with 
dynamic layout.

The Python (and OpenLaszlo and JavaScript) pie menus let you specify the 
layout of each item with parameters like linear (true for linear, false 
for pie), "max_pie_items" to limit the maximum number of pie items 
before making items linear, linear_direction (a string specifying the 
north/up/south/down/east/left/west/right/nw/ne/sw/se linear layout 
direction for the item) (menus can have mixed pie menus and linear items 
in different directions), "linear_order_reversed" to reverse the linear 
layout order (to flip "up" layout so upside-down pull-up menus come out 
in top to bottom order).

There is some rudimentary stuff in there for paging pie menus, to 
support arbitrarily large numbers of items in pages of 8 (or so) at a 
time, but it's not hooked up to the user interface or well tested. The 
idea is that there should be some kind of paging user interface (page 
up/page down keys, as well as arrow buttons somewhere on the screen, or 
an inner ring of four pie menu paging items: first/last/next/previous). 
I've tried scrolling menus that let you wind the cursor around the 
center to "spiral" through a long list of items (demonstrated on Brad 
Meyer's "All the Widgets" video he made for ACM SIGCHI), but that turned 
out to be kind of weird... (But maybe it's good for scrolling through 
DNA sequences!)

This clip from All the Widgets demonstrates the scrolling spiral pie 
menu for selecting country flags, a color wheel pie for changing the 
text and background color in Emacs, pull-out font family/style/size pie 
menus, and a "precision pie menu" that lets you precisely select any 
number from 0 to 360 (by "poking" out through the menu bubble with a 
flexible rubber band), and also a pie menu window manager supporting 
mouse-ahead.

http://www.donhopkins.com/home/movies/JustThePieMenus.mov

>> 2. Activity Menu
>>
>> Until now, we didn't really have a menu system per se for Activities
>> at all.  Instead, we have a toolbar, where buttons/controls in the
>> toolbar can have rollover states which provide panels with additional
>> controls contextual to the button itself.  There is a file menu of
>> sorts in the frame, providing basic features like keep (save), share,
>> etc, but nothing that the activity developer could extend in any way.
>>
>> I propose that we allow each activity to define its own activity pie
>> menu.  This menu could be invoked by right clicking anywhere on the
>> screen where there isn't an object with its own contextual menu.  It
>> could contain any number of items, ranked by the three levels
>> mentioned above.  There could perhaps be a method of selecting which
>> level of detail you want to see through the interface, but I have
>> other ideas for this as well.
>>     
That would be cool! Note that the Python pie menus also support linear 
layout as well as hybrid pie/linear layout, and the application 
developer can control which items go into the pie and linear parts, and 
what directions they're laid out in.

Of course it would be nice to have even more control of the pie layout, 
so you can tag pie items with preferred directions and priorities, and a 
smart layout algorithm could resolve the items into appropriate 
directions, keep the number of items even by inserting inactive items in 
unused slots, and apply hints like preferred direction or axis, keeping 
opposite pairs in opposite directions, etc.

For example, you could add "on" and "off" as a pair, preferring up for 
on and down for off with a high priority, then add "next" and "previous" 
as a pair with no preference for direction, and four other items with no 
pairing or directional preference but a lower priority, so the "next" 
and "previous" would be placed left and right because they were more 
important than the unpaired items, which would go along the diagonals.

If you limited the number of pie items to 8, and added some even lower 
priority items, they would be relegated to linear items, or to an outer 
ring.

There should be some way to set up the maximum number of pie items in 
each ring of a multi-layered pie menu, like the number off electrons 
allowed in each ring of an atom, and a limit on the maximum number of 
rings, and also a way to control the linear layout of items that don't 
fit in the rings.

> Sort of OT but I really dislike the idea of selecting the level of the
> detail of the interface. It's just a question without a good answer. If
> we decide to go this way we should at least analyze the previous
> attempts to this kind of interface and figure out why they failed.
>
>   
Agreed!
>> This activity menu can function in two powerful ways.  The first is
>> that it provides a place for activities to define additional
>> functionality.  Until now, with no menu system, everything had to be
>> exposed on screen.  This is a goal we still want to shoot for, and it
>> should be known to developers that their activities should function
>> completely without need for the activity menu at all.  However, when
>> additional advanced controls or options would benefit experienced
>> users, this menu could provide an outlet for some of them without
>> cluttering the toolbar itself.  By design, I would recommend that such
>> advanced features be ranked as tertiary items.
>>     
Agreed!
> This make sense to me but we have to define clearly the use cases. I
> don't want activities to come with dumb interfaces and then put all sort
> of "advanced" features in the pie to work around the design
> deficiencies.
>   
Agreed!

In the ideal world, all application designers would think carefully 
about the common use cases, make the interface obvious, self revealing, 
and usable without menus, and design nice static or smart dynamic pie 
menus that were easy for novices to learn yet work well as accelerators 
for expert users (like the X11 SimCity pie menus that are shortcuts for 
the tool palette).

Here's a video of the pie menus in the X11 version of SimCity, which 
shows how they support mouse-ahead, and work as an optional accelerator 
for the tool palette, which can be tucked away to save screen space once 
you're used to the pie menus. Notice the funky "totem pole" layout of 
the tool palette icons, which are different shapes and sizes according 
to how they're used and how much they cost. And how they tool palette is 
arranged in the same pattern as the pie menus, serving as a map to help 
you learn the pie menus.

http://www.donhopkins.com/home/movies/X11SimCityDemo.mov
http://www.donhopkins.com/home/movies/SimCityNetDemo.mov

In another other ideal world (which is ideally the same ideal world as 
the first ideal world, but it's too idealistic to expect that), the user 
should be able to edit the user interface to customize it for their own 
particular usage patterns (like Maya's marking menus that the 3D artist 
can tailor to their workflow).

In a game like World of Warcraft (a real-world virtual universe, but 
less-than-ideal world), each user has a very particular set of commands 
they want to use at any particular time (because characters play 
different roles and have different abilities, spells and items, which 
changes over time and with different activities), so it's absolutely 
necessary to support user-defined interfaces constructed by dragging and 
dropping command icons around to create your own custom toolbars and pie 
menus. (I've been thinking about how design user-editable pie menus for 
World of Warcraft, but I've been too busy trying to get my character to 
level 70 to actually implement WOW pie menus yet. But it would be easy, 
given a good design, since WOW's user interface is fully programmable in 
Lua, and already supports dragging and dropping action icons into user 
defined palettes).

>> The second and actually more powerful way to use these menus in my
>> mind is, rather than for advanced functionality, for limited basic
>> control of the interface.  As an example, consider the web browser.
>> The basic set of controls might be "forward" (right), "back" (left),
>> "reload" (up), "view source" (down).  By defining these as the
>> primaries, I can instantly invoke these frequently used commands with
>> gestural ease, without even needing to mess about with toolbar
>> buttons.  This kind of gestural activation of frequent actions could
>> be fantastic.  Similarly, a drawing activity might define "circle",
>> "square", "line", "brush" as it's basic set of tools, making selection
>> of the needed tool a breeze without needing to leave the region of the
>> drawing being worked on.
>>     
>
> I really like this use case.
>   
Me too! We used pie menus for paging and navigation in HyperTIES 
hypermedia browser, and the Emacs authoring tool, which I developed for 
Ben Shneiderman at the University of Maryland Human Computer Interaction 
Lab:

http://www.donhopkins.com/home/movies/HyperTIESDemo.mov

Designing to Facilitate Browsing: A Look Back at the Hyperties 
Workstation Browser
http://www.donhopkins.com/drupal/node/102

*3. PIE MENUS TO PERMIT LOW COGNITIVE LOAD ACTION*

The Hyperties browser uses pie menus as accelerators, to make commonly 
used commands quickly and easily available. A pie menu is a type of pop 
up menu whose selections are laid out in a circle around the menu center 
(18). The menu pops up centered on the cursor, so that each selection is 
adjacent to the cursor but in a different direction (Figure 1 and 6). A 
selection is made by moving the cursor in the direction of the desired 
selection, and clicking. Experienced pie menu users can make selections 
from familiar menus quickly and reliably without even having to look at 
the menu, because the menu selection depends on the direction between 
the two mouse clicks that invoke and select from the menu. The distance 
of cursor motion does not effect the selection, but the further away 
from the center the cursor is, the more precise the control of the 
selection is.

The browser has a control panel at the bottom of the screen, with 
buttons showing the names of available commands, to turn the page, 
return to the previous article, show the index, etc. When users are 
browsing a document by pointing and clicking on highlighted text links 
in the main contents window, they move the cursor down to the bottom of 
the screen to press buttons in the control panel, and back up to 
continue browsing. The permanent display of those controls is important 
for the novice and occasional users. On the other hand, pop up menus 
reduce the distraction of moving the cursor by making these commands 
available wherever the cursor currently is. This reduces perceptual and 
motor load. Pie menus are arranged with their items in easy to remember 
directions. For example the BACK page turning commands are to the left 
(and the NEXT page is to the right) (Figure 6). This arrangement 
facilitates gestural input and encourages development of muscle memory. 
Experienced users can make gestural selections from these menus so 
comfortably and rapidly that it is often unnecessary to display the 
menu. This is called "mouse ahead display suppression", and its point is 
to reduce the perceptual distraction.

[...]

The more recent NeWS version of Hyperties on the SUN workstation uses 
two large windows that partition the screen vertically. Each window can 
have links and users can decide whether to put the destination article 
on top of the current window or on the other window. The pie menus made 
it rapid and easy to permit such a selection. When users click on a 
selectable target a pie menu appears (Figure 1) and allows users to 
specify in which window the destination article should be displayed 
(practically users merely click then move the mouse in direction of the 
desire window) . This strategy is easy to explain to visitors and 
satisfying to use. An early pilot test with four subjects was run, but 
the appeal of this strategy is very strong and we have not conducted 
more rigorous usability tests.

In the author tool, we employ a more elaborate window strategy to manage 
the 15-25 articles that an author may want to keep close at hand. We 
assume that authors on the SUN/Hyperties will be quite knowledgeable in 
UNIX and Emacs and therefore would be eager to have a richer set of 
window management features, even if the perceptual, cognitive, and motor 
load were greater. Tab windows have their title bars extending to the 
right of the window, instead of at the top. When even 15 or 20 windows 
are open, the tabs may still all be visible for reference or selection, 
even thought the contents of the windows are obscured. This is a 
convenient strategy for many authoring tasks, and it may be effective in 
other applications as well.

> I have a couple of general concerns about the activity menu.
>
> One is conceptual. The actions are global but both the visual
> representation and the controller (mouse button) are contextual. I tried
> the RadialContext extension in firefox and this aspect really feel odd
> to me.
>
> The other is about interaction. We conflict with context menus so the
> user would have to be careful to hit a part of the screen where there
> are not other context menus to activate them.
>
>   
>> 3. Handheld Mode
>>
>> Finally, a fantastic side effect of the activity menu is its
>> application in handheld mode.  Previously, we had stated that
>> activities would have to explicitly implement handheld controls, since
>> there would be no mouse and keyboard.  However, when the activity
>> menus are designed such that the primary items are the basic key
>> controls for the activity, any activity can run in handheld mode, at
>> least in some form.
>>
>> We could make the circle button on the right the menu button.  Press
>> and hold this button to display the pie menu, and use the directional
>> pad to select the option you want.  This is where the ranks for items
>> also comes into play.  Since we can read 8 directions from the d-pad,
>> we can show either primary or secondary levels of the menu.  Of
>> course, activities could choose to use this button for something other
>> than the pie menu if they wish - in games, for example - but this
>> would be a really nice default that would make most activities play
>> nice with handheld mode "out of the box."
>>     
>
> I really like this.
>
>   
Absolutely! The Sugar pie menus could display graphics and icons 
corresponding to the hardware buttons to reinforce the correspondence. A 
great side-effect of limiting the number of pie menu items to four or 
eight, is that they map so nicely to other input devices like game 
controllers, joysticks, numeric keypads, arrow keys, etc.
>> I look forward to hearing what everyone thinks.  Also, Marco, I'd like
>> to hear your thoughts on all of these ideas from an implementation
>> standpoint.
>>     
>
> I think it's very unlikely we will have a composite manager for
> generation 1. That means we will have to use shaped windows to implement
> it, which is possible theoretically, but I think no one actually tested
> the performance of this on the OLPC. Before putting a lot of design work
> on this I think it would be wise to try out a prototype on the olpc
> (especially since we have already Don implementation and we can easily
> add a shaped window to it).
>
> Marco
>   

I like your suggestion to factor the code so it uses Cairo but is not 
directly tied to GTK, and is easy to hook into a pure Cairo based 
rendering environment and whatever input tracking system it uses. That 
will make it easy to port the GTK based pie menus to the official Cairo 
based compositing renderer, when it's ready, while in the mean time 
developing window system independent features and drawing pretty 
scalable graphics in Python/Cairo.

    -Don

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.laptop.org/pipermail/sugar/attachments/20070224/b2e1a596/attachment-0001.html