<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#ffffff" text="#000000"> These sound like great ideas! The best thing to do is to code up some of these ideas and see how they look and feel, since it's a visceral thing that is hard to think through in your head, without having something concrete to really play with. I'll comment on Eban's message and Marco's reply at the same time. Marco Pesenti Gritti wrote: <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap="">On Fri, 2007-02-23 at 15:39 -0500, Eben Eliason wrote: </pre> <blockquote type="cite"> <pre wrap="">WHERE Pie menus are always tricky around screen edges, and I'm not a fan of any of the design solutions that aim to fix this. As such, I'm proposing two very specific yet useful cases where these menus will be used. They are not a general purpose replacement for all menus in my mind, but their benefits can be maximized in some cases. </pre> </blockquote> <pre wrap=""> I fully agree on this, so I'm going to comment on the specific cases. </pre> </blockquote> One solution to the screen-edge problem is to move the menu so it completely fits on the screen, and then to warp the mouse by the same offset. But that has to be done very carefully and synchronously to avoid any problems that would interfere with mouse-ahead. You can defer popping up the menu until ALL events have been processed and the mouse is actually paused, and only then move the cursor instantly after popping up the menu, so the user can mouse ahead without popping up the menu or warping the mouse near the screen edge. I agree that there are a lot of undesirable things about that solution, which is quite complex, violates the principle of least astonishment, and is even impossible to implement in web browsers and other environments that don't let you warp the mouse (and on hardware like absolute position input devices like touch screens that do not have robot arms to reach out and move the user's finger on the screen, instead of warping the mouse.) Another approach to the screen edge problem, which I've been using on my Pocket PC implementation of touch screen pie menus, is simply to not use the directions that are off the edge of the screen, for pie menus that pop up along the screen edge or in corners. You can put 1/2 pie menus along the edges (calzone menus?), and 1/4 pie menus in the corners (turnover menus?). There is of course the disadvantage that fewer directions can fit in partial menus, but there is also the slight advantage that when the user starts the gesture on the edge or in the corner of the screen, they have the whole rest of the screen to move out into, to increase the angular precision of the selection. Edge an corner menus allow you to move a greater distance, for more "leverage". (This applies more in the case of a small finger-operated touch screen with full-screen pie menus, since most of the screen is very near a corner or edge!) So you might be able to get away with putting a few more items-per-degree on edge and corner menus, than in centrally triggered menus (which only allow you half a screen of leverage). You can also use the distance independent of the direction, to "pull out" sub-items, so you could have a corner pie menu with three different directions (right, down, and diagonal), or an edge pie with five different directions (left, down/left, down, down/right, right), but each of those distinct directions could "pull out" many different sub-items, because you have the whole width or height or diagonal of the screen to pull out. So one feature I'd like pie menus to support is limiting the start and end direction of the layout, to support fractional pie menus for the screen edges and corners. A couple of complementary (but mutually incompatible) features for exploiting the distance of motion are "pull out" slices with sub-items selected by the linear distance from the menu center, and also "layered pies" with rings that have different numbers of items in them, so the inner-most ring can have four items, the next ring 8 items, the next ring 16, and so on. <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap=""></pre> <blockquote type="cite"> <pre wrap="">1. Contextual Menus/Rollovers The current laptop design includes the notion of rollovers for Objects, which typically exist in two states. A primary rollover appears to provide a text title for the Object, which is otherwise rendered as an icon only, and a secondary extension of this appears after a short delay to provide a preview, summary, and a list of actions associated with the Object. All contextual rollovers should be replaced with pie menus. The menu itself would appear, of course, surrounding the object so the icon is still visible in the center. The text description of the object the menu relates to can appear with the icon, in the center of the menu (eg. the child's name when rolling over an XO). </pre> </blockquote> </blockquote> I think it's important to trigger pie menus on a mouse click (and control them by the instantaneous direction between clicks, but NOT the path taken, in order to allow re-selection and browsing), and to center them on the exact position of the mouse click. The user should have a crisp consistent mental model of how pie menus work (which is NOT the case for gesture recognition). Pie menus should completely cover all possible "gesture space" with well defined behavior (by basing the selection on the angle between clicks, and not the path taken). In contrast, gesture recognition does NOT cover all gesture space (because most gestures are syntax errors, and gestures should be far apart and distinct in gesture space to prevent errors), and they do not allow in-flight re-selection, and they are not "self revealing" like pie menus. Pie menus are more predictable, reliable, forgiving, simpler and easier to learn than gesture recognition, because it's impossible to make a syntax error, always possible to recover from a mistaken direction before releasing the button, and "self reveal" their directions by popping up a window with labels, and they "train" you to mouse ahead by "rehearsal".  Roll-overs that pop up pie menus before you click the mouse could be kind of weird, and could violate the principle of least astonishment. The way some web sites implement roll-over pop up linear menus on web pages is pretty annoying, because the menus often cover up something else that you meant to select, and that is frustrating if you just meant to move the mouse over the menu trigger but not pop up the menu. <a class="moz-txt-link-freetext" href="http://en.wikipedia.org/wiki/Principle_of_least_astonishment">http://en.wikipedia.org/wiki/Principle_of_least_astonishment</a> So whatever design you use for roll-over feedback, it should be sure to re-center the pie menu on the current cursor position (or god forbid warp the cursor to the menu center where it's currently displayed), and not get in the way of reliable mouse-ahead. I can imagine a kind of light weight roll-over feedback that previews the pie menu and reveals its directions without actually popping up the menu itself or occluding the view of the object you're pointing to. There is an issue (known as the "nulling problem") that I've had to deal with for small touch screen pie menus, because you can't see through your finger, and the computer can't forcefully reposition it (hopefully). One solution I've developed for my Pocket PC pie menus is for the pie menus to take over the entire screen, and to lay out the items around the edges of the screen (instead of centering them on the place where your finger touched the screen). Then it draws some light-weight feedback that looks like cracks in the glass centered on your finger touch location, that do not occlude the menu labels (the crack feedback is drawn underneath the labels). The cracks delimit the edges of the target regions, and show you where on the screen to touch to select the different items, while the labels are highlighted according to the direction selected by your finger motion, but the labels are always justified in the same place around the screen edge no matter where you touch. It also draws some animated "grippy" lines in the selected slice orthogonal to the slide direction, that slide in and out to show you how it's tracking the distance inside of the slice. That's one solution to the nulling problem for small touch screens (de-couple the touch feedback from the menu label position), which might work with the OLPC's small screen, or as roll-over feedback. Some interesting info on the "nulling problem" and other issues, by Bill Buxon: There's More to Interaction than Meets the Eye: Some Issues in Manual Input <a class="moz-txt-link-freetext" href="http://www.billbuxton.com/eye.html">http://www.billbuxton.com/eye.html</a> The stuff he writes about chunking and phrasing of gestural input is also very important for pie menus! <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <blockquote type="cite"> <pre wrap="">This menu could contain a list of primary actions associated with the object. For instance, for an XO you could "send message", "make friend", "invite to activity", etc. Each object type might have a slightly different list of actions, but there will certainly be overlap. Where possible, we will define the arrangement of these actions such that they always appear in the same place, regardless of the object itself. This is important for spatial recognition, making gestural activation natural everywhere in the interface. Slices for actions that don't apply to an object will be left empty. </pre> </blockquote> </blockquote> That's a good idea! I made a visual PostScript interface that had a standard pie menu that you could pop up on any object, but which had a submenu in a standard direction that always popped up an object-type-specific submenu. So the top level menu was always the same and had common commands, and the object specific menu was always close by in the same direction. Here's a paper about it that discusses the pie menu design, and has some pictures: <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/drupal/node/97">http://www.donhopkins.com/drupal/node/97</a> And here are some videos: <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/home/movies/PSIBERDemo.mov">http://www.donhopkins.com/home/movies/PSIBERDemo.mov</a> <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/home/movies/NeWSHacking.mov">http://www.donhopkins.com/home/movies/NeWSHacking.mov</a> <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/home/movies/PseudoScientific.mov">http://www.donhopkins.com/home/movies/PseudoScientific.mov</a> <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/home/movies/VisualizerDemo.mov">http://www.donhopkins.com/home/movies/VisualizerDemo.mov</a> <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <blockquote type="cite"> <pre wrap="">The only drawback to this over the previous rollover design is that the actual preview image or summary won't appear automatically after delay. Instead, the preview would appear as the secondary panel of a "preview" action slice in the menu, which would of course be consistently positioned for every object. The advantage, of course, is that taking actions on objects is actually quickly exposed, whereas before they were confined to a secondary rollover state. </pre> </blockquote> </blockquote> This is definitely something that's worth implementing in a few different ways and playing around with to see what feels good. <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap="">I'm not fully convinced about this one. * As we discussed it would probably not be possible to use it on the frame (since icons are near the edges). * It would make sense in the Friends view. Though we allow to reposition Xo's there, so again, they could be near the edges. * What are the advantages to use them in the mesh? The two primary use cases seem 1 explore and 2 join activities. Pie menus doesn't seem to make 1 easier (you can just click on the activities) and probably makes 2 a little harder (more indirect preview). * In the current mesh design you can easily browse through participant names with the little animation you sketched out in the flash demo. Would that be as easy with pie menus? </pre> </blockquote> Pie menus are best when the number of items is constant and the menus don't change and aren't based on dynamic data. That doesn't mean you shouldn't ever do that, just that they're best when they're predictable, and a real human being has thoughtfully planned them out (which is impossible for dynamically generated menus). The Sims uses pie menus whose items are dynamically generated, so they aren't able to fully exploit the advantages of muscle memory. The game controls the order of the pie menu items which are layed out in a circle clockwise starting from the top, but it does not have any explicit control over the direction of each item. Some objects have stable pie menus that are predictable (like the fridge and the toilet), where an item in one mode will be replaced by another item in another mode (the fridge lets you fix breakfast/lunch/dinner depending on the time, and the toilet has use, seat up/down, followed by optional flush at the end of the menu). The pie menus automatically add dummy items to force the number of items to be even (2, 4, 8, and more than 8 go into the linear overflow items). We discussed how to enable the game to control the directions of menu items, but there was no easy solution that didn't place a burden on the game designers defining the actions on the objects, so we just went with dynamic layout. The Python (and OpenLaszlo and JavaScript) pie menus let you specify the layout of each item with parameters like linear (true for linear, false for pie), "max_pie_items" to limit the maximum number of pie items before making items linear, linear_direction (a string specifying the north/up/south/down/east/left/west/right/nw/ne/sw/se linear layout direction for the item) (menus can have mixed pie menus and linear items in different directions), "linear_order_reversed" to reverse the linear layout order (to flip "up" layout so upside-down pull-up menus come out in top to bottom order). There is some rudimentary stuff in there for paging pie menus, to support arbitrarily large numbers of items in pages of 8 (or so) at a time, but it's not hooked up to the user interface or well tested. The idea is that there should be some kind of paging user interface (page up/page down keys, as well as arrow buttons somewhere on the screen, or an inner ring of four pie menu paging items: first/last/next/previous). I've tried scrolling menus that let you wind the cursor around the center to "spiral" through a long list of items (demonstrated on Brad Meyer's "All the Widgets" video he made for ACM SIGCHI), but that turned out to be kind of weird... (But maybe it's good for scrolling through DNA sequences!) This clip from All the Widgets demonstrates the scrolling spiral pie menu for selecting country flags, a color wheel pie for changing the text and background color in Emacs, pull-out font family/style/size pie menus, and a "precision pie menu" that lets you precisely select any number from 0 to 360 (by "poking" out through the menu bubble with a flexible rubber band), and also a pie menu window manager supporting mouse-ahead. <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/home/movies/JustThePieMenus.mov">http://www.donhopkins.com/home/movies/JustThePieMenus.mov</a> <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap=""></pre> <blockquote type="cite"> <pre wrap="">2. Activity Menu Until now, we didn't really have a menu system per se for Activities at all. Instead, we have a toolbar, where buttons/controls in the toolbar can have rollover states which provide panels with additional controls contextual to the button itself. There is a file menu of sorts in the frame, providing basic features like keep (save), share, etc, but nothing that the activity developer could extend in any way. I propose that we allow each activity to define its own activity pie menu. This menu could be invoked by right clicking anywhere on the screen where there isn't an object with its own contextual menu. It could contain any number of items, ranked by the three levels mentioned above. There could perhaps be a method of selecting which level of detail you want to see through the interface, but I have other ideas for this as well. </pre> </blockquote> </blockquote> That would be cool! Note that the Python pie menus also support linear layout as well as hybrid pie/linear layout, and the application developer can control which items go into the pie and linear parts, and what directions they're laid out in. Of course it would be nice to have even more control of the pie layout, so you can tag pie items with preferred directions and priorities, and a smart layout algorithm could resolve the items into appropriate directions, keep the number of items even by inserting inactive items in unused slots, and apply hints like preferred direction or axis, keeping opposite pairs in opposite directions, etc. For example, you could add "on" and "off" as a pair, preferring up for on and down for off with a high priority, then add "next" and "previous" as a pair with no preference for direction, and four other items with no pairing or directional preference but a lower priority, so the "next" and "previous" would be placed left and right because they were more important than the unpaired items, which would go along the diagonals. If you limited the number of pie items to 8, and added some even lower priority items, they would be relegated to linear items, or to an outer ring. There should be some way to set up the maximum number of pie items in each ring of a multi-layered pie menu, like the number off electrons allowed in each ring of an atom, and a limit on the maximum number of rings, and also a way to control the linear layout of items that don't fit in the rings. <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap="">Sort of OT but I really dislike the idea of selecting the level of the detail of the interface. It's just a question without a good answer. If we decide to go this way we should at least analyze the previous attempts to this kind of interface and figure out why they failed. </pre> </blockquote> Agreed! <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap=""></pre> <blockquote type="cite"> <pre wrap="">This activity menu can function in two powerful ways. The first is that it provides a place for activities to define additional functionality. Until now, with no menu system, everything had to be exposed on screen. This is a goal we still want to shoot for, and it should be known to developers that their activities should function completely without need for the activity menu at all. However, when additional advanced controls or options would benefit experienced users, this menu could provide an outlet for some of them without cluttering the toolbar itself. By design, I would recommend that such advanced features be ranked as tertiary items. </pre> </blockquote> </blockquote> Agreed! <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap="">This make sense to me but we have to define clearly the use cases. I don't want activities to come with dumb interfaces and then put all sort of "advanced" features in the pie to work around the design deficiencies. </pre> </blockquote> Agreed! In the ideal world, all application designers would think carefully about the common use cases, make the interface obvious, self revealing, and usable without menus, and design nice static or smart dynamic pie menus that were easy for novices to learn yet work well as accelerators for expert users (like the X11 SimCity pie menus that are shortcuts for the tool palette). Here's a video of the pie menus in the X11 version of SimCity, which shows how they support mouse-ahead, and work as an optional accelerator for the tool palette, which can be tucked away to save screen space once you're used to the pie menus. Notice the funky "totem pole" layout of the tool palette icons, which are different shapes and sizes according to how they're used and how much they cost. And how they tool palette is arranged in the same pattern as the pie menus, serving as a map to help you learn the pie menus. <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/home/movies/X11SimCityDemo.mov">http://www.donhopkins.com/home/movies/X11SimCityDemo.mov</a> <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/home/movies/SimCityNetDemo.mov">http://www.donhopkins.com/home/movies/SimCityNetDemo.mov</a> In another other ideal world (which is ideally the same ideal world as the first ideal world, but it's too idealistic to expect that), the user should be able to edit the user interface to customize it for their own particular usage patterns (like Maya's marking menus that the 3D artist can tailor to their workflow). In a game like World of Warcraft (a real-world virtual universe, but less-than-ideal world), each user has a very particular set of commands they want to use at any particular time (because characters play different roles and have different abilities, spells and items, which changes over time and with different activities), so it's absolutely necessary to support user-defined interfaces constructed by dragging and dropping command icons around to create your own custom toolbars and pie menus. (I've been thinking about how design user-editable pie menus for World of Warcraft, but I've been too busy trying to get my character to level 70 to actually implement WOW pie menus yet. But it would be easy, given a good design, since WOW's user interface is fully programmable in Lua, and already supports dragging and dropping action icons into user defined palettes). <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <blockquote type="cite"> <pre wrap="">The second and actually more powerful way to use these menus in my mind is, rather than for advanced functionality, for limited basic control of the interface. As an example, consider the web browser. The basic set of controls might be "forward" (right), "back" (left), "reload" (up), "view source" (down). By defining these as the primaries, I can instantly invoke these frequently used commands with gestural ease, without even needing to mess about with toolbar buttons. This kind of gestural activation of frequent actions could be fantastic. Similarly, a drawing activity might define "circle", "square", "line", "brush" as it's basic set of tools, making selection of the needed tool a breeze without needing to leave the region of the drawing being worked on. </pre> </blockquote> <pre wrap=""> I really like this use case. </pre> </blockquote> Me too! We used pie menus for paging and navigation in HyperTIES hypermedia browser, and the Emacs authoring tool, which I developed for Ben Shneiderman at the University of Maryland Human Computer Interaction Lab: <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/home/movies/HyperTIESDemo.mov">http://www.donhopkins.com/home/movies/HyperTIESDemo.mov</a> Designing to Facilitate Browsing: A Look Back at the Hyperties Workstation Browser <a class="moz-txt-link-freetext" href="http://www.donhopkins.com/drupal/node/102">http://www.donhopkins.com/drupal/node/102</a> 3. PIE MENUS TO PERMIT LOW COGNITIVE LOAD ACTION The Hyperties browser uses pie menus as accelerators, to make commonly used commands quickly and easily available. A pie menu is a type of pop up menu whose selections are laid out in a circle around the menu center (18). The menu pops up centered on the cursor, so that each selection is adjacent to the cursor but in a different direction (Figure 1 and 6). A selection is made by moving the cursor in the direction of the desired selection, and clicking. Experienced pie menu users can make selections from familiar menus quickly and reliably without even having to look at the menu, because the menu selection depends on the direction between the two mouse clicks that invoke and select from the menu. The distance of cursor motion does not effect the selection, but the further away from the center the cursor is, the more precise the control of the selection is. The browser has a control panel at the bottom of the screen, with buttons showing the names of available commands, to turn the page, return to the previous article, show the index, etc. When users are browsing a document by pointing and clicking on highlighted text links in the main contents window, they move the cursor down to the bottom of the screen to press buttons in the control panel, and back up to continue browsing. The permanent display of those controls is important for the novice and occasional users. On the other hand, pop up menus reduce the distraction of moving the cursor by making these commands available wherever the cursor currently is. This reduces perceptual and motor load. Pie menus are arranged with their items in easy to remember directions. For example the BACK page turning commands are to the left (and the NEXT page is to the right) (Figure 6). This arrangement facilitates gestural input and encourages development of muscle memory. Experienced users can make gestural selections from these menus so comfortably and rapidly that it is often unnecessary to display the menu. This is called "mouse ahead display suppression", and its point is to reduce the perceptual distraction. [...] The more recent NeWS version of Hyperties on the SUN workstation uses two large windows that partition the screen vertically. Each window can have links and users can decide whether to put the destination article on top of the current window or on the other window. The pie menus made it rapid and easy to permit such a selection. When users click on a selectable target a pie menu appears (Figure 1) and allows users to specify in which window the destination article should be displayed (practically users merely click then move the mouse in direction of the desire window) . This strategy is easy to explain to visitors and satisfying to use. An early pilot test with four subjects was run, but the appeal of this strategy is very strong and we have not conducted more rigorous usability tests. In the author tool, we employ a more elaborate window strategy to manage the 15-25 articles that an author may want to keep close at hand. We assume that authors on the SUN/Hyperties will be quite knowledgeable in UNIX and Emacs and therefore would be eager to have a richer set of window management features, even if the perceptual, cognitive, and motor load were greater. Tab windows have their title bars extending to the right of the window, instead of at the top. When even 15 or 20 windows are open, the tabs may still all be visible for reference or selection, even thought the contents of the windows are obscured. This is a convenient strategy for many authoring tasks, and it may be effective in other applications as well. <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap=""> I have a couple of general concerns about the activity menu. One is conceptual. The actions are global but both the visual representation and the controller (mouse button) are contextual. I tried the RadialContext extension in firefox and this aspect really feel odd to me. The other is about interaction. We conflict with context menus so the user would have to be careful to hit a part of the screen where there are not other context menus to activate them. </pre> <blockquote type="cite"> <pre wrap="">3. Handheld Mode Finally, a fantastic side effect of the activity menu is its application in handheld mode. Previously, we had stated that activities would have to explicitly implement handheld controls, since there would be no mouse and keyboard. However, when the activity menus are designed such that the primary items are the basic key controls for the activity, any activity can run in handheld mode, at least in some form. We could make the circle button on the right the menu button. Press and hold this button to display the pie menu, and use the directional pad to select the option you want. This is where the ranks for items also comes into play. Since we can read 8 directions from the d-pad, we can show either primary or secondary levels of the menu. Of course, activities could choose to use this button for something other than the pie menu if they wish - in games, for example - but this would be a really nice default that would make most activities play nice with handheld mode "out of the box." </pre> </blockquote> <pre wrap=""> I really like this. </pre> </blockquote> Absolutely! The Sugar pie menus could display graphics and icons corresponding to the hardware buttons to reinforce the correspondence. A great side-effect of limiting the number of pie menu items to four or eight, is that they map so nicely to other input devices like game controllers, joysticks, numeric keypads, arrow keys, etc. <blockquote cite="mid1172310979.2948.28.camel@localhost.localdomain" type="cite"> <pre wrap=""></pre> <blockquote type="cite"> <pre wrap="">I look forward to hearing what everyone thinks. Also, Marco, I'd like to hear your thoughts on all of these ideas from an implementation standpoint. </pre> </blockquote> <pre wrap=""> I think it's very unlikely we will have a composite manager for generation 1. That means we will have to use shaped windows to implement it, which is possible theoretically, but I think no one actually tested the performance of this on the OLPC. Before putting a lot of design work on this I think it would be wise to try out a prototype on the olpc (especially since we have already Don implementation and we can easily add a shaped window to it). Marco </pre> </blockquote> I like your suggestion to factor the code so it uses Cairo but is not directly tied to GTK, and is easy to hook into a pure Cairo based rendering environment and whatever input tracking system it uses. That will make it easy to port the GTK based pie menus to the official Cairo based compositing renderer, when it's ready, while in the mean time developing window system independent features and drawing pretty scalable graphics in Python/Cairo.     -Don </body> </html>