[sugar] eBook Reader user interface

[Marco Pesenti Gritti]

On Fri, 2007-05-04 at 23:08 -0700, Don Hopkins wrote:
> Goal for improving the eBook reader user interface:
> 
>   I've been doing some exploratory programming with GTK and Sugar,
>   trying to improve the user interface of the eBook reader, and make
>   it useable in book mode with the gamepad.
> 
>   + Support the game keypads in eBook mode. 
> 
>     + Low level game keypad support
> 
>       Need to remap low level keyboard scan codes to Linux keyboard codes. 
> 
>       The setolpckeys.c program remaps the keys and gamepad buttons.
> 
>         Currently it maps both gamepads to the numeric keypad keys (KEY_KP8, etc),
> 	which the X server and GDK translates to directional keys (GDK_Up, etc).
> 
> 	I tried to map them to buttons (BTN_A, etc), but the X server seems 
> 	to ignore keycodes in that range. 
> 
> 	The xorg.conf file has a keycode mask that looked like it might help, 
> 	but I couldn't get it to work. 
> 
> 	Need to have unique keycodes reported for each of the two gamepads, 
> 	which are not the same as any keyboard keys, without any predefined meanings
> 	like arrow keys have. 
> 
> 	Need to define special purpose keycodes just for the OLPC gamepad,
> 	instead of trying to reuse existing but not appropriate keycodes. 
> 
> 	What is the process for defining new keycodes in <linux/input.h>?
> 
> 	Here's my strawman proposal for some new keycodes. 
> 
> 	  Use keys ("KEY_*") instead of buttons ("BTN_*"), since they
> 	  seem to work better.
> 
> 	  The 0x1b* range seems to be unused in <linux/input.h>, 
> 	  and it's between other groups of keycodes, so I'll
> 	  propose using that range for the OLPC. 
> 
> 	  The UP/DOWN/LEFT/RIGHT keys correspond to the directional
> 	  keypad.
> 
> 	  #define KEY_XO_GAMEPAD_UP      0x1b0
> 	  #define KEY_XO_GAMEPAD_DOWN    0x1b1
> 	  #define KEY_XO_GAMEPAD_LEFT    0x1b2
> 	  #define KEY_XO_GAMEPAD_RIGHT   0x1b3
> 
> 	  The NORTH/SOUTH/EAST/WEST keys correspond to the other
> 	  buttons. Those names are agnostic to the button labels,
> 	  which may change from the current Playstation buttons
> 	  (X/O/Triangle/Square). Can anyone suggest better names for
> 	  the four buttons on the right?
> 
> 	  #define KEY_XO_GAMEPAD_NORTH   0x1b4
> 	  #define KEY_XO_GAMEPAD_SOUTH   0x1b5
> 	  #define KEY_XO_GAMEPAD_EAST    0x1b6
> 	  #define KEY_XO_GAMEPAD_WEST    0x1b7
> 
> 	  While we're at it, we could define keycodes for the other
> 	  OLPC buttons and switches on the screen. I think there are
> 	  some other sensor switches that could generate keycodes,
> 	  like opening the screen, rotating it around, and putting it
> 	  into book mode, so I will make some guesses at names for
> 	  them, just to get the discussion rolling. 
> 
> 	  #define KEY_XO_SCREEN_ROTATE   0x1b8
> 	  #define KEY_XO_SCREEN_POWER    0x1b9
> 	  #define KEY_XO_SCREEN_OPEN     0x1ba
> 	  #define KEY_XO_SCREEN_CLOSE    0x1bb
> 	  #define KEY_XO_SCREEN_IN       0x1bc
> 	  #define KEY_XO_SCREEN_OUT      0x1bd
> 
> 	  Is there an exhaustive list of all buttons and switches and
> 	  events on the OLPC? Are any more planned? Which ones should
> 	  be assigned keycodes?
> 
>       Rewrote setolpckeys.c code in Python (just uses ioctl, but needs to know keycodes).
>         Writing utilities like that in Python instead of C makes it easier to 
> 	reconfigure the keys on the OLPC without a C compiler. 
> 
>     + High level action support.
> 
>       GTK uses "Actions" to define the actions available in an
>       application, independent of the user interface used to invoke
>       them. Actions can be bound to user interface widgets and
>       keyboard accelerators, and they can hide, show, enable and
>       disable the corresponding parts of the interface. You can
>       subclass Action to define custom toolbar buttons and menu items.
> 
>       We need to define a generic way of navigating and executing the
>       application's actions from the gamepad.
> 
>       We can make Sugar specific actions that create the appropriately
>       styled and customized Sugar user interface widgets.
> 
>       Actions can be used to support navigation and operation of the
>       toolbar components from the gamepad:
> 
>       Actions have a list of their "proxy" components (toolbar
>       buttons, menu items, etc).
> 
>       Actions know how to execute a callback function, so the user
>       interface components tell the action to activate, instead of
>       calling the function themselves.
> 
>       The actions also know their labels, icons and tooltips.
> 
>       Actions can be shown and hidden, and all their proxies show and
>       hide.
> 
>       Actions can be made sensitive or not (disabled), and all their
>       proxies enable or disable.
> 
>       Actions have methods to construct toolbar buttons and menu
>       items, which subclasses can override to customize the user
>       interface. The higher level GTK UIManager calls these methods in
>       actions to create the user interface, although you can still use
>       Actions without the UIManager, by creating the components
>       yourself.
> 
>       There are three standard kinds of actions: Action, which creates
>       a normal button or menu item, ToggleAction, which creates a
>       toggle button (checkbox), and RadioAction, which creates a radio
>       button (multiple choice). 
> 
>       The GTK toolbars and menus support keyboard navigation of
>       sensitive buttons, as well as showing tooltips on sensitive
>       buttons, but it won't show a tooltip on unsensitive (disabled)
>       buttons, tabbing skips over disabled buttons, and it kicks you
>       out of buttons that get disabled while you're using them, by
>       moving the focus into the next button (whatever that happens to
>       be).
> 
>       All inactive items should show tooltips telling WHY they are
>       inactive, and WHAT you have to do before using them. 
> 
>       Unfortunately you can't get a tooltip on an inactive item. 
>       This needs to be fixed, so we can display helpful tooltips and
>       documentation on any item whether active or not. 
>  
>       Unfortunlatey you can't navigate to an inactive item.  This
>       needs to be fixed, so the user can use the gamepad to navigate
>       to an inactive item, to find out what it is and why it's
>       inactive. (Also, so you can navigate the interface predicably
>       with muscle memory, because the number of presses required to
>       navigate doesn't change depending on the state of the
>       application.)
> 
>       A problem with the current GTK keyboard navigation behavior of
>       not allowing inactive items to have the input focus, is that it
>       violates the principle of least astonishment, makes the
>       interface less predictable, and interferes with type-ahead:
> 
>         If you're focused on the "back page" button, and select it
>         until you get to the first page, then it will become inactive
>         (because you can't go back from the first page), which
>         results in the input focus being kicked out of the "back
>         page" button and throwing you into the "next page" button.
> 
> 	Imagine how hard this "astonishing" behavior would be to use
> 	if you were visually impared. It would not be very obvious
> 	that you had "bounced off" the first page and suddenly were
> 	moving forward. Other toolbars might arbitrarily relocate the
> 	input focus to an even more confusing button, when the one
> 	you're using gets disabled.
> 
>         It would be much less astonishing if the input focus simply
>         remained in the "back page" button when you hit the first
>         page, and a tooltop popped up saying "You can't go to the
>         previous page, because you are at the first page."
> 
> 	I've made a simple API for changing the sensitivity of a
> 	component, that takes a "reason" string (translated text) to
> 	show to the user in the tooltip of a disabled control (after
> 	the normal text). I've changed the code in the eBook reader
> 	that disables actions to figure out the most important reason
> 	(there might be several reasons to disable a control, but
> 	usually one is most important). It passes that reason to the
> 	API that disables the action, which currently just stores it
> 	away in a dictionary. Now it needs to be hooked up so it
> 	actually shows the tooltip with the reason when disabled.
> 
>       GTK has an "AcceleratorList" class that keeps a list of keyboard
>       accelerators which invoke actions. The UIManager helps manage
>       the accelerators for you automatically. 
> 
> 	I think we should use accelerators for normal keyboard
> 	acceleration of application actions, but implement the gamepad
> 	navigation stuff as a higher level framework that uses a more
> 	semantically meaningful model. (Not just low level tab/backtab
> 	focus navigation, or simple global keyboard accelerators, but
> 	an actual framework specifically designed to support browsing
> 	and executing arbitrary Actions via the game controller, and
> 	providing feedback about the reasons controls are disabled.)
> 
> 	The pygtk library has some methods on action_group to add
> 	actions "action_group.add_actions(action_list)", toggle
> 	actions "action_group.add_toggle_actions(action_list)", and
> 	radio actions "action_group.add_radio_actions(action_list,
> 	cur_value, callback, *args)". These all take an action_list
> 	that's a list of action specification tuples. That interface
> 	is more brittle and less extensible than it should be, because
> 	it takes tuples instead of dictionaries, and it only makes
> 	stock GTK actions. 
> 
> 	We need a more flexible API than add_actions and its ilk, which
> 	supports custom Sugar actions, and uses dictionaries instead
> 	of tuples, so we can easily pass in additional optional
> 	parameters without changing the API.
> 
> 	I have taken a first cut at rewriting pygtk's ActionGroup's
> 	add_actions, add_toggle_actions and add_radio_actions
> 	functions in Python, and changing them to use custom
> 	SugarAction, SugarToggleAction and SugarRadioAction classes.
> 
> 	But I still don't think the add_actions_esque interface is
> 	flexible enough. It needs an easy way to add other custom
> 	widgets (like the search text input field), and it should make
> 	it easy to customize the user interface classes and configure
> 	the objects by passing in additional optional parameters and
> 	configuration dictionaries. (For example, the action
> 	specification should be a dictionary that has an optional key
> 	to specify the toolbar button and menu item classes, and it
> 	should be possible to plug in different kinds of user
> 	interface controls than toolbars and menus, like pie menus and
> 	gamepad specific interfaces). 
> 
>       GTK has an "atk" module that interfaces to the accessibility toolkit. 
> 
>         I'm not clear on just what this does and how it can help us,
>         and I would love to hear from somebody who's familiar with it.
> 
>         I think we can do what we need to support the gamepad without
>         using the atk, but maybe it could be helpful. But my
>         impression is that it's more geared towards screen readers, is
>         tied closely to the user interface layout (declaring that this
>         label is associated with that control, etc), and it looks like
> 	it takes a lot of lines of code to use (unless you use some
> 	kind of framework that supports it automatically).
> 
>       GTK has a "UIManager" that knows how to build menu bars, menus,
>       toolbars and buttons from XML files, and tie together actions,
>       accelerators, menus and toolbars. The XML describes the
>       structure and layout of the user interface, but refers to the
>       actions by name to configure the buttons and menu items with
>       labels, icons, tooltips, accelerators, callbacks, and custom
>       component classes. There is nothing in the XML about what label,
>       icon, tooltip or widget class to use -- just an Action
>       name. It's up to the Action to figure out the visual appearance
>       and behavior of the gui. That's why it should be easier to use
>       custom Action classes.
> 
>         I think we need to write our own UIManager and accessibility
>         framework that addresses our specific needs (like the
>         focus/tooltip/disable reason issues discussed above, custom
>         sugar controls, gamepad support, etc), because I don't think
>         the UIManager (plus the pygtk utilities that go along with it)
>         are flexible enough to support our needs. I think it would be
>         easy to implement it in Python, in a way that would be a lot
>         more flexible and easier to extend that the current C
>         implementation of GTK's GuiManager.


GTK UIManager also does not support tabbed toolbars and palettes (which
will display the accellerators).

Marco