[Sugar-devel] How to implement TTS + highlighting

[James Simmons]

Chirag,

The phrase "widget from Read Etexts" in Ben's email gave me a thought.  
You see, there is no "Read Etexts widget".  I just use a regular 
textarea and scrolled window.  But it occurs to me that maybe you could 
make such a widget which combines the textarea and scrollable window, 
plus the text to speech and highlighting code.  You could have a method 
in the widget to activate speech and another to pause speech.  Maybe you 
could provide a method that lets the developer specify a key to start 
and pause speech.  (For instance, "KP_End" is what Read Etexts uses).   
The spoken words would be highlighted using Pango, and the scrollbars 
would be adjusted to make the spoken word visible at all times.  This 
widget could become part of the Sugar Python library.

This kills two birds with one stone.  You could use the widget in Sugar 
itself to do what you are describing.  In addition to that, Activity 
developers using Python would have a simple way of adding in place TTS 
with highlighting to their Activities.  This might have all kinds of 
interesting possibilities.  You could write "Adventure" style games 
where the game spoke to the child and had TTS with highlighting to 
encourage the child to read along with the instructions spoken.  
Multi-player games with collaboration could have a built in chat 
function that used the widget.  Players could send each other messages, 
like "Checkmate!" or "You sunk my battleship!"

I believe doing things this way would make your work more useful.  You 
would have the built in Sugar facility you're proposing to add TTS to 
Activities with no support for it, plus you provide a standard way for 
Activities to do their own TTS in place.  The widget could be made 
simple enough to use that children could use it in their own programs.

James Simmons

chirag jain wrote:
> well acheiving captioning or karaoke style coloring in the same window
> where the text is selected is something a difficult task. I have no
> idea how it can be acheived. My idea is the same as you mentioned that
> a separate window containing the selected text from the current window
> will be opened. The captioning will be acheived in that separate
> window not in the current window.
>
> I will mention it specifically in my proposal also.
> On 4/2/09, Benjamin M. Schwartz <bmschwar at fas.harvard.edu> wrote:
>   
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> James Simmons wrote:
>>     
>>> As you point out in your proposal, highlighting the word as it is spoken
>>> is a big part of the benefit of what you're proposing.  If all you
>>> wanted to do was capture some highlighted text in the clipboard and have
>>> it spoken in a voice you can configure in a control panel, that would be
>>> easy, even trivial.  It's the highlighting that's difficult.
>>>       
>> ...
>>     
>>> If I had to write a facility that did what Read Etexts does outside of
>>> the Activity I wouldn't know how to do it.  It seems to me that
>>> highlighting is best done by the Activity itself.  I can't deny that it
>>> would be useful to have all this work done as you have described without
>>> the Activity knowing anything about it, but it doesn't seem feasible.
>>> You'd have to have something that could work with gtk textareas, the
>>> evince component Read uses, Abiword, and everything else that came along.
>>>       
>> What if we forget about showing the highlighting "in place"?  Instead, the
>> TTS button can pop up a small window or palette showing the text from the
>> "clipboard" (actually the PRIMARY selection[1]).  This window would use
>> the widget from Read Etexts or Listen and Spell, to provide active
>> highlighting.  As soon as TTS completes, the window disappears.
>>
>> That gives us "karaoke" highlighting with any activity that supports
>> selection.
>>