[Sugar-devel] GSoC Proposal: Multimedia Broadcasting

[Geza Kovacs]

On Fri, 2009-04-03 at 00:54 -0400, Benjamin M. Schwartz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Eben Eliason wrote:
> > On Fri, Apr 3, 2009 at 12:02 AM, Geza Kovacs <gkovacs at mit.edu> wrote:
> >> Thanks for the responses. While I did consider the usefulness of
> >> VNC-type remote desktop control interaction, which "Multi-point remote
> >> desktop" seems to be
> ...
> > Right. I'm actually on board with this point. I think VNC-style remote
> > multi-pointer collaboration could be cool, but I, like you, also think
> > there's plenty of benefit to be gained simply from the ability to
> > "watch" someone else: either a single activity, or their full screen.
> 
> This is becoming very confusing.  Two points:
> 
> 1.  Chris's Multi-pointer VNC is a very special, cutting-edge system.
> Multiple pointers are not present in standard VNC.  Geza is right that our
> toolkits and apps are not ready for MPX anyway.
> 
> 2.  Standard VNC, by default has all mice controlling a single cursor, but
> it can also be configured with behavior desired here.  For example, you
> can launch "x11vnc -viewonly -shared", which allows many clients to
> connect, but not provide any input.
> 
> Geza proposes to provide an audio stream, and a video stream that is
> either from the videocamera (presumably as Theora video) or the contents
> of the screen.
> 
> I think this is a really awesome proposal that would be hugely beneficial
> to Sugar.  I see a few major technical difficulties to consider.
> 
> 1.  Screen feeds and video feeds need different transports.  Our hardware
> is not going to be capable of using Theora (or any other video codec) to
> transmit the whole screen at acceptable quality, but it's also not going
> to be capable of transmitting full motion video in VNC.  Therefore, we
> need to use Theora for video and VNC for the screen capture.  This creates
> a bit of a complex situation.  Perhaps VNC could be added to farsight as a
> codec?
> 

Actually, during the fall semester I had access to a few XOs from the
Media Lab, and happened to try this very thing out (use gstreamer to
capture video from webcam and stream it live to an icecast server). With
VGA resolution with Theora, I was indeed getting horrible framerates,
roughly 0.5 frames per second. At QVGA, framerates improved
considerably, going around 10 frames per second; however I suppose QVGA
is too low-resolution to be acceptable. I had slightly better
performance with some older codecs, I believe either MPEG-1 or MJPEG,
with VGA streaming at roughly 3 fps. I didn't try out H.261 or H.263,
but I believe those should also be much less expensive than Theora to
encode and decode, as they're used quite extensively in embedded
devices. In this case, since all traffic is going through the local
wireless network, bandwidth isn't the primary bottleneck, but instead
encoding and decoding video is, therefore using a less cpu-intensive
codec is likely to be advantageous; I believe that if I use the
appropriate encoding options and codec combinations to make compression
as fast as possible then I should be able to stream VGA video at
acceptable frame rates. Since VNC itself, unlike X11, uses a
raster-based format, I don't believe it's going to be any more efficient
CPU-wise than a very primitive video codec like MJPEG or MPEG-1 at
fastest compression settings in any context, so I don't believe I should
even bother with VNC.

> 2. Telepathy does not currently support multi-point conferences.
> Collabora is working on it, but from what I hear it's not likely to be
> ready by June.  Thus, you will not be able to work entirely through
> telepathy-farsight.  Farsight itself does support multi-point conferences
> (I think...), but I am not sure that it has any optimization for the LAN
> case, where all transmissions are essentially broadcast.  Transmitting
> individual unicast streams to 30 laptops over a wireless network would be
> an absolute disaster... but I assume that VNC requires the ordering and
> delivery guarantees of TCP, so if you broadcast it clients will die as
> soon they miss a packet.
> 

Since the objective is to broadcast an audio and video stream to the
masses, I don't aim to use telepathy for this purpose; rather I aim to
use an icecast server which runs on the broadcasting machine.

> 3.  (distant third).  By default, VNC does not transmit the cursor as an
> image.  Instead, it allows the client to show the local cursor.  This
> makes sense, since because hardware acceleration means the cursor is often
> not even in the framebuffer.  You will have to make the cursor visible
> somehow, in order to act as a kind of "laser pointer".
> 
> IMHO, the proposal would be much stronger if it included solutions to
> these problems.
> 
> - --Ben
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.10 (GNU/Linux)
> 
> iEYEARECAAYFAknVlosACgkQUJT6e6HFtqTuZQCfcSWp8gO0z9mkitbIi1u01I8e
> r24An2yOGZi4yVaJ0VT41bXc+36/J7Lf
> =R4Cm
> -----END PGP SIGNATURE-----