[sugar] Ideas for Sugar development environment from HyperLook SimCity

[Don Hopkins]

I love the ideas behind Smalltalk, EToys and HyperCard, and would like 
to combine them with ideas from visual programming languages like Robot 
Odyssey, KidSim, Klik-and-Play, SimAntics, Body Electric/Bounce, 
Max/MSP/Jitter, etc.

Here are some ideas about HyperLook and other systems, that could be 
applied to Sugar:

HyperLook was a PostScript-based user interface development environment 
for the NeWS window system, which Arthur van Hoff created at the Turing 
Institute in Glasgow.
http://www.donhopkins.com/home/catalog/hyperlook/

I helped develop HyperLook into a commercial product, with a editable 
user interface development environment, as well as a redistributable 
non-editable runtime, and I used it to port SimCity to Unix, and develop 
other components and applications .
http://www.donhopkins.com/home/catalog/hyperlook/
http://www.donhopkins.com/home/catalog/hyperlook/HyperLook-SimCity.gif

HyperLook was inspired by HyperCard, but it additionally provided a 
client/server programming model, and more powerful graphics and 
scripting based on NeWS's object oriented dialect of PostScript.
http://www.donhopkins.com/home/catalog/hyperlook/TalkInterfacing.gif

The NeWS window system was like AJAX, but with:
1) PostScript code instead of JavaScript code
2) PostScript graphics instead of DHTML graphics, and
3) PostScript data instead of XML data.

It had a unified programming/graphics/data/networking model based on 
NeWS's extended multi-threaded object-oriented dialect of PostScript, 
instead of a hodge-podge of accidental technologies. (Although I will be 
the first to admit the X11/NeWS merge was quite a hodge-podge and 
huge-kludge!)
NeWS had an object system based on the simple dynamic ideas of 
Smalltalk, implemented with the PostScript dictionary stack, supporting 
multiple inheritance and runtime modification of objects and classes.
http://www.donhopkins.com/home/catalog/hyperlook/HyperLookInfo10.gif

HyperLook was a gui development framework and desktop environment, that 
extended NeWS with a user-editable structured PostScript graphics 
format, a persistence system, a HyperCard-like delegation model using a 
network based client/server library, that passed messages from button to 
page to background to stack, and finally over the network to the 
application (and back), and an entire user interface toolkit, window 
manager, gui editor, clipboard and other desktop tools and services, all 
designed around the PostScript graphics format and message passing system.
http://www.donhopkins.com/home/catalog/hyperlook/TalkInterfacing.gif
http://www.donhopkins.com/home/catalog/hyperlook/HyperLookInfo2.gif
http://www.donhopkins.com/home/catalog/hyperlook/HyperLookInfo11.gif
http://www.donhopkins.com/home/catalog/hyperlook/HyperLookInfo13.gif

Users could create their own integrated applications, task oriented 
interfaces, presentations and journals, by cutting and pasting high 
level components and graphics together, configuring them with property 
sheets and graphical editors, scripting them with PostScript message 
handlers, sending and receiving messages between other stacks and 
applications, and customizing applications to suit their needs.
http://www.donhopkins.com/home/catalog/hyperlook/HappyProps.gif

It provided a user-editable window manager, that allowed you to draw and 
shape the window frame with arbitrary PostScript graphics, cut and paste 
you own resize corners, close boxes, menus, buttons, pie menus and other 
controls, as well as combining and scripting together multiple 
application components and custom graphics.
Because everything was based on PostScript, you could print any window 
to a PostScript printer or copy it to the clipboard, and iconified 
windows were just live miniaturized views.
http://www.donhopkins.com/home/catalog/hyperlook/Clipboard.gif

Developers could create back-end services (like audio mixing) and 
applications (like SimCity) that could send messages back and forth, use 
shared memory for efficient bitmap animation, and share services like 
audio mixing with other applications.
http://www.donhopkins.com/home/catalog/hyperlook/TalkGraph.gif
http://www.donhopkins.com/home/catalog/hyperlook/TalkView.gif

I developed SimCity in parallel with HyperLook, so there was a powerful 
synergy, SimCity drove the development of many of HyperLook's features, 
and the other way around.
SimCity extended HyperLook with client/server based components like the 
city editor, map view, graph display, which could be copied and pasted 
and placed anywhere in the interface.
It was great to have a demanding application like SimCity as an acid 
test, to shake out bugs and limitations of the platform, and prove the 
abilities of general purpose components like bitmap animation and audio 
mixing.
http://www.donhopkins.com/home/catalog/hyperlook/TalkIntro.gif
http://www.donhopkins.com/home/catalog/hyperlook/TalkNewCity.gif

HyperLook was a commercial product, with a WYSIWYG interface development 
environment for developers, and a freely redistributable non-editable 
runtime.
We released it at the same time as SimCity (using SimCity as bait to 
entice people to try out the included HyperLook runtime).
http://www.donhopkins.com/home/catalog/hyperlook/HyperLook.README
http://www.donhopkins.com/home/catalog/hyperlook/SimCity.README

The user interface development environment could be stripped out of the 
system to create a non-editable binary-encoded runtime version for 
shipping turn-key commercial products, which I delivered with SimCity.
But if you had the development version, you could create your own 
stacks, put the interface into edit mode, cut and paste user interface 
components around, and edit their scripts (enabling user interface 
vandalism).
Personally, I think all users should have a user-editable development 
environment, but it's important to be able to lock it down and constrain 
it, so casual user's can't accidentally break the system, or get 
confused with unnecessary details.
http://www.donhopkins.com/home/catalog/hyperlook/TalkRunTime.gif

HyperLook included a "warehouse" of pre-configured object templates, 
which you could cut and paste into your own stacks, and configure and 
script to create your own HyperLook applications, custom interfaces, 
interactive presentations, etc.
http://www.donhopkins.com/home/catalog/hyperlook/ButtonIdeas5.gif
http://www.donhopkins.com/home/catalog/hyperlook/Warehouse3.gif

Like HyperCard, you could copy and paste components around (including 
per-object customizations like scripts, properties and graphics) to 
construct and edit your own interfaces.
Like HyperCard, you could open up a property sheet or script editor on 
any object.
Unlike HyperCard, the scripts were written in object oriented 
PostScript. (Arthur van Hoff also wrote PdB, an object oriented C to 
PostScript compiler, and later went on to write the first Java compiler 
in Java at Sun!)
Unlike HyperCard, the property sheets were implemented as HyperLook 
stacks themselves, which made it easy to develop new components or 
customize property sheets for existing components (i.e. simplified for 
kids, graphically oriented for designers, advanced for developers), and 
the user interface editor itself was even a plug-in component that could 
be removed or replaced (to make a non-editable runtime, or to plug in 
simpler or more advanced gui editors).

HyperLook was able to implement its own property sheets as stacks, 
because it had a sufficiently rich set of built-in components including 
text and graphics editors, and stacks could be scripted to import and 
export properties to control settings (to convert between data types and 
control values, implement apply/cancel, change detection, and selecting 
previously edited objects).

HyperLook included a nice little PostScript graphics editor component 
that you could integrate into your own applications and property sheets.
For example, there is a user-customizable clock that lets you edit its 
face and hands (by incorporating three graphics editors in its property 
sheet), and there are some example clocks, pre-configured in the warehouse.
You can copy them into your own stacks, place and stretch them (since 
they're scalable PostScript graphics),  modify their appearance with the 
clock editor property sheet, and paste your original creations back into 
the warehouse to use again:
http://www.donhopkins.com/home/catalog/hyperlook/NeatClockProps.gif
http://www.donhopkins.com/home/catalog/hyperlook/NeatClocks.gif

Here's a cellular automata laboratory that uses the shared memory raster 
animation library to integrate a Toffoli/Margolis CAM-6 simulator I 
wrote in C with the PostScript graphics editor (so you can cut and paste 
PostScript graphics into live running cellular automata, and copy the 
cells into the graphics editor, and generate garish but seamlessly tiled 
screen backgrounds, and place a live bubbling cellular automata view 
component clipped into a lava-lamp shaped window!):
http://www.donhopkins.com/home/catalog/hyperlook/CAM.gif

The nice thing about having a standard PostScript based structured 
graphics format, is that the entire system supports it, so you are free 
to do fun stuff like making a clock face out of a cellular automata or 
SimCity map, copying an entire window including its user interface 
components as structured graphics, clipping and stretching it in in the 
graphics editor, and using it as a clock hand, or whatever else you can 
think of.

Here's a transcript and video demo of HyperLook SimCity, cellular 
automata,  user interface and graphics editing, and cutting and pasting 
graphics between various HyperLook applications.
http://www.donhopkins.com/home/catalog/simcity/hyperlook-demo.html
http://www.donhopkins.com/home/movies/HyperLookDemo.mov

It is extremely important that the base system includes a standard 
user-editable structured graphics format (higher level that raw 
PostScript, but including Encapsulated PostScript or modern equivalents 
like PDF, SVG, PNG, etc).
It's also essential to have a reusable structured graphics editor 
component, and also that all of the property sheets and applications 
take full advantage of it, so you can edit every visual aspect of the 
user interface, and copy any graphics to the clipboard with their 
structure intact.

HyperLook's graphics editor (HyperDraw) was not too fancy, but it was 
really easy to use for the stuff you want to do 90% of the time.
A bitmap image editor would have been really nice too, but that was hard 
to build into the NeWS server without a C client, unfortunately, so we 
never got around to that (although the CAM stack has some simple drawing 
tools for  playing with the live cellular automata pixels through shared 
memory).
Something like Photoshop or GIMP would be too complicated and 
monolithic, unless it could be stripped down and re-packaged as an 
light-weight, easy-to-use, plug-in component.

I've worked on and programmed in some interesting visual programming 
languages like Bounce (aka Body Electric, VPL's VR scripting language), 
SimAntics (The Sims behavior scripting language) and PSIBER (visual 
PostScript and NeWS debugger), and played around with others like 
Max/MSP/Jitter, ProGraph and KidSim (aka Cocoa, Stagecraft Creator, 
which was at once point actually implemented in ProGraph!).

Visual programming languages are great fun, and can be extremely 
productive at their finest, but they tend to be quirky, nonstandard, 
have more differences than similarities, and extremely weird ways of 
looking at the world.
There is no one best way to design the ultimate visual programming 
language, but there are many good ideas to be tried (and bad ideas to be 
avoided) from previous systems, that could be recombined in interesting 
ways.

There will never be one true visual programming language, just like 
there will never be one true text programming language, so I think the 
system should be designed to accommodate different languages and skill 
levels.
In the same way Lisp is excellent for meta-programming application 
specific mini-languages, I would like a similarly enlightened system for 
creating application specific visual programming languages (like the 
Eclipse Visual Editor project, but for a nice dynamic language like 
Python, instead of Java).

Microsoft's IScriptingEngine interface lets you plug different scripting 
languages like Visual Basic Script, JScript, Perl and Python into the 
web browser and other applications.
But COM/ActiveX/OLE Automation isn't the only way to do that.
I am quite happy that Sugar is using Python, because it makes a great 
common runtime and "virtual machine language," for implementing visual 
programming languages and integrating software modules (especially with 
tools like SWIG for integrating native libraries and C++ code).

My ultimate fantasy gui environment would support plugging in different 
script editors, with not only easy-to-use but also advanced visual 
programming interfaces, tailored to specific tasks and skill levels 
(like handling and sending events, solving constraints, processing 
signals, playing music, controlling robots, blinking lights, dancing 
bears, etc).

Python will serve well as the internal "machine language" that you 
compile the visual graph notations into (or interpret the visual 
languages directly in Python, if practical).
So kids could easily create a high-level graphical scripts like The 
Learning Company's "Robot Odyssey" or Maxis's "Klik-and-Play", and 
developers could meticulously create low-level behind-the-scenes scripts 
in visual languages like "Body Electric" and "Max", or text language 
like Python (or a hybrid of the visual and text languages -- check out 
how PSIBER implements a new visual interface to the existing PostScript 
language, and how Cycling 74's Max/Jitter integrates JavaScript with the 
visual programming language: JS code inside icons, JS objects on wires!):
http://en.wikipedia.org/wiki/Klik_&_Play
http://www.cycling74.com/products/jitter
http://www.cycling74.com/download/JavascriptInMax.pdf

Just imagine a visually scriptable version of SimCity, that lets you 
clone and edit the monster, and reprogram him to tend the forests 
instead of stomping on buildings!

    -Don


Some notes on alternatives to Python:

I love Python, and it's my primary scripting language of choice, but I'm 
not a monolinguist or strict interpretationalist, so I think some other 
languages are worth considering and learning from, in the long term:

Lua:

Lua is an excellent, well designed and implemented scripting language, 
that's extremely light weight and efficient, truly open source, and is 
also nicely supported by SWIG.
It's well worth considering if you're starting from a clean slate, and 
don't need all of Python's modules, and do care a lot about speed and 
size, and want to tightly integrate the scripting language with the 
application, without requiring the installation of a bunch of external 
files, resources and libraries.
It's very popular in the game industry, widely used in World of 
Warcraft, inspired by Lisp but without the macros, easy to learn and to 
read, has a pretty good JIT that works on x86 systems, and has a bunch 
of useful extensions (but not nearly as many as Python).
Lua has an extremely flexible roll-your-own object system (not one 
standard oop paradigm, but the basis for making all different kinds, but 
very simple, not nearly as powerful and general purpose as CLOS, not 
quite as elegant and minimal (nor as bloated a runtime) as Self).

SWIG supports Lua objects at a kind of low level, but can be extended to 
nicely support higher level objects, given a particular model.
But SWIG does not work as well out of the box with Lua as it does with 
Python (since Python's object system is well known and not a moving 
target subject to the whims of the application programmer).
The fact that Lua doesn't have Python's huge set of modules is both a 
blessing and a curse, but Lua actually a bit faster and much smaller 
than Python, vastly simpler, has a few weird but tolerable quirks (like 
one-based array indexing), and is conceptually cleaner and Lispier than 
Python in some nice ways.
According to the computer language shoot-out, it's one of the fastest 
and smallest interpreted scripting languages (and presumably even faster 
with the JIT, which is not included in that benchmark).
Lua's ratio to C's speed is 6.6, compared to Python's 7.3, which comes 
in right behind Lua.

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all

JavaScript:

JavaScript is a horribly designed and implemented language, although 
it's ubiquitous, never going away, well worth learning, and good enough 
to get a lot of stuff done in (Lisp with C syntax, without macros).
However, SpiderMonkey (the original JavaScript interpreter, and the one 
in Firefox) is extremely inefficient and wasteful of memory.
According to the computer language shoot-out, it's absolutely the worst 
language benchmarked, by a long shot.
SpiderMonkey's ratio to C's speed is 31, compared to Ruby's 16, which is 
the next slowest language.
Incredibly, its memory usage is even worse compared to other languages 
(ratio to C's size is 32, compared to the next worst Smalltalk 
VisualWork's 13), so it's not like it's making a memory-vs-speed 
trade-offs to be that slow.

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all

I have a hard time believing that JavaScript will ever run as fast or 
small as Lua, simply because Lua was well designed to be fast and small 
on purpose (and thus avoids inefficient pitfalls by design), and 
JavaScript wasn't designed at all, it just happened accidentally.
It would be better to take all the cleverness that you'd have to apply 
to JavaScript to make it fast, and apply it to Lua instead -- you'd be 
way ahead of the game, you wouldn't be fighting against but remaining 
compatible with bad designs, and you'd end up with a much nicer language.
Another important thing that would make me choose Python or Lua over 
SpiderMonkey/JavaScript, regardless of the speed and size differences, 
is that SWIG supports Python and Lua well, and it doesn't currently 
support SpiderMonkey.

However, there is an interesting recent development in JavaScript's favor:
Adobe has donated the source code for the ActionScript VM that's in the 
Flash 9 (AVM2)!

http://www.mozilla.org/projects/tamarin/
http://developers.slashdot.org/article.pl?sid=06/11/07/1324224
http://blogs.business2.com/utilitybelt/2006/11/web_20_bombshel.html

It turns out that AVM2 does not totally suck. (There are no benchmarks 
available comparing it to other JavaScript VM's, but you can easily see 
and feel the huge difference between Flash 8 and Flash 9).
The previous versions of Flash were based on an absolutely horrible and 
inefficient virtual machine, which has finally been entirely rewritten 
by grown-ups, and includes a JIT, so it has drastically better performance.
They have not made the entire Flash player open source, just the 
ActionScript byte code VM, which does not include an actual JavaScript 
source code to AVM2 byte code compiler.
(Although they are developing a prototype JavaScript compiler in 
JavaScript, it is not fully complete and tested, although it's a great 
ambitious idea.)
Adobe has their own ActionScript to VM byte code compiler, but it's not 
open source.
Of course a web browser needs a JavaScript compiler, since it downloads 
JavaScript in source code form.
But it also totally doesn't suck to be able to download pre-compiled 
binary JavaScript byte code (i.e. SWF files).

OpenLaszlo is developing its own open source JavaScript to AMV2 byte 
code compiler (the Legals project), so OpenLaszlo will be able to 
compile code that it will run in AVM2.
The Mozilla project plans on incorporating AVM2 into the browser to 
accelerate JavaScript, but there's still a lot of work to be done.
But I think that AVM2 is a great long term strategy to solve the 
performance problems of SpiderMonkey in the web browser, that will 
dovetail with OpenLaszlo, and be useful for developing efficient AJAX 
applications, using a completely open source software stack.

http://www.openlaszlo.org




Here are some references to work I've done in the past, describing some 
ideas that might be applied to Sugar.

NeWS is James Gosling's "Network extensible Window System", which was 
based on a multithreaded, networked, object oriented dialect of PostScript.
I developed pie menus for NeWS at the UMCP HCIL, worked on the Open Look 
NeWS Toolkit at Sun, and various NeWS tools and applications for other 
companies.
http://www.donhopkins.com/drupal/node/92

I developed lots of other fun stuff with NeWS, including various 
implementations of pie menus and tab windows:
http://www.donhopkins.com/drupal/node/92
http://www.piemenu.com/DDJPieMenuArticle.html
http://www.piemenu.com/

The NeWS Toolkit was an Open Look user interface toolkit written 
entirely in NeWS PostScript, which I helped develop at Sun, and 
integrated with HyperLook, and used to develop pie menus and tab windows.
http://www.donhopkins.com/home/catalog/images/tnt.gif

Pizza Tool was a graphical Open Look gui for editing and ordering pizzas 
via FAX, that I wrote as a programming demo for The NeWS Toolkit:
http://www.donhopkins.com/home/catalog/images/pizzatool.gif

The HyperTIES hypermedia browser, with pie menus and "applets" scripted 
in PostScript (for Ben Shneiderman at the UMCP HCIL):
http://donhopkins.com/drupal/taxonomy/term/61

Emacs "thin wire" NeWS display driver for UniPress Emacs (and later Gnu 
Emacs), with pie menus and multiple tab windows, and an authoring tool 
for HyperTIES:
http://donhopkins.com/drupal/node/101

HyperLook (aka HyperNeWS and GoodNeWS) was a HyperCard-like user 
interface development environment for the NeWS window system.
I worked on HyperLook with Arthur van Hoff at the Turing Institute, and 
used it for various applications.
http://www.donhopkins.com/home/catalog/hyperlook/

SimCity is a constructive simulation game from Maxis.
I ported SimCity to HyperLook and X11, and designed a multi player 
scriptable user interface using TCL/Tk.
http://www.donhopkins.com/home/catalog/simcity/index.html

PSIBER is a visual interface to PostScript and interactive debugger for 
NeWS.
I developed it at the UMCP HCIL and Grasshopper Group.
http://www.donhopkins.com/drupal/node/97

Bounce (aka Body Electric) is a real time data flow visual programming 
language, developed at VPL by Chuck Blanchard and used by Jaron Lanier 
and others for integrating midi and various i/o devices, driving 3d 
rendering engines, scripting VR, scientific, training and artistic 
simulations.
I programmed multimedia and character simulations in Bounce, ported it 
to PowerPC, developed its user interface, a COM plug-in and type 
extension system, and multimedia support, with David Levitt at Levity 
and Interval Research Corporation.
http://www.donhopkins.com/home/catalog/lang/bounce/bounce.html

SimAntics is the visual programming language for scripting AI and 
behavior in The Sims, pie menus (aka marking menus, compass menus) are 
an efficient Fitts'-law-friendly menu selection technique.
I programmed in SimAntics, ported it from the Mac to Windows, and 
developed its user interface (Edith) and primitives (character 
animation, etc), with Will Wright at Maxis:
http://www.donhopkins.com/drupal/taxonomy/term/35

The Dumbold Voting Machine is an educational agit-prop plug-in for The 
Sims, that I programmed in SimAntics to educate people about real voting 
machine problems -- check out the "high concept" stuff at the end of the 
page:
It's meant to stimulate discussion, illustrate how computer programming 
is free speech, and why all election software should be open source and 
publicly inspectable.
http://www.dumbold.com/dumbold

This is a movie that demonstrates the character animation system, pie 
menus, direct manipulation object placement and architectural editing 
tools that I developed for The Sims, as well as an in-depth demo of 
SimAntics visual programming with the Edith tool (stepping through the 
Satan Generator code):
http://www.donhopkins.com/home/movies/TheSimsPieMenus.mov