Tag Archives: user interface

Lost in the flood #1: programming, Japanese gardens, music, machine learning

The problem: an ever expanding set of Chrome tabs of read/study/watch/respond-to later material. I’m trying a new solution: put the links in a blog post, write why I haven’t closed the tabs, close the tabs. (And come back to them later. Maybe.)

Let’s go.

What is embodied programming? blog post by Alex McLean (@yaxu). Alex is trying to give a name to something he’s calling embodied programming. From a quick skim I think it’s related to programming in a cyber-physical loop with the machine. But I need to read Alex’s post properly to be sure.

I first encountered the term cyber-physical in a paper by Andrew Sorensen (@digego). Programming With Time – Cyber-physical programming with Impromptu (Sorensen and Gardner 2010). Andrew Sorensen recently posted a video of his latest work: Physics Playroom at QUT’s new Science & Technology precinct, and gave a detailed technical description on the extempore mailing list. Physics Playroom is an impressive piece of interactive programming making use of a huge aggregated video wall, a cluster of computers, and bank of multi-touch screens capable of simultaneously interacting with a hoard of school kids. Playroom is developed in Andrew’s latest live coding system extempore. (If you’re reading this Andrew, thanks very much for answering my question in such detail!)

Emotional responses to music: The need to consider underlying mechanisms (Juslin and Västfjäll 2008). Related to my recent emotional response on the music-dsp mailing list.

One thing I did find time to watch was Dream Window: Reflections on the Japanese Garden (1992) a stunning documentary. IMDb describes it as “Exquisite exploration of landscape and Toru Takemitsu’s music for a Japanese moss garden.” Highly recommended.

My interest in Japanese gardens arose from an ongoing reflective process concerning software design. I was thinking about the concept of negative space — the space around things — the “anti-object”. This lead me to the related (but I don’t think synonymous) Japanese word “ma(here’s another definition of “ma”, and an interesting discussion about Ma at Metafilter). In the process I found MA: Space/Time in the Garden of Ryoan-Ji a video artwork by Takahiko Iimura. The artist’s notes about the work provide an interesting insight into the compositional process. I would like to know more about ma.

Generic programming with C++ concepts and Haskell type classes—a comparison (J.F. Bernardy et al 2010). Lately I’ve been writing C++ code that is better expressed using compile-time polymorphism (templates) than C++’s good old runtime polymorphism (virtual functions). I’ve been thinking about the not-yet-extant C++ concept specifications, which lead me to the linked paper. My knowledge of Haskell is limited. The relation between C++ templates and Haskell type classes has been on my radar since I encountered Bartosz Milewski’s Monads in C++ blog post a couple of years back.

The Fast Cauchy Transform and Faster Robust Linear Regression (K.L. Clarkson et al 2012) paper at arXiv.org and summary at Nuit Blanche (via @IgorCarron). Fast, Robust, Randomised, what more could you want? Fast L1 regression is a hobby of mine. If nothing else this paper will give me a chance to practice my Linear Algebra.

Machine learning – Random forests video of lecture by Nando de Freitas. Random forests, aka decision forests, and ensemble methods. Seems like something I should know more about.

NI have released Traktor DJ for iPad. I am intrigued by the UI design.

Striking it Richer: The Evolution of Top Incomes in the United States (Updated with 2011 estimates) by Emmanuel Saez (via @accessjames). My version of following the movie stars.

Reflections on Bret Victor’s “Explorable Explanations”

I’ve been reading and thinking about maths quite a bit lately. In the process I happened upon Bret Victor’s killmath posts on interactive dynamic systems, explorable explanations and usable math. Some of the killmath examples really got me thinking. I’ve actually started coding a few experiments around this, but for now I just want to summarise some reflections and comments I have about Bret’s research. There is a lot more to Bret’s site than I touch on here, so I really recommend you go check it out if you havn’t already.

Explorable (Visual) Explanations

To give a feel for the material: one of Bret’s posts in Explorable Explanations gives an interactive example of a digital state variable filter (it’s not actually Chamberlin’s topology as I had previously tweeted). The example presents different graphical and symbolic representations of the filter and allows you to manipulate the filter’s parameters and hear the result. One of Bret’s videos demonstrates an interactive interface for exploring the Predator/Prey dynamical system with multiple cross-linked interactive visualisations. I discuss both of these examples below.

If you’ve read Edward Tufte’s book Visual Explanations — images and quantities, evidence and narrative you’ll have an idea of some of the concepts and considerations involved. Pictures, diagrams, sounds, and dynamic time-based demonstrations can all tell stories in ways that numbers and formulae can’t — potentially allowing us to gain new insights, solve problems and draw conclusions that are not accessible (to the uninitiated, if at all) by simply looking at mathematical formulae or unstructured data. Explorable Explanations extend visual explanation with interactivity allowing the reader/user to engage, explore, play, and listen, in order to gain a deeper understanding of a system’s dynamics and behaviour under varying conditions.

Of course, animations, simulations and interactive examples in the form of Java applets and Flash interactives have been circulating for years. To give a random example: sorting algorithm visualisation applets from the field of algorithm visualisation. Often though these applets are demonstrative rather than truly interactive and explorable. Closer to the mark is a recent site from the U.K. Department of Energy and Climate Change: My2050, a Flash interactive that allows users to experiment with a parametric model of energy generation and consumption, lifestyle choices, and modes of transport to design their future world. By moving sliders on the screen, users explore the environmental effects of their choices while being challenged to achieve a workable set of parameters. It’s a pretty neat web site that illuminates some of the environmental trade-offs we face.Click here to try it

To me there is something implied by Bret Victor’s work that isn’t present in DECC’s My2050 or in simple illustrative Java applets. DECC’s My2050 site supports developing intuition about the relationships between energy consumption, production and carbon footprint, however the underlying mathematical model and data set are not visible. In contrast, it seems to me that Bret’s examples are not just about increasing user intuition about system outcomes. They also support increased user awareness and expression of explicit models of system structure and dynamics. For example, by simultaneously showing the various representations of the state variable filter, or by showing multiple overlaid trajectories in the Predator/Prey model. I think this is a really interesting direction, but as I discuss later, the mechanisms for illustrating structure and inter-relationship deserve further development.

Scrubbable numbers

This brings me to Bret’s recent post on the Scrubbing Calculator. One facet of the Scrubbing Calculator that caught my attention was scrubbable numbers — numbers that you can click and drag on to change their values (not unlike a spin box) to support interaction with typed mathematical expressions. Changing one number changes the others according to their context in the surrounding text. Try clicking and dragging up and down on the numbers in the sentence below and you’ll see what I mean:

1 plus 2 equals 3

The point here is that this functionality would be available as you type, not just in a pre-written document like this one. The system would also deal with solving multiple simultaneous equations. Text is no longer just text, it’s “active” text. Looking at it another way, “widgets” are integrated with the text, making creation of an interactive user interface as easy as typing a sentence. Since behaviour is implied by context, the Scrubbable Calculator seamlessly provides a layer of interactive mathematical functionality without the user needing to explicitly invoke a separate “math tool” (calculator).

This reminded me of an idea I had a few years ago for a live coding user interface. (Live coding is a performance practice where audio-visual art is coded on the fly in performance.) The as-yet unimplemented idea was for a live coding text editor where typing something like:

oscillator.setFrequency( Slider() );

Would be immediately expanded in-line, as soon as you type it, to:

oscillator.setFrequency( Slider() );

When the code is executed, the slider would be automatically linked to the live audio algorithm thus allowing interactive manipulation without needing to explicitly construct or code a GUI — while at the same time retaining the direct visual linkage between the slider-manipulated parameter and the code it’s related to (an important factor for audience comprehension in live coding). Personally I think this would be an improvement over the current situation in many live coding environments where you need to re-execute lines of text to change variable values — something I sometimes see when watching Andrew Sorensen’s performances with impromptu for example. Of course, similar dynamic linkage in a live coding editor could be done with scrubbable numbers or other types of widgets instead of sliders.

Bret’s scrubbable example doesn’t end there though. He also illustrates dynamic creation of constraints (relations) between values by dragging linking lines, a bit like visually authoring spreadsheet equations or routing patch-cords embedded in free-form text. And this, I think, points to the overall direction that Bret is working on: tools for making mathematical systems less abstract, more dynamic, visual, situated, understandable and usable.

More expression of relationships between representations please

One thing that struck me about Bret’s state variable filter explorable example is that it presents a disjoint set of visual representations and formulae. The user is left to deduce (or perhaps assumed to already understand?) the relationships between the various representations. A formula for the transfer function is given, along with a pole-zero plot and the frequency response, but there is no built-in way to explore the mathematical relationships between these pictures. The relationship between the frequency response and the pole-zero plot is not illustrated. It isn’t at all clear that the pole locations on the z-plane plot are are the roots of the transfer function polynomial. Not to mention that casual readers may not even realise that the variable z is a complex number. To me this seems like an “interactive representation jumble” rather than a coherent, explorable presentation of what is in-fact a crystaline structure of interrelated mathematical meanings.

To be fair, Bret calls the SVF diagrams an “Explorable Example” — it serves to provide an interactive illustration within a larger textual narrative rather than as a self contained “Explorable Explanation.” That said, completely self-explanatory explorabile illustration is desirable. The interested reader should be able to examine everything in-situ, potentially down to the workings of the number system. If the interface employs multiple visual representations their relationships should be clear. Visual linkage could be provided with animation or additional elements showing the cross-relationships between points on the various graphs (as in the Predator/Prey example). Mathematical relationships could be expanded (click to zoom) or contracted to show the big picture. Clicking on a complex polynomial could visualise each component and expand to provide further detail. Clicking on any node in the data flow graph could show the transfer function for that node. Et cetera.

My impression is that Bret is working on these ideas. The link-creation system in the Scrubbable Calculator explores explicit creation of relationships between values, and the Predator/Prey video includes clear indicators on each graph to show how data points relate to those on each of the other graphs. There are other clues in the texts.

I’ve found Bret’s Explorable Explanations fascinating and inspiring. Do go check them out.