Reflections on Bret Victor’s “Explorable Explanations”

I’ve been reading and thinking about maths quite a bit lately. In the process I happened upon Bret Victor’s killmath posts on interactive dynamic systems, explorable explanations and usable math. Some of the killmath examples really got me thinking. I’ve actually started coding a few experiments around this, but for now I just want to summarise some reflections and comments I have about Bret’s research. There is a lot more to Bret’s site than I touch on here, so I really recommend you go check it out if you havn’t already.

Explorable (Visual) Explanations

To give a feel for the material: one of Bret’s posts in Explorable Explanations gives an interactive example of a digital state variable filter (it’s not actually Chamberlin’s topology as I had previously tweeted). The example presents different graphical and symbolic representations of the filter and allows you to manipulate the filter’s parameters and hear the result. One of Bret’s videos demonstrates an interactive interface for exploring the Predator/Prey dynamical system with multiple cross-linked interactive visualisations. I discuss both of these examples below.

If you’ve read Edward Tufte’s book Visual Explanations — images and quantities, evidence and narrative you’ll have an idea of some of the concepts and considerations involved. Pictures, diagrams, sounds, and dynamic time-based demonstrations can all tell stories in ways that numbers and formulae can’t — potentially allowing us to gain new insights, solve problems and draw conclusions that are not accessible (to the uninitiated, if at all) by simply looking at mathematical formulae or unstructured data. Explorable Explanations extend visual explanation with interactivity allowing the reader/user to engage, explore, play, and listen, in order to gain a deeper understanding of a system’s dynamics and behaviour under varying conditions.

Of course, animations, simulations and interactive examples in the form of Java applets and Flash interactives have been circulating for years. To give a random example: sorting algorithm visualisation applets from the field of algorithm visualisation. Often though these applets are demonstrative rather than truly interactive and explorable. Closer to the mark is a recent site from the U.K. Department of Energy and Climate Change: My2050, a Flash interactive that allows users to experiment with a parametric model of energy generation and consumption, lifestyle choices, and modes of transport to design their future world. By moving sliders on the screen, users explore the environmental effects of their choices while being challenged to achieve a workable set of parameters. It’s a pretty neat web site that illuminates some of the environmental trade-offs we face.Click here to try it

To me there is something implied by Bret Victor’s work that isn’t present in DECC’s My2050 or in simple illustrative Java applets. DECC’s My2050 site supports developing intuition about the relationships between energy consumption, production and carbon footprint, however the underlying mathematical model and data set are not visible. In contrast, it seems to me that Bret’s examples are not just about increasing user intuition about system outcomes. They also support increased user awareness and expression of explicit models of system structure and dynamics. For example, by simultaneously showing the various representations of the state variable filter, or by showing multiple overlaid trajectories in the Predator/Prey model. I think this is a really interesting direction, but as I discuss later, the mechanisms for illustrating structure and inter-relationship deserve further development.

Scrubbable numbers

This brings me to Bret’s recent post on the Scrubbing Calculator. One facet of the Scrubbing Calculator that caught my attention was scrubbable numbers — numbers that you can click and drag on to change their values (not unlike a spin box) to support interaction with typed mathematical expressions. Changing one number changes the others according to their context in the surrounding text. Try clicking and dragging up and down on the numbers in the sentence below and you’ll see what I mean:

1 plus 2 equals 3

The point here is that this functionality would be available as you type, not just in a pre-written document like this one. The system would also deal with solving multiple simultaneous equations. Text is no longer just text, it’s “active” text. Looking at it another way, “widgets” are integrated with the text, making creation of an interactive user interface as easy as typing a sentence. Since behaviour is implied by context, the Scrubbable Calculator seamlessly provides a layer of interactive mathematical functionality without the user needing to explicitly invoke a separate “math tool” (calculator).

This reminded me of an idea I had a few years ago for a live coding user interface. (Live coding is a performance practice where audio-visual art is coded on the fly in performance.) The as-yet unimplemented idea was for a live coding text editor where typing something like:

oscillator.setFrequency( Slider() );

Would be immediately expanded in-line, as soon as you type it, to:

oscillator.setFrequency( Slider() );

When the code is executed, the slider would be automatically linked to the live audio algorithm thus allowing interactive manipulation without needing to explicitly construct or code a GUI — while at the same time retaining the direct visual linkage between the slider-manipulated parameter and the code it’s related to (an important factor for audience comprehension in live coding). Personally I think this would be an improvement over the current situation in many live coding environments where you need to re-execute lines of text to change variable values — something I sometimes see when watching Andrew Sorensen’s performances with impromptu for example. Of course, similar dynamic linkage in a live coding editor could be done with scrubbable numbers or other types of widgets instead of sliders.

Bret’s scrubbable example doesn’t end there though. He also illustrates dynamic creation of constraints (relations) between values by dragging linking lines, a bit like visually authoring spreadsheet equations or routing patch-cords embedded in free-form text. And this, I think, points to the overall direction that Bret is working on: tools for making mathematical systems less abstract, more dynamic, visual, situated, understandable and usable.

More expression of relationships between representations please

One thing that struck me about Bret’s state variable filter explorable example is that it presents a disjoint set of visual representations and formulae. The user is left to deduce (or perhaps assumed to already understand?) the relationships between the various representations. A formula for the transfer function is given, along with a pole-zero plot and the frequency response, but there is no built-in way to explore the mathematical relationships between these pictures. The relationship between the frequency response and the pole-zero plot is not illustrated. It isn’t at all clear that the pole locations on the z-plane plot are are the roots of the transfer function polynomial. Not to mention that casual readers may not even realise that the variable z is a complex number. To me this seems like an “interactive representation jumble” rather than a coherent, explorable presentation of what is in-fact a crystaline structure of interrelated mathematical meanings.

To be fair, Bret calls the SVF diagrams an “Explorable Example” — it serves to provide an interactive illustration within a larger textual narrative rather than as a self contained “Explorable Explanation.” That said, completely self-explanatory explorabile illustration is desirable. The interested reader should be able to examine everything in-situ, potentially down to the workings of the number system. If the interface employs multiple visual representations their relationships should be clear. Visual linkage could be provided with animation or additional elements showing the cross-relationships between points on the various graphs (as in the Predator/Prey example). Mathematical relationships could be expanded (click to zoom) or contracted to show the big picture. Clicking on a complex polynomial could visualise each component and expand to provide further detail. Clicking on any node in the data flow graph could show the transfer function for that node. Et cetera.

My impression is that Bret is working on these ideas. The link-creation system in the Scrubbable Calculator explores explicit creation of relationships between values, and the Predator/Prey video includes clear indicators on each graph to show how data points relate to those on each of the other graphs. There are other clues in the texts.

I’ve found Bret’s Explorable Explanations fascinating and inspiring. Do go check them out.

SuperCollider internals book chapter

In case you haven’t heard, the SuperCollider Book is now available. Congratulations to everyone involved in bringing this to press.

The publisher has been kind enough to allow my chapter on SuperCollider’s server internals to be posted online as a preview chapter here. I think this is great, because understanding SuperCollider’s server internals will be of interest not just to SuperCollider developers and users, but to anyone involved in creating dynamic real-time audio software.

I first became interested in the internal workings of SuperCollider during the early days of AudioMulch development. At that time, SuperCollider was a closed-source project and information about how it was implemented was fairly scarce. Luckily, SuperCollider’s creator James McCartney was active on the music-dsp mailing list and was open to answering questions and explaining how things worked under the hood. Later, after SC became an open source project I performed initial work on porting scsynth to Windows, and in the process had a good look at what was going on in there. One thing that really interested me was how SuperCollider supported re-patching the audio signal processing graph in real-time without glitches or interruptions to the audio stream. The answer turned out to be simple and elegant.

Class diagram of significant scsynth domain entities -- one of the class diagrams from the book chapter.

Real-time dynamic audio graph manipulation

In SuperCollider, the dynamic audio processing graph (the graph of Nodes) is a linked data structure that is traversed at each audio buffer period to compute audio data. Modifications to the dynamic Node graph are achieved by posting asynchronous commands to the real-time audio processing thread via a lock-free queue. These commands are interpreted at the start of each buffer period. They cause the Node graph to be modified, changing the signal flow and/or adding and deleting processing Nodes as necessary. When required, dynamic memory allocation is performed in the audio thread using a special-purpose thread-specific real-time memory allocator. A general mechanism, also based on lock-free command queues, is provided to execute non-real-time operations (such as file i/o) asynchronously in a separate thread.

Precompiled Graphs and Unit calculation functions

In a slightly confusing case of overloaded terminology, scsynth defines a Node subclass called Graph, which evaluates a precompiled schedule of Units (unit generators). That is: scsynth has a dynamic graph of Nodes, some of which are Graphs whose purpose is to evaluate precompiled graphs of Units. Got it? The diagram above from the book chapter should help.

Allocating and evaluating Graph objects is relatively efficient because Unit instances are stored sequentially in a single memory block, and a Graph’s evaluation loop involves calling each Unit’s mCalc calculation function pointer in sequence. One interesting by-product of this design is that a Unit can change its calculation function pointer at runtime — this can be used to implement a Unit with different states. For example, an envelope unit generator whose final state returns a constant could set its mCalc member to a function that simply returns the constant. Thus less conditional logic need be executed for each Unit at each time step.

These mechanisms are two of the many aspects of the scsynth implementation described in the full book chapter available here. I cover the implementation and usage of these mechanisms in detail, along with the concurrency structure of scsynth including threading and inter-thread communication. The chapter is illustrated with a number of class and sequence diagrams. I encourage you to check it out, and of course, to get the book, which is full of insights and examples of how people have built whole artistic worlds using SuperCollider.