title

Pen-driven symbolic programming

dated

Fall 2023

An important way of expressing relationships that aren’t easily expressed visually is by using formulas. We want a way of inputting formulas that is as “pen-driven” as possible. While the nodes & wires approach we used in Crosscut feels “like drawing” in a limited sense, it also feels somewhat frustrating to have to construct basic mathematical formulae using elaborate diagrams. Especially since most people have already spent years of their life learning math notation in school, it seems unnecessary to come up with an entirely different notation system.

Marcel spent some time designing a simple visual language that tries to combine some of the advantages of nodes & wires and text-based approaches.

The idea here is to allow the user to express things through both wires and positional semantics. For example, connecting two nodes with a wire makes them equal, but so does snapping them together. This simple idea gives us quite an expressive set of physics.

One of the open questions here is how to input basic formulas in a way that stays close to the pen. An obvious answer here is to pop up a keyboard, similar to what we did in Habitat. But one variant that was under-explored so far was the use of simple recognition of pen gestures to create things like numbers and math operations. One of our ideas was to rely on recognition as little as possible, by allowing labels to remain hand-written rather than converting them to text/strings.

We spent some time in the middle weeks of phase 2 working on a math formula editor that leveraged the $Q handwriting recognizer. We’d hoped that by putting some limits around how numbers and symbols could be written (e.g., creating an explicit grid of cells each allowing only a single character) we could achieve something good enough for prototyping without needing to resort to a hardware or on-screen keyboard. This approach was a mixed success. Our formula editor functioned well enough to allow us to construct a few compelling demos where formulae and constraints interacted pleasingly.

But we also discovered that building a robust formula editor with reliable handwriting recognition would be a large project unto itself. For phase 3, we plan to figure out some sort of temporary “designer pink” alternative input method for formulae, so that we can focus on the semantics of the system and how formulae will participate in that.