Programmable Ink Lab Notes
title
Ingredients from Alex's Studies
dated
June 2023
author
Alex Warth

This page summarizes a bunch of studies that I’ve done in the past few months. Rather than discuss each study separately, I group them into “ingredients” (each explored in one or more studies) that may turn out to be useful for upcoming projects.

Relax-PK: A New Relaxation-Based Evaluator for Crosscut-like Languages

I thought it would be useful for us to have a language that feels similar to Crosscut, but in which computation can flow in any direction. Applications I had in mind for this language include specifying relationships among multiple objects and/or properties (sketchy CAD is a special case of this) and back-of-the-envelope calculations.

My evaluator is based on van Overveld-style relaxation. To make it more performant and stable, I added a step called “propagation of knowns” (PKs) at the start of each frame. The idea is to evaluate any operator whose inputs are all known, e.g., points to which a fixed coordinate constraint has been applied, and treat the results of those operators as known values as well. We do this until there are no more known values to be found. Once the PKs step is done, we iterate on relaxation until either the total error in the system is negligible, or 1/60th of a second has elapsed.

This worked out even better than I expected. Propagation of knowns made it possible for me to construct models that felt rigid in the right places, unlike other systems that are based on relaxation. Performance was also decent – I didn’t go out of my way to optimize the evaluator, and identified several things we could do to make it faster.

My initial implementation of the evaluator included objects and constraints that were inspired by Crosscut: there were points, variables, adders, multipliers, property pickers, even a time generator. It also included default visualization for the built-in constraints, which were helpful for understanding the model that was being constructed.

After that, I did several follow-up studies in which I added other interesting features. I’ve included some noteworthy ones below.

But first, for more information on the evaluator, see:

Spreadsheet Cells

Combining Multiple Evaluation Mechanics #1: Bidirectional Spreadsheet Cells in Relax-PK

Constructing large, complex expressions in Crosscut can be cumbersome. There are also times when the user wants a certain operation that is not yet available in the system. I added “spreadsheet cells” as a solution to both of these problems. The idea is to allow the user to enter an expression symbolically (interactions and affordances TBD) and turn it into a special kind of variable object that computes its own value based on the inputs that are wired up to it.

Spreadsheet cells use relaxation just like the rest of the system, which makes them “omnidirectional” just like the built-in operators. E.g., if the user uses the expression “x * x” in a spreadsheet cell and wires an equality constraint to the value of that cell that forces it to be 9, the system will run the computation backwards and make the input to that spreadsheet cell be 3.

Avoiding unpredictable behavior when the user tries to set the result of the formula to an impossible value was the trickiest part of this study. The solution that I arrived at was to (1) detect when the cell’s desired output value is “impossible” (i.e., not in the range of the function defined by the user’s formula), and (2) not change anything when that is the case. The video below illustrates this scenario. It uses a spreadsheet cell to make the red point’s y coordinate equal to the square of the blue point’s x coordinate. Note that when the user drags the red point into the region where ys are negative (around 0:12), the spreadsheet cell stops updating and so the blue point stops moving. Once the user drags the red point into the region where ys are nonnegative (around 0:14), the blue point starts to move once again.

(This is also a neat solution to a problem that @Marcel Goethals and I have seen in angle constraints, which tend to become unstable when the lines involved get very short. We could modify the implementation of that constraint to make it a no-op when either of the lines has a length that is less than some small epsilon.)

Linkages

Experiments w/ Linkages

One of the ways in which I stress-tested the evaluator was to construct linkages using constraints, and see if I could use them to build puppet-like structures that I could manipulate performantly. Results were decent, as you can see in the video below. (See that study’s write-up for details.)

Encapsulation

Encapsulation / “Objects” Part 1

TODO

Visualizations to Help Understand Unstable Constructions

Document (and get a better understanding of) unstable constructions

In this study, I added a visualization of the “deltas” that are generated by the constraints – they are rendered in the UI as arrows that are pushing/pulling the points in various directions. I used a different color for each constraint type to enable the user to see what’s going on. This helped me get a better understanding of why certain (somewhat contrived) constructions were unstable.

In order to be truly usable, we need to make these visualizations more general (e.g., how do we visualize a delta that is being applied to a variable?) and combine them with other features such as time travel (that way the user can rewind, watch the model become unstable in slow motion, pause it at interesting moments, etc.).

Ethereal Objects, Constraints, Etc.

For some applications, it’s useful to associate additional state with objects that the user created. For example, suppose we want create a field that adds a “shadow” to every point and line that is inside it; the shadow itself is made up of points and lines. Note that the objects that are part of the shadow are second-class, in way: it only makes sense for them to exist while the original objects that they’re associated with are still around. If the user decides to delete some of them, or if they are no longer in the field, the corresponding shadow objects should disappear.

I worked on a couple of studies to try to come up with a mechanism that would make this kind of state management straightforward, and figure out how that mechanism should work. Ideally, the same mechanism could be used with fields, spatial queries, and spreads.

@Marcel Goethals provided a more interesting example, where the system would recognize a certain configuration of objects as a slider, and associate whatever additional state to those objects in order to make the slider behavior work.

The slider field (shown in the video above) associates several objects with the point that becomes the “thumb” of the slider:

Note that when I drag the thumb point far enough away from the slider (at 0:18) it is no longer recognized as a thumb, and therefore temporarily loses all of its associated state: not only does the value go away, but so does the constraint that was keeping it on the slider.

For more information, see:

Maintenance of “Ethereal” Objects/Constraints

Maintenance of “Ethereal” Objects/Constraints (Part 2)

Managed Side Effects Considered Harmful

(a non-ingredient)

Consider the following variant of the slider field, in which the endpoints of the slider temporarily change color, while they are recognized as part of a slider:

This version of the slider is arguably more polished-looking than the previous one. The endpoints here are almost invisible — you can see them just enough to know where to drag if you want to move the slider, but they don’t interfere visually with the slider’s illumination.

I spent some time thinking about how to support scoped side effects like the change in the endpoints’ color in this example, but the more I thought about it, the more I realized that it was ultimately a bad idea.

One problem is that it’s possible for an object to be in more than one field or spatial query at the same time, and when that happens, those fields / spatial queries may have conflicting wishes for the value of the same property of that object. Should one of them “win” and the others “lose”? If so, how should the system decide which one wins? And what if there are other properties that are also being modified, but without conflict? Is it OK that the object won’t be what any of the conflicting fields / spatial queries wanted? (If the properties in question are all visual, like color, the worst that could happen is that the object will look strange, But what if they are more semantically meaningful, and the combination of the changes throw the object into an invalid state?)

Another tricky question arises if the user decides to change the color of one of the endpoints of a slider while it was in its temporary white state. What should happen once that point exits the slider field? Should it return to its original color, or should it continue being whatever color the user changed it to, while it was one of the endpoints of a slider?

These issues are very messy, and I feel we’re much better off sidestepping them. We can do that by not managing side effects.

For more information, see:

Managed side effects #1

An Architecture that Makes Notebook Functionality a Top Priority

I want us to want to use our augmented notebook whenever we find ourselves reaching for pen and paper. In order for this to happen, it’s important that we seldom experience delays or noticeable latency when drawing and erasing strokes / creating a new page / flipping to a different page / etc. – a.k.a. notebook functionality –  even when there are lots of computations and dynamic behaviors running on the page. But as @James Lindenbaum points out, it’s also important for computations to update in (near) real-time: without that it would be difficult for the user to get an intuitive understanding of the models they’re interacting with when they manipulate and move objects around.

This is not a solved problem, and it’s not just a matter of engineering. I’ve started thinking about and sketching an architecture that could be used to build a system with these properties.

For more information, see:

Commentary:

A Model of “Well-Formed” Circuits in Crosscut

In Crosscut, the user can create circuits that don’t make sense / don’t work. As an example, suppose you have an “add” operator whose top and right inputs are both wired to the same variable. A change coming in from the left input makes the operator want to change the value of the variable (because it’s connected to the operator’s right input) but that’s not allowed (because it’s also connected to the operator’s top input). In this case the problem is easy to spot, but larger circuits can have similar problems (like loops) that are not so obvious.

I came up with a set of well-formedness checks that the system can run on every change – like a typechecker – to ensure that the user only constructs well-behaved circuits. This sort of thing can make for a better user experience and avoid a lot of confusion.

For more information, see:

Commentary:

Rendering Dynamic Ink w/ Ploma

I experimented with Dan Amelang’s Ploma, a library that creates very realistic-looking renderings of BIC pen on Moleskine notebook (Alan Kay’s favorite):

Static renderings of ink appear to perform pretty well, but animating even a single stroke currently requires re-rendering all of the strokes on the page. I spoke with Dan about this, and he told me that this doesn’t have to be the case but fixing it would require making changes to the library.

For more information, see: