Programmable Ink Lab Notes
title
Ink Studies Report
dated
June 2023
author
Curated by Marcel Goethals

This is a summary of what we think are the most salient studies that were done during this period.

Analytic constraints with term-rewriting

Try to implement a better dataflow system using e-graphs for term rewriting. Findings here:

Complex Constraints with Dataflow

Qualitative Spatial Reasoning

A good amount of time was spent thinking about different ways to deal with spatial queries. Roughly, there are two approaches that I tried:

Global Query System

Within the Ink group, we usually call this area “Spatial Queries”. I think there really are two distinct but interconnected parts to a system that does this kind of thing.

Qualitative Spatial Relations #1

“Global Queries” is an approach that we’ve tried in Untangle, and is somewhat analogous to the potluck style matching. Basically, the Query system is looking at a page with you, and trying to interpret what is going on in real time.

In a first study I explored the idea that we could define spatial queries in a way that is very similar to PEGs. In this model a spatial query is a composable higher order function that takes other spatial queries as it’s input. For example: A to the right of B takes two other queries as its input.

This query says: Find boxes with Xs inside with text to the left. Underneath that, there could maybe be more text.

These queries use bounding box geometry as their shared interface. This was a particularly interesting insight from this study: We can reify almost all qualitative spatial relationships by drawing boxes.

The "to the right" query can be implemented in the abstract because it's just operating on basic geometry, no need to know any details.

End-user Editing of Global Queries

Visual UI for Qualitative Spatial Reasoning

This idea of a parser is compelling in the abstract, but how would an end user actually construct these kinds of queries? This second study explores a UI for constructing this kind of query in context.

I build here on the idea that most queries can be simplified to just relationships between boxes. There is an editor window that the user can use to construct queries in a diagrammatic way, using concrete examples from the page.

There are no built in notions like “inside” or “on top”. The system only allows you to add a second box in relation to a previous match. Which allows the user to make a wide variety of matches by direct manipulation.

I did a second study to figure out the question of how to deal with the results of these queries. This is where I developed the idea of “truth tables” to construct compound queries. In the example below we are querying for checkboxes containing either a cross or a checkmark, and we can specify a projection for either.

As concrete as possible query system

As concrete as possible spatial relations #1

As concrete as possible spatial relations #2

By this point I started to see that in trying to make a UI for the query system, the most logical direction was to make things more concrete.

Secondly, the global pattern matching with ink has an obvious downside: We should assume that the parser & symbol matching will never work 100% reliably. In this study I took the opposite approach, trying to make spatial reasoning as concrete as possible.

I identified four different kinds of functionality that could be solved using a global spatial querying system, that we could reify into four distinct primitives:

I did a few smaller studies inspecting these aspects individually:

Instantiation

instantiation can be done using a pseudo mode and gesture recognition of components.

What is interesting in this little study that I don’t think is obvious from the video is how fluid it felt. I drew this little schematic in 30 seconds, roughly the amount of time it would take me to draw it on the back of the envelope, but because I’m instantiating these components it could be a live model, that I could use to calculate the value of a resistor.

We could instantiate number objects and expressions in the same way:

Modal properties & Connectors

In this study I’ve reified the bounding boxes from the global query system, and turned them into concrete objects on the canvas like everything else.

What is interesting here is that we’re making a conceptual shift where we’re treating the thing that is inside the box as a value. So it’s much like filling in an input field with text.

We can wire up values using crosscut style meta-ink to a truth table, which is also concrete on the page.

This could allow the user to construct a component for a boolean network in a concrete way.

Repetition

A final primitive that global pattern matching does is: looking for “rows” or “columns”.

This little study explores how to deal with that concept in a concrete way, by detecting that two components were instantiated underneath each other, and then surfacing a “stack” affordance.

Concrete Repetition & Collections

Concrete spreads, repetition & collections

There is a lot of overlap between pattern-matching on rows and creating elements through repetition. This study tries to extend the idea of “Fields” from the previous study, and explores what it would look like to do repetition in a concrete way.

There are two important ideas explored in this study

  1. Turning the iteration number into an affordance on the canvas. Each iteration has its own “scope” that can be focussed on independently. Each scope holds references to previous elements in the collection — this is how you get concrete & declarative iteration “for free”.
  2. Snapping to elements inside the scope allows the user to construct elements that auto-repeat.

Using this process we can make this spiral:

I developed these ideas into a small code-prototype that implements a version of this snapping behaviour.

A useful insight that arises from this prototype is the notion of indirect manipulation affordances. Manipulating a point or an angle doesn’t directly modify its internal property. Instead each concrete element holds a reference the function that produced the concrete element.

So when updating the angle in the star shape, all internal angles update, because they’re all instances of the same repeated angle.

I think this notion of concrete repeaters is a pretty useful concept when we relate it to the example of fields that can be wired together from the previous study.

If we combine concrete repeaters with expressions, suddenly we have a way of implementing array functions in userspace. Here’s an example for aggregators like SUM:

If we approach spatial queries in a similar way, where we can snap to instances that match the query, we have a very concrete way of dealing with results of queries, because the results are just the elements that were found, but with additional affordances.

→ An open question here is how to deal wit disambiguation when snapping to points.

Insights & Hypotheses

Semi-structured inking could feel more fluid, not less.

A big assumption (for me, and I think for the ink-track in general) is, that premature structure is bad, and that therefore we should ideally have a system where everything is “hand drawn”. We can see this approach in inkbase and untangle:

In Untangle one of the core insights was that while we imagined hand-drawn input would enable drawing symbols and shapes that mapped closely to a particular problem domain. In practice, most problem representations looked like tables

In Atelier/Habitat we tried a particular flavour of hybrid ink. Where all ink has both informal and formal properties. In hindsight, I would argue, this approach misses the mark. “Atelier ink” is a weak version of both freeform and formal ink.

These studies point towards is a “barbell” strategy instead, where the formal and freeform worlds are separate, but have some common ground in the middle where they interact

Bi-directionality is maybe not as important as we thought

(And if we really need it, it’s probably fine to turn it into a specific interaction)