The Periodic Table of the Interactions

Nearly 50 years after the personal computer revolution, our ways of interacting with and within computational mediums, the actual atomic gestures like clicking, scrolling, dragging-and-dropping, have barley evolved. As Alan Kay, whose work at Xerox PARC instigated that revolution more than anyone else, put it, "The computer revolution hasn't happened yet."

Innovations like Apple's MultiTouch and pseudo-spatial virtual reality mechanics are progress, but by-and-large we remain trapped within a web of very low-common-denominator gestures, unable to make use of the vast majority of our physical, sensory, mental, even emotional facilities. How did this bottleneck of interaction come to exist, and what can account for its tremendous staying power?

Did we simply get graphical user interfaces right the first time, and to keep pushing the envelope is to make an error in the vein of Chesterson's Fence? On the contrary, what new affordances might we implement that raise our bedrock abstractions to new heights, ones that trade discoverability and familiarity for scope and efficacy?

To reference computer history once more, would the contemporary version of "dealing lightning with both hands" look like?

¶ i

A variant of the “Literally no one...” meme I came across recently[1]:

No one: ... literally no one: ... WWII: Computers.

I’ll offer up a snowclone:

No one: ... literally no one: ... 80s white-collar workers: Modern Computing Experience.

1
Try as I might, I can't seem to find the original Tweet containing this meme, but will link it here should it ever turn up.

Fig. — the snowclone, typeset live

Much like how the modern formula for indoor climate, for assembling bodies and buildings together, was born of an obsession over a few metrics and fewer psychometric profiles, in a furious, homogenizing loop of defining comfort ever more in terms of itself [1] and that, like a blitzkrieg of steamrollers, ran roughshod over a previously diverse, highly dimensional design space, [2] the modern formula for information work, for assembling minds and machines together, was derived from the ways of the ideally-conscientious 80s paper pusher, a period piece preserved in silicon rather than amber, forging the socio-technical contract of getting what we see and seeing what we get: the files, the folders, the desktops; the windows, icons, menus, pointers; that are now self-evident as computing itself, such that venturing into hitherto unexplored design spaces, stretching along new dimensions, has become a game of Jenga most all deem impractical to play. Inventing fire is monumental; playing with fire thereafter is folly.

1
"Sick Building Syndrome and the Problem of Uncertainty : Environmental Politics, Technoscience, and Women Workers," M. Murphy, 2006
2
"Modern Architecture and Climate: Design before Air Conditioning", Daniel A. Barber, 2020

iii

That’s what "work looks like"[1], we say of the user experience innovations came out of Xerox PARC in the 70s and 80s. They have become the ontological scaffolding for each and every iota of “knowledge work” produced since. Unlike the more gradual inroads that industrialization made into the home, gradually over decades, haphazardly, authored by hundreds if not thousands of designers directly and hundreds of thousands of inventors, business owners, and shareholders indirectly, the work processes and technological systems[2] of what we now know as digital knowledge work came almost exclusively out of PARC, ex nihlo.

1
"Office, Messaging and Verbs," Ben Evans, 2015
2
Both terms from the introduction to Ruth Cowan's "More Work For Mother"

As with all true HCI innovations, it started with hardware. A key memo penned in 1972 by project leader Charles “Chuck” Thacker[1] outlined at a high level the technical specifications for the Xerox Alto, a first-of-its-kind machine meant to make computing personal.A screen that resembled a sheet of paper, oriented in portrait mode, to sit on top of your desk, and a magic thirty-inch-by-thirty-inch box that sat underneath, animating the whole apparatus all by itself⏤look, ma, no terminals! A keyboard, standard fare, and, much more exotically, a mouse,to be used in conjunction with the shiny, bitmapped 606-by-808 pixel screen to bring a new kind of user experience to the masses: one mediated not just by text, but graphics.

1
Sourced from "The Dream Machine," M. Mitchell Waldrop, 2018

Fig. — the Xerox Alto assembles itself, 606 × 808

On its screen, the “arcane abstraction of software” was transformed into “something visible, tangible, almost tactile.”[1] Users could manipulate things directly, making use of the on-screen elements’ pseudo-physical properties to communicate their intentions, rather than invoking abstract, textual functions to perform the same action. This new form of interaction enabled the engineers at PARC, almost as if they were the first settlers of a new town staking out main roads and landmarks, to establish a new way of life for millions thereafter, consisting of windows, icons, menus, pointers, What-You-See-Is-What-You-Get editors,and so on.[2] No other fundamental arrangement of computing experience has since supplanted it, no subsequent ontological remodeling has taken place, save for perhaps Apple’s introduction of the multi-touch equipped smartphone some thirty years later.

1
From "The Dream Machine," p.365
2
From "The Dream Machine," p.368

Fig. — windows, icons, menus, pointers: each demonstrating itself

But PARC’s eminence lay less in the unbounded creativity of its inventions than the measured practicality of its solutions. Rather than suffer the fate of yet another technological Cassandra, PARC had the wherewithal to shape the world to its vision, avoiding the fate of so many others, whose clever inventions await popularity or as consolation, history, either in vain, to declare them visionary enough not to be bothered with actually building anything. The constraints PARC faced were daunting; the paltry computing power of the day alone should have thwarted them. Much more a concern, however, was daunting prospect of creating a new archetype from scratch: that of the computer “user.” For better or for worse, PARC succeeded in creating this inaugural, and still-dominant, archetypical user for computing.

vii

Perhaps it was a good amount of canniness on their part, or just their lucky historical break to be owned by Xerox, but somehow PARC honed in on a demographic that had both tremendous upstart potential and saying power: white-collar office workers. The daily tasks of this species, heavy on the documents, were highly amenable to their inventions of text editing, window viewing, and file organizing. While optimizing for this readily-apparent use case might have been what enabled PARC’s innovations to get off the ground in the first place, it’s worth questioning why this archetype has had such staying power ever since. Did PARC really just get it right the first time? Or might it have been something of a victim of its own success, perpetuating up until the present day a mode of user experience for computers that has gone unquestioned only because it is the default?

viii

As HCI researcher Jakob Nielsen pointed out in 1996, hardly two decades after PARC, the constraints under which the Alto and its user interface principles were long outdated. Moore’s law had seen compute power increase several-hundred-fold. Users were no longer totally naive to interacting with the digital. Though the range of applications available had broadened somewhat, they still heavily indexed on the “knowledge worker” of the day. And the Internet, nascent as it still was, was lightyears beyond PARC’s ethernet in terms of making connectivity a first-class citizen of computing experience. Two and a half decades after Nielsen’s marvelous piece, in which he calls into question the fundamental nature of the palette of interactions birthed at PARC, we’re still clicking and dragging away, shuffling our windows to and fro, seeing what we get instead of what we might mean.

1
"The Anti-Mac Interface," Jakob Nielsen, 1996

In a similar vein, the purpose of the Periodic Table of the Interactions is to seek out new, alternative modes of dynamic computing experience. PARC represented one configuration, and by all lights it did a wonderful job at solving a lot of constraints elegantly. But there are other possible configurations, other resonances to be sought out, and by breaking down and laying out attributes of the interactions we’ve accumulated so far along various dimensions with the table, it is my hope that new and otherwise unimaginable modes of human-computer interaction should emerge. Understanding PARC through one of these attributes, dimensionality,is how I will conclude this opening section.

More than anything else, PARC represented an evolution in the dimensionality of computing experience. This property is listed in the top-right corner of each element on the Table for a very compelling reason: I believe it to be the most important aid in making sense of how the table’s interactions have shaped what we know as computation, and for orienting ourselves as we play at extrapolating via the table into new and uncharted territory. The dimensionality of interactions is an intuition pump that collapses affinities among elements of historical context, computer literacy, and the interpolation of computation into geometry into four points along a single continuum. The explicit thesis of this essay, and the implicit reasoning behind my creation of the table, is to first demonstrate how PARC, through the design and implementation of specific, tangible interactions, became impetus for moving human-computer interaction from a “one-dimensional”, or “1D”, plane of experience, to a “two-dimensional”, or “2D”, plane of experience; second, it is to demonstrate how some of PARC’s prescriptions have overstayed their welcome and prevented us from transitioning from the 2D realm to that of the “three-dimensional”, or “3D” one; and finally, explain why this may be a good thing, as the leap from 2D to 3D may not be possible, while one from 2D to the “fourth dimension” of user experience, or “4D”, which encompasses 3D as well, might be, although it will take as much creativity, fortitude, and luck as the PARC-led jump from 1D to 2D did, if not more.

Prior to PARC, computing was a one-dimensional, henceforth written as 1D, affair. Computing was controlled solely by inputs and outputs of text, or, even earlier, punchcards. I/O, I/O, I/O ad infinitum, exclusively the calculation strain of computation, rather than ordination, a distinction that I identified in my previous work Computation Becomes Geometry[1]. 1D was still dominant at the time when the foundations for mass computing were laid, and, as a result, it seems the de-facto base-state of computing today, ground zero for what is “really” is going on in a computational medium. High-level, textual programming languages are modern computing’s bedrock abstraction[2]; it contains what is essentially going on. Certainly it may be compiled down to, or explained in terms of, lower-level instructions such as assembly, binaries, etc., but in this case reduction does not entail construction.

1
https://jane.flowers/computation-becomes-geometry
2
"You Have Made Your Bedrock, Now Lie In It" (http://www.loper-os.org/?p=55)

xii

Meditate on the phrase “source code” a bit, and then consider how the idea of a “source” anything else is patently unimaginable. No source shapes, no source sounds, no source qualia. Or, rather, there are these, only they take place inside a fluent programmer’s mind before they are sent through the low-bandwidth channel of textual-logical representation. The limits of language are the limits of the computer’s world.[1] This low couplingis perhaps a necessary evil: although many have tried, abstract representation remains the superior format for understanding complex logical procedures and transformations. It’s worth keeping hope, however: at some point in history, textual representation is just geometry when you get down to it. Perhaps we can find a way to pull off a similar trick, only this time with dynamic media instead of static symbols.

1
Wittgenstein, of course.

xiii

Windows, Icons, Menus, Pointers. The idea that “what you get” can be programmatically represented by “what you see,” and nothing more. This has become the base state of computing experience. PARC didn’t invent two-dimensional (”2D”) computational experience, of course⏤its hallmark, the Graphical User Interface, was invented by Douglas Engelbart in the decade prior. But as covered above, they irrevocably established it as the embodiment of computation to the majority of us; even the wizards among us who CLI everything do so now from a “terminal” that’s housed in a in a 2D window on their desktop. And on the opposite end, many AR/VR experiences, which take place entirely in a “three-dimensional” environment, still feature 2D dropdown menus as control aids, in a sort of reverse-skeuomorphism.

xiv

The litmus test for a 2D interface is that you can still functionallyrun it from the command line, as a 1D compute. In other words, the graphics are aidsto understanding what’s really there, rather than being what’s really there.This is not at all to minimize the load they’re bearing⏤remember that this skeuomorph was what made computing “real” to so many⏤only that there’s a one-to-one correspondence between what one seesand what one gets.They are enumerable, unlike a 3D interface, which doesn’t reduce down from its geometrical state without some loss of information.When you drag and drop a file into a folder, the bitmapped graphical representation, all the x-y plotting going on, there’s exactly one thing happening to one entity, and all the rest’s incidental. 2D interfaces are clientsto some underlying**reality, hosted in 1D. “No-code” tools that are currently in vogue foreground this relationship, and as a result are effective to the extent that they index on the “no” more than they do the “code.”

An interface becomes three-dimensional,or 3D, when one can’t easily trace one-to-one relationships between what one can see, and what one gets. One or both may be innumerable, therefore rendering the relationship moot. John Palmer invokes a metaphor that helps illustrated this relationship between 2D interfaces and 3D:

A deck of cards consists of so many objects, that can be used in so many ways, amongst so many players, that there are almost infinite ways to use them. It's so complicated to think about designing an app to replicate the full functionality of a deck of cards, that still, no one has done it...If you start designing a deck of cards app as a traditional 2D app the way we think about them today, with 2D panes and menus and buttons, it becomes painfully difficult to think about.But what about in 3D? It seems that the only way to build an app that replicates the full functionality a deck of cards is to build a 3D simulation or game. Model the cards in 3D and put them on a 3D table. As long as you have controls for reaching out and picking up the cards, and moving them in space, you can do anything you can do in real life...Preserving the higher dimensionality makes it simpler and more intuitive, not less. [1]

1
"Spatial Interfaces" (https://darkblueheaven.com/spatialinterfaces/)

xvi

This sort of “catch-all” quality which reality so casually boasts still must be painstakingly engineered, of course, only at a lower level of abstraction, programmatically, handled by some sort of simulation engine. This quality is very interesting and often goes unremarked upon, and will be covered in detail in the final section Unfortunately, the “catch-all” quality cuts both ways: it is very permissive by dint of not being very specific. It is is why, despite advocates arguments to the contrary, fully 3D interfaces are unsuited for most “knowledge work,” and why the vast majority of interfaces that aren’t ends unto themselves (games) remain 2D. To extend Palmer’s card deck example, it’s true you don’t have to tediously program a specific “shuffle” randomization function because with a 3D simulation you can get it “for free” simply by allowing your users to shuffle the cards themselves. But in that case, why play with a real deck of cards in the first place? You’re merely slavishly recreating reality without harnessing the power of dynamic media.

xvii

Perhaps because conceiving of a best-of-both-worlds modality of human-interaction⏤what this essay is primarily concerned with, and what it dubs a four-dimensional (”4D”) interface⏤is very difficult? At the risk of introducing some amount of Ballardian baggage, a 4D interface is perhaps best thought of as the simulacrum to a 3D interface’s simulation. If it were, say, a chessboard instead of a deck of cards, you could play something like Git Chess[1], on top of the physical properties you get for free with 3D. In essence, there are some sort of extra-simulation information channels present in addition to the simulation itself, that fully allow for the dynamic medium of computing to come into play, while still maintaining the immediacy, tangibility, and that “getting things for free”[2] quality 3D simulation affords us. What I’m getting at with 4D interfaces is as ambitious and unprecedented as PARC’s establishment of the GUI, and throughout the rest of this essay with the help of the relationships and resonances the Periodic Table of the Elements allows us to see with full clarity, I hope to demonstrate the potential of 4D interfaces to shape the next generation of computing.

1
https://twitter.com/itsjaneflowers/status/1356431216078016513
2
https://twitter.com/search?q=from%3Arsnous%20%22for%20free%22&f=live

xviii

Before we fully flesh out a programme for 4D interfaces, however, it's vital that we understand in greater detail exactly what PARC accomplished with their shifting of mass computing from a 1D paradigm to a 2D one. In the next section, we'll examine a way of building user interfaces that PARC made dominant, first reviewing the reasons it was first implemented, the problems it often causes for user experience design, and finally considering a solution that orients us away from the pitfalls of pure 3D simulation and towards a new reality where 4D interfaces may interface with humans at an entirely unprecedented level.

A Study In Skeuomorphism

¶ i

I was perhaps unkind in comparing the work accomplished at PARC to the high-modernist exercise that created modern climate control; those at PARC were humanists to the core, and their work reflected this disposition. Rather, it’s probably best to think of what Xerox PARC gave us as the end-user-experience-equivalent of Von Neumann’s eponymous architecture: an arranged marriage, attributable to the blind matchmaker of history, where by all lights the parties involved would’ve married each other unforced. We may opine from time to time about our possible lives with others, who might possess qualities whose novelty, rather than their saliency, may seduce us for a time, but we know better than to indulge in counterfactuals; we’re thankful for what we were handed, that there was never the opportunity to the risk of skidding too far down the slippery slope of optimal stopping theory on our own.

1
A fun and totally irrelevant thing a different architecutre than Von Neumann Architecutre would allow us to do: "Another advantage is that that means that you cannot treat instructions as data, and THAT means that with the right computer design you literally cannot hack the machine through software. I heard about a voting machine that used this architecture and the only reason it was hacked was because of a faulty cartridge design allowing them to slip a jump instruction into a buffer overrun, which then they pieced together a turing-complete language from bits of functions (which took man months to do)." (https://stackoverflow.com/questions/1806490/what-are-some-examples-of-non-von-neumann-architectures)

One trait of our beloved we’ve learned to sometimes love, and sometimes merely live with, is a software architectural pattern known as “Model View Controller,” or MVC. As explained in the previous section, actively engaging with computation today is mostly a 2D affair, where the 2D interface is a skeuomorphic, a half-geometric and half-literate facsimile corresponding to some undergirding 1D reality. MVC’s near-monopoly on dictating how this correspondence is actually handled in real programs, by what means the interpolation is handled in real time, is one of the most interesting and understudied aspects of modern computing experience.

iii

Introduced by Trygve Reenskaug during a stint PARC in the late 70s, MVC was an attempt to establish a more robust framework for the design and implementation of end-user interfaces, seeking to extend the nascent practices already in-use at PARC. Up until Reenskaug introduced MVC, interfacial representation was tightly coupled with logical foundation via direct manipulation:

One of the great inventions of the Smalltalk group at Xerox Palo Alto Research Center (PARC) in the seventies was the idea that objects can be made visible on the computer screen so that the user can see and manipulate them directly. This makes the abstract computer data appear concrete and the underlying object model visible. The user can easily adjust his mental model to this computer model and operate on it with confidence. The well-designed direct manipulation object interface is intuitively obvious and therefore easy to learn for the uninitiated.[1]

1
"Naked Objects," Richard Pawson 2006, p.27

Fig. — the naked object vs the mediated one. Click a verb; time the pulse

A decade prior, the idea of object orientation was introduced to programming languages with Simula. Via Smalltalk, PARC merely allowed this innovation to bubble up through the surface of the GUI. With MVC, however, Reenskaug was arguing for a regression of sorts: sacrifice a little clarity in the name of scalability:

This strength of the direct manipulation object model is also its main weakness. Each object can only appear once on the screen and must always be presented in the same way to preserve the illusion of concreteness.This is insufficient for large and complex models where we need to view objects in different ways.[1]

1
"Naked Objects," Richard Pawson 2006, p.27

This makes intuitive sense: there are cases where direct manipulation can start to feel less empowering, and more tedious. Nielsen provides us with a simple example of this:

Using direct manipulation, users interact directly with objects in the interface. The archetypal example is to move a file from one directory to another by opening the original folder and using the mouse pointer to drag the file icon to the destination folder. This procedure works well for simple actions with a small number of objects, but as the number of actions or objects increases, direct manipulation quickly becomes repetitive drudgery. The dark side of a direct manipulation interface is that you have to directly manipulate everything. Instead of an executive who gives high-level instructions, the user is reduced to an assembly line worker who must carry out the same task over and over.

1
"The Anti-Mac Interface," Jakob Nielsen, 1996

Fig. — Nielsen's drudgery: one drag delights, a hundred indict

Rather than slavishly stick with “direct manipulation” out of principle, we should seek the solution that best serves the user. A purpose-built interface that allows us to batch tedious actions like the file-moving exercise above would at the very least be the much more expedient solution. MVC is best thought of as an abstraction of this design strategy. MVC isn’t a protocol or technical standard like TCP/IP; it’s a framework for organizing one’s thinking about how to architect an end-user computing experience. Broad strokes, it prescribes a separation of concerns between the content an experience mediates, from how it looks, from what it can do. Information, output, input. Model, viewer, controller. The abstraction is a compelling one; it appears to fall right along some shearing layers inherent in managing information with computers, judging by how many times it has been implicitly echoed or explicitly reinvented in the course of subsequent architecting.

vii

Riding on the coattails of the Smalltalk-stoked rise to prominence of object-oriented software development, the balance MVC strikes with regards to this fundamental tradeoff between control and capability has since become entrenched in the development of user interfaces today to the point of invisibility. Reenskaug, however, regrets introducing MVC into the water supply, sans any accompanying theoretical guidance:

The original MVC was later modified in Smalltalk-80 to become a technical solution that separated input, output and information. The most important participant in the original MVC architecture, the user’s mind, was somehow forgotten...[1]

1
"Naked Objects," Richard Pawson 2006, p.27

viii

Without an explanation as to whythe design of the interface was usurping some direct control from the user, without showing that this strategy was a means to an end, that of greater total user empowerment, it became an unquestionable architectural norm that users**are to have less control, well, just because:

The original version of MVC was never published. In my naïveté, I believed that everybody wanted to empower their users so that MVC was merely an obvious solution to a common problem. I was wrong. There are two traditions in the applications of computers; one is to employ the computer to empower its users, and the other is to apply the computer to control its users. I am sorry to say that the latter seems to be prevalent in mainstream computing today...[1]

1
"Naked Objects," Richard Pawson 2006, p.27

In his paper Naked Objects,an analysis of what went wrong with MVC and what steps might lead beyond it, Richard Pawson concurs with Reenskaug, and formalizes this tendency for the Controller to become inverted, resulting in the user being restricted rather than empowered, with the term “use-case controller.”

MVC thus encourages the extraction of certain behaviours of an entity object and placing them in separate structures. This is deliberate, and is based on the argument that the advantages outweigh the disadvantages...Controllers were originally defined (see above) as being concerned solely with managing input. However, this definition has been progressively distorted over the years so that the term Controller is now commonly defined as ‘governing the flow of control associated with a complete user task.’[1]

1
"Naked Objects," Richard Pawson 2006, p.27

Fig. — the use-case controller inflates; your cursor rides its rail

Why might this be a bad thing? After all, many times it is beneficial to the user themself to let the application take the wheel, especially if they are new and unfamiliar with it. Pawson seems to be more concerned with the purity of the user’s mental model remaining intact, rather than sharing Reenskaug’s concerns that control that is rightfully theirs is being usurped. Quoting another HCI researcher on this:

This extract-data-then-shove-it-elsewhere approach requires you to know too much about how the model-level objects are implemented. A system based on that approach cannot be called object-oriented: there’s too much data flowing around for the system to be maintainable.’[1]

1
"Naked Objects," Richard Pawson 2006, p.27

Pawson, echoing his early conclusion reiterates:

MVC thus encourages the extraction of certain behaviours of an entity object (i.e. those concerned with representing itself to the user) and placing them in separate structures. This is deliberate, and is based on the argument that the advantages outweigh the disadvantages. But in subtle ways, the MVC pattern encourages the extraction of other behaviours from the entity (Model) objects. It is very tempting to build small amounts of business logic into the Views, for example.

And, finally:

The challenge that this research presents is to find a technique that will combine the simplicity of the original objects-display-themselves approach with the flexibility offered by MVC, but in such a way that is does not encourage the insertion of use-case controllers between the domain model and the user interface.

1
"Naked Objects," Richard Pawson 2006, p.27

xii

The key maneuver in Pawson’s solution, dubbed his Naked Objectsframework, is extracting the control functionality of the use-case controller and imbuing it directly into the View, forming what he calls, variably a “naked object” or a “behaviorally complete object”:

The inventors of object-oriented programming conceived ‘objects’ as representations of the entities that model a chosen domain, with each object encapsulating the state of that entity (i.e. its attributes, including any relationships to other objects) together with the behaviors associated with that entity. In other words, objects were originally conceived as being ‘behaviourally- complete’.[1]

1
"Naked Objects," Richard Pawson 2006, p.27

xiii

If you’ve noticed, we’ve reached something of an impasse⏤with behaviorally-complete objects, we’re essentially back with Smalltalk’s direct manipulation scheme. Reading through the rest of Pawson’s paper, there’s not much in the way of grand solutions offered. Pawson’s framework does offer a few small tactical suggestions, in particular specific ways to improve upon these objects auto-generating their own interfaces and controllers based on their internal properties.

xiv

But he seems to unearth more problems than solutions, or at least solutions that would generalize. He points out that direct manipulation fails in cases where there is no object to manipulate, or “zero-parameter arguments,” and without a use-case controller in the picture, we’d be unable to perform these actions:

Certain required user actions are not obviously associated with any particular object instance. Examples include: creating a new instance, finding an existing instance, and creating a list of instances that match some criteria.[1]

Pawson points out that the the same goes for “multi-parameter objects,” where manipulating all the relevant objects at once is impossible. [2]

1
"Naked Objects," Richard Pawson 2006, p.27
2
"Naked Objects," Richard Pawson 2006, p.33

I believe there is another way. It will require a bit of rethinking of how we relate to software, as I believe the tradeoff outlined above is the result of the poor framing of what dynamic computational interfaces can do, and should be. I will show how using this new framing, together with the Table as a guide and sandbox, as a platform for creating new gestures and interactions that add up to more than the sum of their parts, resulting in higher-level innovations like the elimination of above-mentioned tradeoff. In the next section, we’ll learn how one might learn to start pulling rabbits out of hats, both as designers, and as users.

Naked Objections

iii

¶ i

Discoverability is a neologism of the HCI trade that refers to how self-evident a given affordance is. It describes the extent to which both unknown-unknowns ("Is this a button?") and known-unknowns ("What does this button do?") are rendered apparent by the design of the affordance’s form alone, sans written documentation, onboarding tutorial, or experienced power user positioned over-shoulder. An affordance is “highly discoverable” when it requires a user to undertake only an abbreviated period of discovery before it becomes apprehensible, thus usable, to them. The ideal case of discoverability entails the user knowing inherently what to do prior to encountering the affordance⏤or, rather, any process of discovery is absent entirely. Bundled up in this definition are four components worth examining.

Fig. — probe the blank. Is this a button?

First, the contradiction: Discoverability seeks to minimize discovery. It values the end and while seeking to eliminate its means. Both user and application exist in a sort of stateless, eternal present, where no investment towards a more mutually-beneficial future can be made by either party. There’s now prospect of growth here; the base case is that in which your interface is single-serving, meant to stand out on account of its own form (other than perhaps some graphic design flair if you’re trying to sell something). We are still in the age of low fruit regarding software, where raw compute by itself is so potent that very little human symbiosis is required to get something of value. Under these conditions, optimizing for throughput rather than retention makes sense in most contexts.

iii

PARC, in some sense, can be seen as supplying us with the activation energy to achieve bare-minimum mass computing literacy. This wasn’t learning to play an instrument, or even drive a car: it was a perhaps a handful of atomic gestures, even less so if you already were familiar with typing. But we still, somehow, collectively learn them through a process of discovery. Once we got the hang of things like clicking scrolling, they disappeared into the background. With that initial investment of discovery on the user’s part were over the initial hump. No major innovations, save perhaps two or three multi-touch-enabled gestures introduced by Apple in the wake of the smartphone in the late aughts, has since added to this repertoire of atomic, gestural level. Having made the leap from zero to one, the timing for when we’ll make the leap from one to n is still uncertain. Designers will need to take the risk that the true value of their application won’t be immediately available to users, and users will have to trust designers that their investment won’t be in vain.

Second, the assumption: Discoverable to one is discoverable to all. It's a common refrain among practicing HCI-ers that our physio-spatial intuitions, honed from birth onwards, are a trove of untapped potential. The deftness and dexterity with which we perceive and interact with the physical world are so rich and expressive inputs, and dynamic media so much richer and expressive outputs than anything in the physical world, that it’s heinous to imagine us caught in our current equilibrium of the two being so loosely coupled, so low-bandwidth, via, at best in the larger marketplace, the two-dimensional “Novocaine to the wrist” that is “pictures under glass”[1]. The impedance mismatch alleged has a lot of merit, and remains criminally under-explored in HCI. But still, before adding historical or cultural resonances to the fray in the next paragraph, contesting that any sort of physio-spatial arrangement is inherently discoverable⏤or, rather, is primordially discoverable⏤is an appeal to a timeless, universal human nature regarding perception of form is a mistake many in the arts and design have made before[2].

1
From Bret Victor's "A Brief Rant on the Future of Interaction Design" (http://worrydream.com/#!/ABriefRantOnTheFutureOfInteractionDesign)
2
"Doctor Lawyer Indian Chief: '"Primitivism" in 20th Century Art' at the Museum of Modern Art in 1984" (https://www.artforum.com/print/198409/on-doctor-lawyer-indian-chief-primitivism-in-20th-century-art-at-the-museum-of-modern-art-in-1984-35322)

Third, the association: When it comes to the design of an interface, discoverability is synonymous with "good." This is an artifact of the SaaS-driven obsession over customer acquisition, and has imbued products with a gradient of hostility that sets the path of user advancement is oriented against the grain. The more aquainted a user becomes with your interface, the more alienated from it they become. As soon as the honeymoon phase of “onboarding” is in the rearview mirror, suspicions start to arise that one's been walking the plank all along. Retention is more savvily handled via extra-interfacial means, like difficult-to-cancel subscriptions, vendor lock-in, or outright monopolization, so there’s little incentive to favor journeys to mastery over shortcuts to competency. An application’s “power user” often does not achieve such a status because of their hard-won facility with its features and uses⏤what it can and should do⏤ but by dint of their tacit knowledge gained from hours spent in search of workarounds for shortcomings or bugs⏤what it can’t or shouldn’t do. The latter isn’t exclusively a negative or entirely avoidable. Hating one’s tools for their shortcomings is inevitable as part of mastery[1]. And to most designers, there’s a sort of prestige to their creation being used in ways they had never intended it to be, certifying them as a sort of meta-design wunderkind.

1
https://www.hillelwayne.com/hate-your-tools/

Unfortunately, this quasi-KPI, a sort of amalgam of robustness and flexibility, is all too often sacrificed at the fate of actual, measurable, more prosaic KPIs. From a cost-benefit perspective, it’s often cheaper to just expose a rate-limited API and let those with programming prowess have at it themselves. This makes the 1D-adept among us one-eyed in the land of the blind, at least if the API design is half-decent. But what if we consciously designed towards binocular vision, for all? In service of this analysis does not prescribe product developers become “feature factories,” stuffing their products to the gills with hundreds of affordances, each purpose built around a “user story” or some other ghastly pointy-haired-boss jargon. Chef’s knives, after all, are featureless, save for one, yet one can spend years mastering its affordances, which are hardly “discoverable.” Yet no one would fault the cutler for not forging a diagram for how to slice scallions directly into the blade.

vii

There are a very few dynamic interfaces today which embody these qualities[1]. Proper fluency in Vim, for example, is more of a departure from touch typing than touch typing is from hunt-and-peck⏤and the few who have mastered it are ferociously the better for it. But designing this amount of open-endedness into a product is immensely difficult. Even leaving designer skill out of the equation, there’s this sort of ferralness to objects that demand so much of their users to domesticate them, that no software business today, given how low-fruit an endeavor virtually any software product remains, is willing to accept and invest in getting to market. Video games do have some stake in guiding users along a path of expertise, which is proof-of-concept that humans can become incredibly adept at wielding software, but when will more general users of computers achieve anything remotely close to this level of fluency?

1
"In the mind of today's technological entrepreneur, the ideal user (and employee) is semi-skilled - or unskilled entirely. The ideal user interface for such a person never rewards learning or experience when doing so would come at the cost of immediate accessibility to the neophyte. This design philosophy is a mistake - a catastrophic, civilization-level mistake. There is a place in the world for the violin as well as the kazoo. Modern computer engineering is kazoo-only..."(http://www.loper-os.org/?p=861)

viii

Fourth and finally, the arrestation: Discoverability limits computer experience to what we can readily and rapidly perceive. Warhol’s “people should fall in love with their eyes closed” might be updated to say, “people should learn interfaces with their monitors off.” This is hyperbole, but barley: if people must form entirely anew their mental model of a piece of software based solely on what’s visible within the confines of the rectangle on their lap or in their hands, how can we ever hope to build interactions for those who wish to advance past this least-common denominator and interactwith dynamic objects like computers on higher planes?

And what happens when compute is harnessed to create dynamic experiences that are very purposefully non-referential? When it comes to experiences deliberately designed to short-circuit and subsequently expand our engrained intuitions, relying on those very same intuitions as shortcuts towards this end obviously won’t do. Gray Crawfords’s “Developing Embodied Familiarity with Hyperphysical Phenomena”[1] for instance, demonstrates the tremendous value of a 4D interface for increasing our understanding of the world⏤in this case both literally four-dimensional as well as in the sense employed throughout this essay.

1
https://www.graycrawford.com/thesis

Discoverability, because it sets up an impossible contradiction, because it makes an appeal to a non-existent universal, because it markets itself as the imperative, and because it prioritizes what we first see over what we may later feel, is a false god that must be torn down from its pedestal. When it comes to the design of end-user interfaces, no longer is it virtuous to view a user’s novice ability as their most salient, most enduring attribute. We must seek to end this practice of emphasizing that which stunts their collective growth.

How might we begin to effect this change? There are two examples I wish to highlight of good work being done, though not explicitly yet, towards these ends. First, in recent years Apple’s Human Interface group has shown the fortitude to gradually push iPhone users towards pulling rabbits out of their hats. The abolition of the “home” button was by far and away the shining example of the decade of what I’m getting at⏤a half-inch or more of screen real estate dedicated to first-time users still finding their bearings. This, coupled with innovations like App Switcher, also hardly discoverable, have created an environment where users who take a few minutes to learn the rules of the road can operate their devices with a very high degree of fluency for years to come.

Fig. — the home button dissolves; swipe up from the bar

xii

Second, there is a wealth of material, stretching from the 60s to the early 90s, that deals with human-computer interaction, and, in a manner not unlike low-background steel that was forged pre-Trinity, is free of PARCs vice grip on the discipline to varying degrees, in a way that computer users and HCI researchers of more recent years are not. One such example of this I’ve recently become enthused with is David Kurlander’s excellent “A History-Based Macro By Example System,”[1] which is a clever, if a little dated graphically, means of empowering users with opened ended “meta” capabilities to add a layer of “pulling rabbits out of hats” style control on top of any existing application.

1
"A History-Based Macro By Example System," David Kurlander, 1992

Fig. — macro by example, after Kurlander: record on one, replay on another

xiii

In closing, I have come to understand a certain tenet of “pulling rabbits out of hats” that must guide our efforts to advance beyond the miracle of PARC and onto the next one. It is that architecture and literacy exist as two extremes of the same continuum. At the level of designing both atomic interactions and full applications, these will always be in tension. Some say that eventually the personal computer will effect a revolution in literacy on the scale or greater than that of Gutenberg’s. It’s difficult to say what this will look like, but it’s worth noting that the latter didn’t happen because everyone went out and became printers themselves. So, too, will the coming revolution not occur with everyone moving from fully architecture to fully literate, with everyone becoming programmers, or, at the very least, programmers in the 1D sense. It’s always an interplay; it is the designer’s task to continually strike new balances.

xiv

In my previous work, Computation Becomes Geometry, I closed things with a quote from one of the opening paragraphs of Bruno Latour’s monumental Visualisation and Cognition: Drawing Things Together. Here I will close with another quotation from his essay, from the concluding paragraph, on the deceptively fragile nature of media. Is is for this reason I assign such a deep responsibility for those who work on new media, especially the dynamic media computing affords us:

This is precisely the paradox. By working on papers alone, on fragile inscriptions which are immensely less than the things from which they are extracted, it is still possible to dominate all things, and all people. What is insignificant for all other cultures becomes the most significant, the only significant aspect of reality. The weakest, by manipulating inscriptions of all sorts obsessively and exclusively, become the strongest. This is the view of power we get at by following this theme of visualization and cognition in all its consequences.[1]

1
"Visualization and Cognition: Drawing Things Together," Bruno Latour, 1985

Pulling Rabbits Out of Hats

Works cited

The Anti-Mac User Interface
Put That There
A History-Based Macro By Example System
Cognitive Dimensions of Notations: Design Tools for Cognitive Technology
Alto User's Handbook
Origins of the Apple Human Interface
Notes for a Modeling Theory of Science, Cognition, and Instruction
Visualization and Cognition: Drawing Things Together
Naked Objects
Sketchpad, A Man-Machine Graphical Communication System
Augmenting Human Intellect
The Design of Everyday Things
Knowledge in Pieces
Architectual Intelligence

Annotated reading list →Back to the table →