(2019-08-30) Palmer Spatial Interfaces

John Palmer on Spatial Interfaces. Think about this: there is no app that replicates a deck of cards. (CardDeck)

If you start designing a deck of cards app as a traditional 2D app the way we think about them today, with 2D panes and menus and buttons, it becomes painfully difficult to think about. But what about in 3D? It seems that the only way to build an app that replicates the full functionality a deck of cards is to build a 3D simulation or game. Model the cards in 3D and put them on a 3D table. As long as you have controls for reaching out and picking up the cards, and moving them in space, you can do anything you can do in real life. Preserving the higher dimensionality makes it simpler and more intuitive, not less.

Historically, it has seemed obvious that displaying things in 2D, and therefore reducing dimensionality, makes them easier to understand.

We can see this in examples like galleries of images, menus with buttons, forms with columns of text inputs, etc. Most of the software we use today, in fact, is two-dimensional. 2D interfaces living on 2D screens.

It turns out that there are some applications where increasing the dimensionality to 3D (or leaving them in their original 3D form, like in the deck of cards example) can make the applications more intuitive. I think this is going to be a major force in the near future of software.

Why now? There are two main reasons.

First, we tend to associate these concepts with gaming, and gaming (computer game) is growing extremely fast.

Second, we're at a turning point with the ease of 3D programming, animation software, and the browser's ability to render 3D graphics. Creators now have a variety of tools at their disposal for making spatial software: WebGL, Three.JS, Unity, Unreal, etc. This is something that software designers should seriously consider.

some applications use spatial relationships to suggest how they should be used. Incarnations of this thinking are called spatial affordances.

Sometimes, a spatial affordance can be an instance of skeuomorphic design. This occurs when the spatial affordance directly analogizes the real world, as in skeuomorphism.

If skeuomorphism is making UI elements represent their real-world counterparts, skeuomorphic spatial affordance is making UI elements represent their real-world counterparts and their spatial relationships to their surroundings

Snapchat is an example of an app with completely spatial navigation. The user swipes in different directions to navigate to different screens, which are laid out beside, above, or below each other in 2D.

Video conferencing UX is broken, and there's a lot of greenfield for spatial thinking in the space

For example, there are only two ways to view a meeting: Speaker View or Gallery View. When in real life are you ever looking at a grid of faces? Never. It's not natural.

You could also design things like an interface that can pan left and right to see different people's faces as if you were standing in a circle. Different spatial interfaces could make it easier to run all kinds of meetings.

Spatial thinking can be powerful in social software by establishing a stronger feeling of closeness. In games, unlike most social software, the concept of "presence" is at the forefront.

Surprisingly, some software with spatial concepts built in can be used reasonably well for social interaction, even if not the intended purpose. I recently tried using Figma to hang out with friends in the evening due to its sense of presence. Live cursor positions let us all know that we were actively participating. In fact, as we did this together, we immediately began drawing bodies for ourselves, with chat bubbles to type into. This produced a much stronger feeling of togetherness and emotional connection.

Back when I worked at Snapchat, I formulated an app idea based on the same notion: spatially representing the way speaking to someone feels in person.

In this concept, the person you're messaging with is shown "across the table" from you. There can only ever be one chat bubble in between you. Every time someone types some text and presses enter, the bubble is replaced with the latest message. Throughout my time at Snapchat, I actually designed and proposed a number of features based on spatial thinking, but none of them were ever built.

Web browsing

Over time there have been many different ways of thinking about navigating the Internet spatially. Surfing the web, a series of tubes, hyperlinks as portals to other pages, etc. Well, these spatial concepts can be translated into interesting UI innovations

One example is an idea I call "trails."

A breadcrumb in this case is a single pixel that you can place in a precise location on a webpage. Placing a breadcrumb could be as simple as Option + click. While navigating the web, you could leave breadcrumbs on different pages you find interesting over the course of a browsing session. When you're done, that sequential "trail of breadcrumbs" would be saved

Another idea for spatial thinking on the Internet is creating a spatial map. We talk about "Crypto Twitter" or "VC Twitter" as concepts. But don't they feel a bit like places?

Some of the best parts of conferences happen outside of the scheduled programming. The magic is in the unexpected interactions with people you bump into. You could build a "conference simulator," a lot like a video game, and gain the ability to run conferences remotely.

The ability to run conferences this way would make them easier to attend and better for the environment. Second Life was a piece of software that understood using gaming for this kind of socialization.

Why does that image from Second Life make me more excited? It feels more real.

We're already noticing the signs. People start teaching their kids at a school inside of Second Life. Marshmello plays a concert to a live crowd in Fortnite. Movie theaters get converted to virtual playgrounds. Employees start working together in VR.

We could have games for anything. Games for attending classes, co-working, and making art. Games for work. Games for just hanging out. We're going to make these kinds of games. But at this point, it's time we stop thinking about them as games and start considering them part of a broader field: spatial interfaces.