Rewind reality and experience it again

“I’ll have to leave now” is a phrase often heard in video conferencing. But imagine a future in which you could not only disconnect from online meetings but also leave the hubbub of the office with a click whenever you wanted to focus on an important task. The events happening around you would become invisible to you. Instead, they would be recorded and you could play them back later on demand. With recordings in 3D and so lifelike as to be almost indistinguishable from reality.

Researchers at ETH Zurich have achieved at least part of this vision. Christian Holz, Assistant Professor of Intelligent Interactive Systems, and Postdoc Andreas Rene Fender equipped an office with depth cameras capable of capturing objects and people in three dimensions. A computer system continuously records everything that happens in the space and detects how events are causally connected. The researchers recently presented their concept and the corresponding system at a conference.

In this “future office”, the user wears a virtual reality (VR) headset and headphones. Per default, the headset renders the physical office space around the user as well as virtual display screens and objects, embedding these in the physical environment. When the user wants to withdraw from what’s going on around them, they can activate “Focus Mode”, which hides everything that isn’t happening in their immediate workspace. Later on, they can play all the “hidden” events back and experience them as if they would happen “here and now” from the user’s point of view.

Much more than an answering machine

It’s tempting to compare this system to an answering machine, but Holz qualifies that idea: “When someone leaves a message on an answerphone, they actively formulate a brief message, putting thought into what they want it to say. But in asynchronous reality, the ‘caller’ doesn’t have to actively communicate at all – they can just do and say things as if the other person were present. Later, when they are ready, the ‘recipient’ can play the events back in the same space and interact directly with objects to experience for themself what has been recorded. That makes the recording feel very real.” So the two people involved experience the same events in the same space – only at different times.

A system that preserves causalities

Holz stresses how the system differs from a typical video recording. Within Asynchronous Reality, the user can selectively control the playback of the events they missed. The immersive headset displays any objects connected with an event as glossy shapes. As soon as the user approaches one of these objects, the system plays back the appropriate recorded event.

However, if playback were tied to a single event each time, it’s possible that the viewer wouldn’t be aware of other events without which the recording makes no sense. So the system ensures that all relevant prior events are also played back. Fender says that the computer system’s ability to recognise and point out these causalities is one of the core innovations of this work. After leaving Focus Mode, users can approach a particular object that they are interested in. Then, instead of the object just appearing, they see the chain of events that are related to that object and how it got there, “therefore the user’s reality always remains coherent, and no causalities are violated”, Fender says.

Reconstructing the past

This “stepping away” from reality is just one application scenario. The system also makes it possible for users to experience past realities in which they weren’t even physically present. This means for instance that teams developing physical prototypes can work seamlessly in shifts, with the system helping them to understand what changes the prototypes have undergone in their absence – down to the prototype’s individual parts.

Another potential application is in teaching, when schoolchildren or university students are working on models or experiments. Teachers could then “relive” the situation at any time to provide more precise feedback or mark finished work based on how it evolved.