Immersive Cinema And The Age of Voxelpunk

on Thursday, May. 1st

In the wake of the Facebook acquisition of Oculus VR the issue of the future of virtual reality beyond games has stepped into the media spotlight. The conversation has drifted into speculation about how VR will relate to the film and television industries.

It would be overstepping to say that immersive cinema is going to be completely different from the kinds of entertainment that we are used to today. Just what the composition of this new medium is going to look like, however, will likely catch a lot of people flat footed. Virtual reality opens up new possibilities for immersion, what the VR evangelists are calling “presence,” but it also brings with it a host of design challenges.

“The things we are finding out now are the things that don’t work,” said Eugene Chung, Director, Film & Media for Oculus VR.

I spoke with Chung at the Game Developers Conference in San Francisco this March, along with Oculus’ co-founder and VR of Product Nate Mitchell. Our conversation took place in the middle of the Oculus booth on the show floor, surrounded by game devs who were checking out the latest development kits from the company.

My purpose at GDC was not to discover games, but to talk immersive cinema with virtual reality pioneers. I wanted to get a handle on what skills filmmakers and others would need to develop as they sought to conquer this brave new medium.

“Early experimentation shows that the idea of the cut doesn’t really work that well,” said Chung. “It can be incredibly disorienting.”

The cut. The humble film edit: gone in the new medium. A century with the film editing standby, used by everyone from Robert W. Paul all the way to Walter Murch and Sally Menke, tossed out the window.

“The basic elements of editing a film and telling a story on film rely on the frame and cuts. Neither of these exist,” Oscar nominated filmmaker Danfung Dennis told me in another part of the GDC hall. He was there giving sneak peaks of his in-development VR film Zero Point.

“As a photographer I’ve always put the story inside of four lines and focused the viewer into a frame. But here when you’re looking at this full 360° the audience can look anywhere.”

Zero Point stitches together the feeds of a handful of RED cameras to create a panorama view. One scene is static, with edits coming in the form of slow dissolves from one camera position to the other. Another scene involves a tracking shot through the halls of another game conference, the Electronic Entertainment Expo. The tech is early, and the “stitch lines” where the different cameras converge remain prominent. Yet with a little imagination it is possible to see what Zero Point could be.

There is a chance here to transport viewers into breathtaking vistas, or drop them into chaotic scenes that unfold all around them.

For my money, however, that’s not the most interesting possibility for immersive cinema.

Those who have put on an Oculus Rift know that one of that device’s “hidden powers” is the way that it cuts you off from two major sources of sensory input–visual and aural–and replaces them with the designer’s dream. There’s a kind of magic ritual to strapping on a Rift, and when the spell is done you find yourself in another world. One that is strangely more immediate than our own.

There’s less inside the Rift, for starters. No cellphones with urgent texts. No pop-up ads to break flow. No other tab at the edge of the window tempting you to switch over to Facebook. Not yet, anyway. (This is my own number one fear in the wake of the Facebook purchase: that the world’s greatest maker of distraction ware will find a way to foul the waters.)

The isolation of the Rift opens up the possibility of more intimate experiences. Not in the cybersex sense, although it is inevitable that will happen too. I’m talking about psychological intimacy. A literary sense of perspective that allows for users to slip inside the skin of a protagonist the way that great novels transport you, but coupled with aspects of what an actor does.

A heightened relationship between the user and the character encountered in the Rift is possible. You will be able to look at character in the eye and they will look back. How will creatives design for those scenarios? Movie and TV creators are used to presenting drama, and video game makers are just beginning to explore the edges of psychological intimacy in games like Gone Home.

Another emerging design discipline may hold the key: immersive theater.

Through The Looking Glass

Shows like Punchdrunk’s Sleep No More have been entrancing audiences with elaborately constructed and choreographed worlds built here in the real world. Sleep No More, an adaption of Shakespeare’s Macbeth, uses the “sandbox game” logic of some video games as the canvas on which its action plays out. Interactivity is low. Patrons can look at, but for the most part cannot touch, the performers having the status of ghosts haunting a scene.

Oculus’s Mitchell is a fan of Punchdrunk’s work.

“If you were going to sit down and make an immersive theatre-like experience or an immersive film that way it would actually be perfect for the medium of VR,” said the company co-founder.

Back at GDC Mitchell and I compared notes on our Sleep No More experiences, and then I ranted excitedly at him about my personal favorite immersive: Third Rail Projects’ Then She Fell.

Where Sleep No More is rooted in spectacle and a ghostly veil between worlds Then She Fell seeks to get inside your head. The work is a riff on Alice in Wonderland, with the scandalous back story of Lewis Carroll and Alice Liddell thrown in for a twist. Unlike Punchdrunk’s work Then She Fell is a guided experience, patrons are deliberately led from scene to scene by cast members.

Also unlike Sleep No More, which values the silence of patrons seemingly above all else, Third Rail Projects asks that the audience speak only when spoken to. As it turns out you are spoken to a lot. The narrative arc of any given scene isn’t changed by the questions that are asked, but the answers I found myself giving resonated in unexpected ways. I was finding my way through the looking glass even though I was walking down a path almost identical to every other patron.

While I was bearing witness in Sleep No More, the experience of Then She Fell was more akin to what it is like to act in a play. I was in scenes with Alice. We had a little moment in a small room. And at the end of any cinematic or theatrical experience it is those little moments that matter the most.

One problem with immersive theater, from a commercial perspective, is scale. Then She Fell gets fifteen people through its doors with each show. Virtual reality experiences could be accessible for many more people, and they would not be limited by pesky things like physics.

“Imagine you’re sitting with Alice in that small room,” said Oculus’ Nate Mitchell, “and the room begins to dissolve and change and morph. It can grow and you can get that sense of scale of being there; and anything’s possible. The canvas is wide open.”

The conversation with Chung and Mitchell solidified an idea that had been growing since I first took note of immersive theater and the return of VR. These are not separate disciplines, but two sides of the same coin. The core design challenge of creating 360° worlds is identical. The only difference is the material from which those worlds are made.

From Cyberpunk to Voxelpunk

The previous, abandoned, era of virtual reality unfolded under the banner of cyberpunk. The extreme idea there was that we were going to abandon the real world for a digital construct. Some called that The Singularity. This notion is anchored in anxieties over personal and political apocalypses. Read any good old cyberpunk story and you’ll see a world on the edge of environmental collapse ruled by corporate oligarchs.

Cyberpunk is an apt diagnosis of our current condition, but we need something more than a diagnosis right now. We need some inspiration.

I find that in the humble voxel.

Designers in 3D rely on voxels–sometimes known as “volumetric pixels”–to build their models and worlds. Where 2D screens use pixels to create images, voxels add an element of depth to represent volume. I find the idea of the voxel a good shorthand for thinking about the similarities in the work of physical and virtual designers. Both need to fill space with meaning, and that practice is served well by thinking about the details.

So let this era be a step beyond the network-based, pessimistic thinking of cyberpunk. Let this be the era of voxelpunk: when we got over trying to escape into our computers and began designing better worlds.

UPDATE: The companion piece to this is up at Bitter Lemons, a Los Angeles based theatre site. It is a call to action for theatre artists to seek out VR.

Image: Rachel I. Berman (Alice); Publicity image by Darial Sneed for Then She Fell.

Special thanks: Robin Arnott for indulging my rants at GDC, Leigh Alexander  for the “distraction ware” line, Third Rail Projects for being incredible, and you for reading this.


To Build The VR Education of Tomorrow One Scholar Turns To The Past

There are plenty of folks in the education technology field who are excited about virtual reality as the next great educational tool.



We’re Closer To Our Photorealistic VR Future Than You’d Think (INTERVIEW)

A look into the virtual world of tomorrow with USC researcher Paul Debevec.


Prepare Yourselves For The Personal VR Video Revolution

Virtual Reality is rapidly approaching a watershed moment.

Ralph Echemendia

Spoiler Alert: Hollywood Isn’t Taking Cyber Security Seriously (LA Film Fest)

Tonight at the Los Angeles Film Festival, squeezed in between movies and red carpet events, a symposium on Cyber Security is being held at the Grammy Museum.

Gumroad Rentals H

Rent Video Straight From Social Media Via Gumroad

The people who brought the “Buy Now” button to Twitter are going all-in on film distribution.