A look into the virtual world of tomorrow with USC researcher Paul Debevec.
The build up of excitement over virtual reality is a palpable thing online. You can see the coverage of the budding industry growing every day. At first it’s easy to dismiss as a a jaded tech press looking to hype the next new thing. That’s a myopic view.
What we’re witnessing is the culmination of years of research and development in computer graphics and display technology. Tiny jumps that have gone on behind the scenes which add up to one big leap.
This past weekend at the Los Angeles Convention Center some of the fruits of that research were on display at the VRLA Expo. VRLA is a gathering of VR developers and enthusiasts which has become an international attraction. While at the Expo it was possible to run into attendees who had come in from as far away as India just for the event.
The expo floor was filled with all kinds of demos, but just as interesting was what was being discussed during the gatherings presentations. It was here that even further glimpses of the future were possible. That is thanks to researchers like Paul Debevec, who shared their insights.
Debevec is the Chief Visual Officer at the University of Southern California’s Institute of Creative Technologies.
He gave a talk entitled “Light Fields and Photoreal Virtual Actors for Virtual Reality,” which delved into the history and science of a technique that will likely be the foundation for what we will think of as cinematic VR.
Before that talk I had the opportunity to talk with Debevec about light fields and virtual actors. There are, after all, so many questions. They start with just how the hell does this seemingly magical technology work and go all the way to whether or not aspiring actors should just give up now.
LIGHT FIELDS – THE CAMERA TECH OF TOMORROW
In Debevec’s talk he spoke of the two-decade plus foundation of light field capture technology. To understand the technology you have to change the way you think about images. Instead of thinking of a photograph as a flat image picture it instead as a record of the rays of light that make up what the lens sees. When multiple images are taken at different angles it is possible to know where two rays of light intersect, and thus create a kind of virtual camera.
This makes it possible to create a 3D image of a real world scene which you could then “move around in” using VR.
Debevec and his colleges demonstrated the technique in a video for OTOY, a cloud-rendering company in LA. In the video it takes a camera, a fisheye lens, a motorized rig and a proprietary software set to capture the light field of an office and turn that into a full stereoscopic image.
“It really is magical,” said Debevec, “because now when you move your head around in VR—whether it’s OTOY’s offices or one of the other scenes that have been recorded you get the corresponding perspective shifts of the scene that’s there. You can even push your head outside of the viewing volume a little bit—forward and back—and still reconstruct views because the cameras that were around you still saw all the rays that you need.”
Creating photorealistic VR environments can be tricky. Anyone who has spent time with the current incarnations of the technology knows that the stitch lines on images are a barrier to full immersion. It’s a problem like that which faced computer animation in the late 90s. the closer a digital character came to looking real the more critical of their appearance audiences became. It seems that the more realistic something looks the more we expect it to look real.
Light field capture solves big chunks of this problem, right down to how things look.
“All of a sudden what the surfaces of the scene are made out of become very apparent and they come alive,” said Debevec. “It gives you a sense not just of what it looked like from a particular point, you get this much greater visual sense of what everything in the scene is made out of.
“You just feel all of this a lot better.”
That feeling is all-important. VR creators chase the holy grail of presence with a knight-errant’s passion. Yet the way that the reality part of virtual reality courts the uncanny valley is the sharper part of a two edged sword. One that’s ever so applicable to the characters who will inhabit virtual worlds.
“If they don’t look at you— if you can’t interact with them— then it’s really going to feel like this existentialist thing where you’re there but you can’t do anything.”
There are two main approaches to licking this problem. One is to use real actors to play the parts of the characters. That’s pretty much what video games do these days—from the Uncharted series to the latest Halo cinematic. Pre-rendered performances are one thing, but it is also becoming possible to render them in real time. This is something that the wizards at Industrial Light & Magic’s xLab have shown off.
Both of those options have major restrictions, however.
Pre-rendered digital puppets performed by actors won’t be able to diverge from the script, or respond to the actions of a user. In a realistically rendered light field scene that would crack the illusion right open.
Real time motion capture would solve that problem, but it would also mean having an actor and a capture rig at the ready. That could get pricey real quick. Which means live actors will likely remain a premium option for those who can afford it.
In time interactive VR producers will look into creating characters that exhibit some form of artificial intelligence.
“There’s twenty years of research heading in that direction. It’s not easy, it’s not general purpose yet, but VR is really going to light a fire under that. We’re going to see a lot of interesting things come in the next ten years I’m sure of it.”
The algorithms that run these characters will have to take care of everything. From realistic motor skills to making narrative choices based on emerging situations. They’ll have to do everything that a skilled improviser does in order to create a convincing illusion of character.
“Actors don’t have anything to worry about yet,” because of this, said Debevec, “and probably not realistically for maybe 15 or 20 years.”
While speaking with Debevec I wondered aloud if what if happening with with this technology is akin to what happened in the Renaissance. That era when the visual arts underwent a drastic acceleration.
“When they started getting perspective right,” said Debevec, “it just felt like a level of reality that was completely natural, completely expected and up until that date completely missing in a way that you didn’t even think that you were going to get that. You weren’t even led to expect that and once you have it. Oh my gosh. It’s so much like the real world. More of your brain turns on to accept the scene.”
It feels more and more like the same could be said about today’s virtual reality experiences. The work left to go to take us into photorealistic VR realms is staggering, but it turns out we might be farther along that we realized.