How Unreal can use real-time rendering in virtual production

Jul 21, 2018 at 02:00 pm by Press Release


At NAB 2018 in Las Vegas, AMD partnered with ARwall to showcase a new use for real-time rendering in virtual production: an augmented reality wall that responds to changes in camera perspective, thus eliminating the need for compositing green screen footage with rendered CG in post-production.

The ARwall Effects (ARFX) System, running on AMD hardware, was on display for all NAB participants to demo.

With the ARFX System, real-time images, computed with Unreal Engine, are displayed on a screen large enough to serve as a scene’s backdrop, and the images update in real time in response to the camera’s movement. When live actors stand in front of the wall, the result is an instantly “composited” shot — a real-time 3D virtual set extension.
 

AMD showcases ARwall technology at NAB 2018


“This is akin to what you'd do with a green screen, but now there's no green screen,” says Frank Vitz, Director of the Immersive Technology Team at AMD.  “The camera is free to move around. You see your composite happening right there in the camera.”
 
With the ARwall system, the artifacts that can affect a green screen shoot are no longer a concern. “You don't have green spill. You don't have problems with hair,” says Vitz.
 
The ARFX System is an evolution of rear projection techniques, where a filmmaker projects a previously recorded film of the environment on a screen behind an actor during live filming. However, such techniques can only support a viewing angle perpendicular to the screen—any other angles immediately ruin the effect.
 
With the ARFX System, the camera can move and shoot at any angle, and roam anywhere in the virtual set with a hand-held camera for a fully immersive filmmaking experience. Perspective of near and far elements in the CG environment respond as expected in real time, with none of the limitations of a 2D projection. In addition, actors can now see and react to the environment, helping to enhance their performances.
Physical_Set_Only.jpg

Nvidia Store
Sample use: practical set is a walkable platform (shown in front of a blank video wall).

Physical_Set_and_ARwall_railings.jpg

Projection on video wall can provide virtual set extensions (railings, staircases) calibrated to line up with practical set. When the camera moves, perspectives of set extensions change to always remain “attached” to the practical set at the appropriate angle.

Vantage-point1.jpg

The rest of the environment is added to the projection, providing a full-screen backdrop for the scene updating in real time, even with a moving or handheld camera.

vantage-point-3.jpg

Actors on the practical set can be shot live against the video wall for an in-camera composite, with background angles updating in real time as the camera moves.


“It's kind of the Holy Grail of virtual production to be able to make a seamless composite on the fly,” says Vitz. “This is amazing new technology that adds another tool to the toolkit for virtual production.”

To generate real-time updates to the background, ARFX uses Unreal Engine in conjunction with a set of trackers attached to the camera. The system can also employ AMD’s Advanced Media Framework (AMF) and IRT sensor to provide lighting information to Unreal Engine. A Zcam 360 camera collects lighting information which Unreal Engine uses to light the virtual set extension to match the practical set and respond to any lighting changes in real time. The system shown at NAB is powered by the AMD Ryzen Threadripper and two Radeon Pro WX 9100 GPUs. 
 
"Filmmakers are the most demanding creators when it comes to realism and fidelity,” says Rene Amador, CEO of ARwall. “We knew that in order to make ARwall successful, we would need an engine that could handle whatever we threw at it, as well as the rigor and demands of a production environment. Unreal Engine has met these needs and more. Every update that shows up is like Christmas morning for the team."

Want to try out Unreal Engine? Join the Unreal Studio beta today and get access to Unreal Engine plus export/import tools, learning videos, and more!


Nvidia Store

Sign up for our newsletter