Survival of Compositing in Virtual Production Landscape
In the world of visual effects, a new technology is making waves, and The Mandalorian is at the forefront of this revolution. The television series, produced by Industrial Light and Magic (ILM), uses a virtual production technology that is set to redefine the landscape of the visual effects industry.
The heart of this innovation is the immersive LED wall, a 20-foot tall, 75-foot diameter performance space that wraps 270 degrees around the stage. This LED wall displays photorealistic digital environments around actors, enabling real-time parallax and lighting integration. Over 50 percent of The Mandalorian Season 1 was filmed using this new methodology.
This technology offers numerous advantages. For instance, the virtual environment on the LED walls moves in perfect sync with the camera's movement on-set, providing a level of realism and efficiency unheard of in traditional VFX compositing techniques. The LED wall also lights both the set and the talent with light from the VR environment.
However, this technology is not without its challenges. One such challenge is the issue of Moiré patterns, occurring due to the size of the LED's in the wall compared to the camera's imaging sensor. To avoid this, the simplest solution is to restrict shot composition, or additional lighting from above and/or from the front is necessary for most shots.
Another challenge is the colour of costumes and props, which can shift when lit by the RGB LEDs of the LED wall due to metamerism. To address this, matching renders between CG elements and the VR environment is necessary to avoid needing post-production fixes.
The virtual set invariably has a practical floor that requires colour matching and seam removal in post. Classical effects such as glamour passes, wire removal, and costume repair still need to be added by roto, paint, and prep artists. Virtual production has a long list of issues that will still leave plenty of work for compositors.
The use of virtual production technology, as exemplified by The Mandalorian, shifts the role of compositors from traditional green-screen keying and background replacement to more integrated tasks involving real-time visualization and LED volume environment enhancement. Compositors now engage earlier and more interactively in the virtual production pipeline, facilitating real-time background adjustments, edge refinement, and integration of CG elements displayed live on LED volumes.
Traditional tasks such as rotoscoping and tracking still remain crucial for shots requiring additional fine-tuning beyond the virtual set. Tools like Silhouette (for rotoscoping and paint) and Mocha Pro (for planar tracking and stabilization) continue to provide essential functionality.
The future of VFX tools lies in integration with virtual production pipelines, supporting metadata exchange, camera tracking data usage, and hybrid compositing workflows that blend in-camera effects with post-processing. This technological shift allows for faster iteration, better lighting consistency, and enhanced actor interaction, reducing some post-production compositing overhead but increasing the value of tools capable of refining and extending virtual production outputs.
Thus, virtual production transforms compositing from a primarily post-production discipline into a hybrid role spanning on-set collaboration and post-production refinement. The visual effects industry's foundation is rapid technological evolution, and artists have historically risen to the challenge by developing new and indispensable roles to respond to these changes. This new technology is no exception, requiring artists to stay abreast of changes through education and training.
Technology, such as virtual production, is reshaping the visual effects industry, as demonstrated by The Mandalorian. Thischange in technology allows compositors to transition from traditional green-screen keying and background replacement to more integrated tasks, including real-time visualization and LED volume environment enhancement. However, the technology is not without challenges, like avoiding Moiré patterns and addressing the issue of costume colors shifting when lit by the LED wall.
