Virtual production and AI are only as inspiring as the humans creating with them

Even by the standards of a rapidly-evolving industry like VFX, where technological innovation scales as quickly as consumer demand, the last few years have been fast-paced. Two developments have dominated the conversation—virtual production and artificial intelligence. To a casual observer, they have little in common. For the people working in VFX, however, both are disruptive forces that pose challenging questions about how they can be deployed; the risk they pose to existing human operators; and the future of motion capture in VFX.

Virtual Production has been around as long as VFX companies have been tracking people and cameras alike. In-camera VFX (ICVFX), however, was in its nascent years when adoption scaled intensely in response to the global pandemic. As we enter 2024, ICVFX is beginning to mature and find its place in the production pipelines of the entertainment world. When enjoying any of the incredible spectacles captured in movies or TV shows that were only made possible by ICVFX, it’s easy to forget that just a few years ago it was still a disparate set of disciplines being cobbled together on the fly in a complex, global tapestry of users and experiments.

Normal questions (What does it cost? How does it work? Will it scale?) were, for a time, subsumed by the drive to deliver, deliver, deliver in the face of the streaming platforms’ ever-growing appetite for content. This, alongside the obstructions presented by the pandemic lockdowns, led to the rapid bootstrapping of new technologies and methods in the service of getting the job done. Technical and situational knowledge were acquired on the fly until a body of expertise and best practices emerged and began to settle.

The maturation of ICVFX (and virtual production more broadly) was, of course, to be expected. What could not have been anticipated was Covid accelerating deployment to the point that it’s still finding its place in production pipelines even as the technology reaches mass adoption.

While virtual production, and all its associated technologies and workflows, will continue to evolve, it’s already clear that ‘traditional’ methods of motion capture and digital creation still have an important place in our industry.

If virtual production is maturing and going through a stage of adolescence, AI is in its infancy by comparison—though the scale and speed of its growth may dwarf everything else we’ve seen to date. It’s clear that it will impact every part of the information economy and, further down the line, markets that are more reliant on physical processes, too.

We’re seeing it in music production, report generation, statistical analysis and a whole host of other sectors. And Hollywood is already in the throes of debating just how AI will be deployed in the world of movies and TV. Many think the conversation shouldn’t get that far until we’ve determined if AI even should be used, with applications in storyboarding, scriptwriting, concept art and environmental art all on the table. These are all professions dominated by talented, competent individuals with centuries of experience between them, all with livelihoods to consider. To say the topic represents an existential challenge, alongside an ethical and technical one, is no exaggeration.

At a more granular level we’re beginning to see how artificial intelligence might impact animation, with AI animation-generation and AI motion capture both showing early promise. There’s plenty of exploration and experimentation underway, the excitement about which is accelerated by an ever-growing thirst for user-friendly, entry-level solutions for content generation. Much of this process is also about identifying and addressing concrete use cases, since the value proposition (and thus, adoption rate) of any technology ultimately comes down to making users’ lives easier. It’s very early days, so it’s to be expected that we’re yet to see many concrete examples actually make it to our screens.

We’re still in the looking glass stage of predicting exactly where the use of AI is heading in our industry. There’s lots of speculation, but we’re still very much in an exploratory phase and making concrete predictions is therefore very difficult. . Projects are being completed, but there’s yet to be a single example that has put the role of AI in our ecosystem into proper context. Tempting as it might be to state with absolute confidence that ‘AI will change the world in these ways’, humans are predictably unpredictable, and we will always surprise each other with our approaches to new challenges. 

Nevertheless, it is inevitable that AI will have a significant role in VFX. Its potential, and the current climate for content creation, are just too ripe for it to be left on the shelf. Irrespective of the ongoing ethical questions—which I think will mature and lead to conclusions far more slowly than the actual adoption of the technology, which raises numerous questions itself—it represents an opportunity for reduced costs. It ‘democratizes’ certain parts of the business, and as the demand for production just keeps on increasing, it’s difficult to argue against maximizing the benefits it provides .

The inevitability of AI in VFX is driven largely by the absence of comparable solutions to this increasing disconnect between the scales of supply and demand. Creators are fighting to keep pace with audience expectations in terms of quality, scale and spectacle. They will ultimately defer to whichever technology facilitates this.

 

So, an obvious question—what connects virtual production and AI?

It’s an oversimplification to think of either of these things purely as solutions, technologies, or groups thereof. They are forces and, as such, they represent significant change for the VFX industry, despite being at very different stages of gestation. ICVFX  is settling into its role within production pipelines and broadening the adoption of Virtual Production as a result, while AI is further back along the curve and yet to find its place in our world.

Nevertheless, both are the way that content gets made.

Initially, ICVFX presented an incredibly compelling solution to just ‘getting things done’ at a time when options were limited. We have been out of that environment for a while, so it’s gradually finding a longer-term place in content creation where it can offer the most value.

One of the most exciting elements of the increased adoption of ICVFX is how it has encouraged (and indeed, necessitated) creative collaboration between technical specialists. It’s unlikely that this will stop being the case. We can always expect ICVFX stages to be places of experimentation and innovation with a view to ‘getting the job done’—regardless of where it finally settles in production pipelines.

As far as ‘getting things done’ is concerned, AI appears poised to significantly increase the industry’s options, but the question of where it can actually offer the most value still looms large. Likewise, AI is faced with a myriad of ethical questions, such as ownership and the legalities surrounding data scraped from the web which was never formally intended for such use. It would be naive to think that legislation will not have to evolve alongside the use of AI itself, which makes predicting where it eventually ‘settles’ a far more complicated prospect. 

So, we have this interesting tapestry where both are at different stages of their adoption and are faced with distinctly different challenges, while presenting similar promises to their respective userbases; flexibility, control, scope. All the while, the explosion in streaming is creating more and more demand for content. Even while that business model itself is facing challenges of its own, that demand and level of customer expectation is not going away. And audiences expect quality and spectacle in their games and VFX shows which was previously only available in summer blockbuster event movies, resulting in spiraling production costs.

One knock-on effect of this boom in demand is the extreme pressure being put on creators. More than anything, they need flexibility. ICVFX was a real-time example of how technology can bring that to the VFX world, building on several years of development and innovation within the broader virtual production space. It also offers a recent example of how quickly a new technology can ‘settle’ once it becomes better understood. It has become one more tool in the creator’s toolkit, offering increased flexibility in what can often be an inflexible business.

For all the uncertainty that a new technology such as AI can bring, I hope that we as an industry can reflect on what we’ve learned from the boom in virtual production. The true impact of these solutions comes from the way in which they’re deployed by their users. As exciting and promising as the technology is, at the core of its output are groups of people with diverse skillsets pursuing unified goals. As is the case with virtual production, AI will only be truly transformational if it is put to work by creators with the vision to see new possibilities and new ways of working. It’s this human response that will be truly inspiring.