AI is changing every single industry on the face of the planet, and filmmaking is not escaping. Every product release, new software, hardware launch or trade show has AI at the forefront of branding and marketing. The film industry is far from sheltered from this technological change, but is instead very much involved in driving forward the software and hardware to make it all possible.
A little while ago I had the pleasure of catching up with Freddy Chavez Olmos to discuss how he was using AI to create films. We discussed the pros and cons as well as the opportunities for the industry as a whole. It is impressive to see Freddy managing to walk the tight balance between AI and creativity, somehow making sure that it is humans who drive the creative process and not the other way around.
We also discussed some of the tools he was using to help him. At Creative Bloq, it got us thinking about what our top five favourite AI tools would be for filmmakers. We’ve focused on a range of AI tools that enable things like text-to-video and music creation. Here are the top 5 AI tools for filmmaking.
01. Nuke Copycat (Rotoscoping)
CopyCat in Nuke makes it possible to automate the rotoscoping process. This is one of the most tedious and lengthy parts of the editing process and can therefore cost a lot of time and money. When paired with generative AI in Photoshop, and Unreal Engine for set extensions, it is possible to quickly and easily eliminate unwanted areas. For more on CopyCat in Nuke, read our interview with Chaos' Kam Star.
This is a tool that Freddy made great use of in his film BYE-BYE. He said said “As a result, [of using CopyCat] the limitations of using the virtual production LED wall were easily overcome. Even with moving camera shots and low lit backgrounds, the AI generated elements were perfectly integrated into the scene with no issues whatsoever.”
02. Runway Gen 2 (Text-to-video)
The second iteration of Runway (one of the best AI image generators) is causing quite a storm in the world of text-to-video video creation. Using simple prompts, it is possible to create engaging and, at times, 100% believable video clips. It still has a long way to go but this is the case for all generative AI tools. Runway is very much leading the way and producing some of the best tools out there for this type of workflow (also see the new Luma Dream Machine).
As well as using text prompts, it is also possible to provide the multimodal AI system with reference imagery to further drive the results towards a desired destination.
03. Pika Labs (Text-to-video)
An alternative to Runway Gen 2 is Pika Labs. It does a great job of capturing the industry’s imagination by claiming to “create what’s in people’s heads”. It does this using text prompts and reference imagery but has the added benefit of being able to adjust existing video clips.
The ability to modify anything in a frame using Pika Labs is indeed very powerful. Functionality that you can imagine doing manually with After Effects or Nuke is now possible with simple text prompts. To help focus the AI tools, users can use a crop region, very much like what is seen in the generative tools in Photoshop.
Pika’s showreel on YouTube shows a range of Pixar style results that are truly impressive. It won’t be long before whole films are created with Pika. At the very least, it’s a very capable tool for use in the editing post-processing stage of a project.
04. Chat GPT-4o (Scripts)
Video generation and editing isn’t the only part of the filmmaking process that can benefit from AI tools. In the area of script development, Chat GPT-4o is one of the best in the game. At this stage, I would always prefer to read a story or watch a film that has been written by a human. It seems more natural and relatable, even just knowing a human has crafted it. Not everyone agrees though, and if you’re happy letting it do a lot of your script work then give it a go.
It also does a fantastic job taking a script and providing outlines. This is one area that I think using AI is a no-brainer. It takes what has been crafted and breaks it down into something usable by others.
Carlos Viejobueno has experimented with using Chat GPT for script coverage and he’s quote as saying “it was able to identify most of the main characters and their motivations.” Additionally, “it was also able to give competent feedback and constructive notes.” This is an area of filmmaking that AI will continue to penetrate.
05. Udio (Music)
No film is complete without a soundtrack that moves the viewer along with the storyline. AI is making inroads into the film industry here as well. Udio is one of the best in the game at the moment and with a few simple prompts makes it possible to generate new audio tracks. There are huge question marks over copyright infringement here but that does seem to stop the technology evolving.
Udio contains a huge library including a track called ‘Carolina-O’. By entering the prompt: “Country Pop, Contemporary Country Tropical Rock, Country Rock, mellow”, Udio was able to generate a new track which, to my ears, sounded incredibly realistic. Filmmakers who are used to using text to filter down tracks from repositories will be used to this workflow.
Creative Bloq's AI Week is held in association with DistinctAI, creators of the new plugin VisionFX 2.0, which creates stunning AI art based on your own imagery – a great new addition to your creative process. Find out more on the DistinctAI website.
