Creating a video from a picture or even a simple text prompt has become easier thanks to a suite of new tools from artificial intelligence startup Runway. The new features include setting which part of an image you want to animate and improved image generation.
Unlike generative photography, with multiple companies and models competing for attention, generative video is a relatively new field. Meta recently announced Emu Video, an AI model that could form part of a future Instagram or Facebook — but for now, the choice is limited.
There are a handful of tools like Synthesys and Pictory, which are focused on avatar creation, and there's HeyGen, which is building translation tools. However, Runway is a more general creation tool, producing a full suite of video editing and creation tools.
How to create a video with Runway Motion Brush
Motion Blur is one of the more high-profile of the new Runway updates. It allows you to do exactly what it says — create motion effects using a brush tool. The default way Runway works is to create motion across the entire photo, but this allows you to say “only make this bit move."
The other new feature comes from a series of style presets that allow you to act like an art director for your movie, telling it what to look like, how the movement should work and lighting effects. Director mode has also had an upgrade with more fine-tuned camera controls.
To see just what is possible I decided to try it out. I used an image I’d created with Stable Diffusion’s SDXL 1.0 model. The prompt was: “A cowboy walking into the sunset in the Wild West.” I then refined it to 16x9 and in a hyperrealistic splash art style with a side-light and colorful background.
Create a Runway account
You can sign up for a free account to try it out. Runway lets you register directly with an email address or to use a single-sign-on service from Apple or Google. Once you’ve registered you’ll see the Runway project page — click start with image.
How to select the image
To add a photo that can be turned into a video click upload a file to select an image from your computer. In my case, it is the cowboy picture I previously generated. You can also enter a simple text prompt to have Runway create the image for you.
How to add motion brush
Once you have the image loaded click the Motion Brush (beta) button. This will load the motion brush view where you simply draw in the area you want to animate.
The horizontal, vertical and proximity controls will prompt the AI to set the direction and size of the motion within the window you have drawn. Clicking the save button will take you back.
How to make the video
There are other fine-tuning options, including upscaling the image if you are on one of the paid plans, removing the watermark, and smoothing out the frames. You can also set the motion and direction of the camera. Once you are happy click Generate 4s. This will create your first 4-second clip.
Your video will then go into a queue until processing time is available. Paid plans are given priority over free accounts but you rarely wait more than a few seconds. It can take a few minutes for the video to generate once out of the queue.
Generating the video
Once the video has been generated you can play it, download it for use in your video editor, or turn it into a gif — I recommend using Adobe’s free video-to-gif tool for this — or even publish it to the Runway library.
The other option is to extend it by another four seconds - up to 16 seconds. To do this simply click Extend 4s in the bottom right. It will create a new instance that is four seconds longer.
This is useful if you have a walking motion like in my video that needs a few more frames. Remember each extension uses more of the limited free credits that come with the account.
