A man and woman are smiling while using a laptop.
A man and woman are smiling while using a laptop.

State of the art-ful AI: Video generation using runway.ml (part 5)

We use Runway's 'image + description' option to generate an AI video clip and adjust it after the fact.

Welcome! This month we’re introducing one of the world’s most popular generative AI toolsets, runway.ml, and using it to generate a set of short video clips to create a promotional video for a conference. The series continues below.

Project overview

As generative AI develops at speed, we’re delving into the apps and tools currently available to help with ‘real’ creative projects. In this case, we’re using the video generation tools available at runway.ml to create short video clips to edit together to create a short promotional video for an upcoming conference. We’ll use chatGPT to suggest a script, then use runway’s text-to-speech tool to create a realistic-sounding voiceover.

Note that this is not an explainer series, though we do detail the main steps to follow at runway.ml.

Using runway’s AI video clip generator: Use the ‘image + description’ option to apply changes to generated clips.

For the next clip in our video sequence (clip 6) we’ll use runway’s ‘text+description’ option to generate a clip then make after-the-fact adjustments to it.

Our next clip is clip 6, where we request a clip of two people engaged in a happy activity on a laptop.

1: Start by adding the text prompt and using the ‘free preview’ option in the usual way to generate possible start frames for the clip:

A screen shot of a video screen showing several images.
A screenshot from runway.ml showing preview images.

2: Select a prefered preview image then select the ‘image + description’ tab. Enter a new text prompt for the changes to make:

A screen shot of a video editor on a laptop.
A screenshot from runway.ml showing the image + description tab being populated

3: Play back the generated video to see if you’re happy with the results:

A screen shot of a video editor on a laptop.
A screenshot from runway.ml showing particles rising from a laptop

The generated video clip (clip 6)

You’re welcome to download this clip and use it in your own video / generative AI projects.

Clip 6: Two people interacting and smiling over a laptop as particles rise from the screen.

Issues / errors observed

We expect there to be visible issues with generated video clips right now, though many such ‘glitches’ will be eradicated as the AI models develop. Think of these issues as part of the fun!

  • The request to add ‘holographic particles rising from the screen’ has not been completely successful, though ‘sparkles’ have been added. This prompt will require additional refinement to return the desired result.
  • The people are too ‘airbrushed’ and exhibit overly-perfect, bright white teeth. This is very common in AI video generation at this time (January 2024).

Join us for part six of this series when we generate the remaining short video clips for our conference promotional video.