We continue our series introducing one of the world’s most popular online generative AI video production toolkits, as we create a short video for a brand new immersive experience.
The AI tools and models used in this series
We’ll use runway.ml to generate video clips and assemble them into a sequence, and we’ll use additional AI tools to help with initial ideation and to create text prompts to use at runway to help generate high quality video clips.
The main steps in the project:
- Generate an overview for an ‘immersive experience’ promotional video using perplexity.ai
- Generate still images using the flux-1.1-pro model at replicate.com to use as ‘first frame’ prompts at runway
- Generate short video clips using runway’s latest Gen-4 model
- Use runway’s online video editor to sequence the promotional video
- Use the speech-02-hd AI model at replicate.com to generate a realistic-sounding voiceover
- Use udio.com to generate a background music track
- Add text titles and export the video file
Part 6: Use Runway to generate an AI voiceover
Generative AI has been capable of producing ‘realistic’ human voices for a few years now, and AI-powered ‘text to speech’ can be found in many creative applications. In part one of this series we asked perplexity.ai to block out the main scenes for the video promo, and it suggested the bare bones of a script which can go with each short scene:
“What if you could step inside the story?”
“Welcome to [Experience Name]-where reality blurs with imagination.”
“Explore, interact, and lose yourself in a world designed to ignite your senses.”
“Every moment is unforgettable. Every step, a new adventure.”
“Tickets are limited. Don’t miss your chance.”
“Book now at [website]. Experience the extraordinary.”
We’ve manually refined the above suggestions to better match the style and content of the ad:
Where we’re going, we don’t need goggles…
Welcome to total immersion, where reality blurs with your imagination.
Lose yourself in a world created to ignite your senses
Experience the extraordinary. Experience the future.
This is reality remixed. Book now.
Step 1: On the Runway dashboard select ‘Generate Audio’:
Step 2: Try out the various available voices by selecting a category then playing back the audio examples provided from the available matches:
Step 3: Enter the text for the voice generation at the top left, then click the ‘Generate’ button:
Here’s the generated audio for Benjamin, as selected above:
To add a generated voiceover audio to the video project, open the video project then open the ‘Audio’ assets folder. Select the audio clip then drag it on to the timeline underneath the existing video track:
To time the narration correctly, it needs to be separated into individual sentences. Align the vertical playhead at the end of each sentence in turn and click the ‘Split layer at playhead‘ button as highlighted below:
Separate out the individual audio clips along the timeline and place them roughly where they need to be:
This is a rough cut. Here’s the current project exported to a 1080p video:
Despite trying many of the voices, we’re not completely happy with the result. We require a deep, gravelly voiceover. We revisited replicate and used the speech-02-hd model to generate a slower, deeper male voiceover. Dedicated audio models such as this can afford more control over final audio than Runway, so bear this in mind if your project includes lots of ‘text to speech’.
Here’s the voiceover audio file downloaded from replicate:
In the next part of this series we’ll add the updated audio to the video project.
