Respeecher CEO is working on Star Wars, Boba Fett Luke’s book, and more

If you’ve ever watched a Star Wars show on Disney Plus, chances are you know the work of a tech startup Speech, whether you realize it or not. The Ukrainian company’s AI-powered voice cloning platform provided Mark Hamill’s aged vocal performances in both The Mandalorian and Boba Fett’s Bookas well as for an as yet unidentified character in Obi Wan Kenobi. Lucasfilm has asked Respeecher to keep this character’s name a secret for now – and with so many franchise veterans returning for the series, there’s certainly no shortage of potential candidates.

Polygon sat down with Respeecher CEO Alex Serdiuk to better understand a process that, for many fans, undoubtedly borders on sacrilege: using technology to create completely bespoke performances for one (or may -be even two) of the most emblematic characters of the Star Wars saga. Right off the bat, Serdiuk emphasizes the human element behind the platform itself. “Through our technology and services, we can create a digital copy of a particular voice and allow another person to speak with that voice,” he explains. “And so we allow [studios] to evolve voices, age them and even resurrect voices for certain projects.

So, far from the mental image conjured up by terms like “artificial intelligence” and “voice cloning” – that of a sound engineer running lines of dialogue through a computer algorithm that then spits out audio files – the work from Respeecher on Star Wars is surprisingly well driven. While Darth Vader himself may be more machine than man, if the Ukrainian startup provides his voice (and remember, we said “if”), the essence of the character’s voice is still In the flesh.

Picture: Lucasfilm

“There’s no AI yet, and I don’t believe it would exist, it would allow us to just use it on a turnkey basis to create the performance we want to create. […] We need another human voice [to provide] because that human voice gives all the inflections, accent, speech style and rhythm that AI is not good at creating,” Serdiuk insists. He adds: “Our system requires a performance at the entrance, so it can be done by the same person who is aging for example, or someone else. […] It takes all the performance, all the acting of what we call a “source voice,” and then we do the conversion.

Additionally, Respeecher’s pipeline allows actors like Hamill to record different takes as they would on an actual set, which the company’s experts can then adjust at their end based on notes from showrunners like Jon Favreau or directors like Deborah Chow.

“With studio projects and films, they could record thousands of takes for each line, and that means we would need to convert all those [into the younger voice], send them back and maybe send different versions, because we used to train different models with different setups,” he says. “And sometimes we also have to meet creative expectations so they can just direct us, Can you try to do [a line reading] sound a bit more like that? and we would work to make it look a little more like what they are asking for.

Thus, the ultimate goal is to recreate a traditional representation by non-traditional means — in the case of The Mandalorian Where Boba Fett’s Bookas if Hamill’s lines were somehow carried straight from the 1983 set Star Wars: Return of the Jedi? “The goal is to give the impression that it was recorded yesterday in the studio by the target voice itself”, confirms Serdiuk.

Of course, there’s always the risk with voice-cloning technology like Respeecher’s that the synthesized performance it produces sounds contrived, even if viewers aren’t quite sure why. Serdiuk even admits that Hamill’s outdated voice sounds “much, much better” in Boba Fett’s Book than he did in The Mandalorian, thanks to minor but significant improvements in how Respeecher’s AI model was “trained” to emulate the actor’s voice. At the same time, the CEO is also quick to point out that while many fans have complained about the visual effects used to portray 20-year-old Luke Skywalker in The Mandalorian finale, few realized the Jedi Master’s voice was also synthetic until Lucasfilm spilled the beans in a Disney Gallery “making of” special a few months later.

Serdiuk mentions that Hamill’s convincing aging voice in The Mandalorian and Boba Fett’s Book were all the more impressive considering the quality of legacy assets Respeecher received from Lucasfilm. “This [data] was quite old, so we had something from tapes, we have an old ADR recording, something from a video game,” Serdiuk recalls. “And the thing is, you have to train that data in your model to allow it to produce the output quality that would fit into a modern production. In many projects that involve aging or resurrecting [performer’s voices]this could be the main challenge as the lack of data and the quality of the data introduces additional hurdles in terms of sound quality.

The Respeecher CEO argues that overcoming these data hurdles was worth it, now that industry heavyweights like Lucasfilm have embraced the work they do. “[We] started with the idea of ​​constructing a synthetic discourse [platform] at the level where it would go through the sound engineers and the Hollywood studios and land in the big productions. So when they accept our sound, when they say something good about the sound that we’ve been able to produce — and it’s a very complicated and cumbersome technical challenge, to do synthesized speech at a level where it would be impossible to distinguish it from a real recording — in such cases, it really encourages us and helps us grow,” he says.

I told Serdiuk that the growing acceptance of voice-cloning technology could mean that studios are also no longer bringing in talented voices to replace deceased actors. For example, wouldn’t it Rogue One: A Star Wars Story – in which Guy Henry impersonated the voice of the late Peter Cushing as Grand Moff Tarkin – would be the kind of project that would automatically land on Respeecher’s desk now? Not necessarily, according to Serdiuk, who sees Respeecher’s voice-cloning technology as one of many viable options available to filmmakers.

Rogue One - Governor Tarkin / Death Star

Governor Tarkin looks at the Death Star
Picture: Lucasfilm/Disney

“There are always different visions of how things should be in the industry and fans have different ideas of how they should be. I wouldn’t say that. [Respeecher] very suitable for directing or judging [which approach is best],” he says. Serdiuk also clarified that if Lucasfilm ever calls on Respeecher to recreate the voice of a deceased actor, the company will only do so with the approval of that actor’s estate. And while the response de Serdiuk will put voice actors at ease for the time being, the fact that Lucasfilm is a regular customer of Respeecher suggests that the startup has made serious strides in voice cloning technology.

Indeed, Serdiuk already has a vision for Respeecher’s future that goes beyond aging voice actors, though he remains adamant that what the company has planned will expand filmmakers’ creative horizons, and no will shrink them. He talks about democratizing the technology so that smaller film and television studios and video game developers can use it to further stretch their budgets. He also speaks enthusiastically about Respeecher’s platform’s groundbreaking healthcare apps — even citing a case where the company collaborated with a voice actor who lost his voice to allow him to perform again.

Still, looking ahead doesn’t mean Serdiuk has lost sight of what it means for Respeecher to be part of a certain space opera set long ago in a galaxy far, far away. “It’s something special. I mean, you are part of this story. We can say that Star Wars has advanced Hollywood a lot, can’t we? They’ve been disrupting the industry, from a technical standpoint, from the very beginning, and the way they make their films is exceptional. So it’s a great honor to be able to work with these people and learn from them.

Comments are closed.