AI Can’t Caption

PLUS: Getting Black & White Right in "Ripley"

Hola Hollywood tech nerds!

In this week’s post:

Subscribe to get Hollywood Tech Nerds magically delivered to your inbox every Tuesday!

AI Can’t Caption

Back in February I wrote about the ways that the entertainment industry continues to underserve the deaf and hard of hearing, a large audience whose content accessibility is often shockingly behind the times.

IndieWire recently ran a piece about a related issue: the importance of the “human touch” on captioning work and how that can get lost with AI-generated transcription. I’ve seen a lot of focus on that in the business, particularly over the past few years. Everyone wants the cheap option. Why pay for a human to create captions when an auto-transcription program like Whisper can do it for free?

“It’s frustrating when you’re reading captions and there are commas there for no reason,” said Antoniou. “Voice recognition will hear a pause and just immediately add a comma. It breaks up the flow of how you’re meant to read and understand. It happens all the time that you can tell something has been captioned by a program that doesn’t comprehend everything that’s happening.”

That lack of comprehension is the chief concern for the push towards automation just for the sake of speed and money. AI can indeed transcribe some human speech surprisingly well. However, that doesn’t mean that the transcript competently evokes everything happening onscreen. A producer certainly wouldn’t want to skimp on the sound mix and score for a hearing audience, but sometimes it seems treating the deaf and hard of hearing as though they are disposable or irrelevant and not in need of a quality viewing experience is surprisingly less important.

As the article notes:

Bad captions can certainly undo an artist’s intentions. That’s why Shoshannah Stern is so invested in the captions on her projects. A deaf artist herself, she co-created and starred in “This Close,” a comedy about two deaf friends that premiered on Sundance Now in 2018. “I was very involved with the captioning process,” she said. “We were switching in between sound perspectives in Season 2, in an episode where [my character]’s hearing aids go out. When we got the cut, the captioner had written ‘NORMAL SOUND RETURNS.’ It was so interesting to think about, because normal is a subjective word.”

Stern inserted her artistic perspective into that caption. “I remember choosing to use ‘AMPLIFIED SOUND COALESCES’ instead, because it felt so specific to the experience of having your hearing aid turned on again. When I saw those words on screen, it felt exhilarating and really representative of our narrative.”

“We’ll just have AI do it” is not only disrespectful to the deaf and hard of hearing audience, but to the work’s creators and their intentions. Audiences with accessibility issues should be considered just as important as everyone else!

Getting Black & White Right in Ripley

Netflix’s limited series Ripley was one of my favorites from the past year. One of its many qualities is its remarkable black and white cinematography by the legendary Robert Elswit.

VFX Voice has a fascinating and thorough breakdown of how the show achieved its visuals, utilizing a staggering amount of visual effects. A few tidbits from the piece:

It was shot entirely in color, with visual effects also being created in color before converting them to black and white.

VFX Producer Joseph Servodio notes “…what made the most sense was to work on the shots in color, then our team presented them to Steve, predominately within cut context in our black and white look. Occasionally during reviews, we would flip back to the color look just as a quality check. Since black and white tended to be more forgiving, we would sometimes be able to see flaws in the color that we otherwise didn’t catch in the black-and-white viewing.”

Ripley has over 2,000 visual effects shots!

“There were 2,146 visual effects shots in total, most of which consisted of train windows, bus windows and apartment windows, as well as environment extension work.”

Wētā FX worked on action sequences, including a section on a boat that was almost entirely digitally-created.

Wētā FX’s work for the show included two types of shots: the boat action sequence, where each shot had a unique action, and a series of shots in a static location on the water. “For the action shots, our animation team crafted dynamic boats that could simulate motion through the water,” White says. “For the static shots, we invested in an automatic setup. Once the look of the sequence was established, we could run multiple shots through a setup and tweak each shot to taste. The setup was crafted as a primary lighting and comp scene, establishing the base look. The digi-double shots were the most challenging, but they also brought the most satisfaction. They needed to be spot-on likenesses to the actors, with the complexities of underwater clothing simulation, bubbles, hair motion and lighting while preserving detail. The way light exits skin underwater differs from in the air, so particular attention had to be paid to underwater digi-double rendering.”

Check out the full article for all the details on the look of this terrific show. Mild spoilers contained therein, so if you haven’t watched it yet, do that first and then read!

Here’s a round-up of cool and interesting links about Hollywood and technology:

What do the Bungie layoffs portend for the gaming industry? (link)

Adobe’s Photoshop AI tools are a hint about its future plans for video. (link)

AI is becoming a technology of faith. (link)