Semantic search across podcast and educational video libraries — every word, every moment, every speaker, discoverable.
Podcasters and educators sit on hours of recorded content that's functionally unsearchable. A listener remembers what was said, but not which episode or what minute — so the content might as well not exist.
The usual workaround is metadata: episode titles, chapter markers, hand-written show notes. That holds up until a library grows past a few dozen hours, and even then it only surfaces what somebody thought to write down. The most useful two minutes of an interview are rarely the two minutes that made the title.
Keyword search over raw transcripts isn't much better. Transcripts are noisy, speakers phrase things differently than searchers do, and a word match tells you nothing about whether that moment is the one you wanted. Someone looking for “how they priced the first product” will never match a host saying “we charged forty bucks and hoped.” The information is right there in the audio. The index just can't see it.
A semantic search engine built on Marengo and Pegasus by Twelve Labs that indexes video and audio at the frame and transcript level — not just metadata. Users search by meaning (“the part where they talk about pricing strategy”) instead of exact keywords, and get back the specific moment, not just the episode.
Marengo covers the embedding side: one shared vector space for what appears on screen and what is being said, so a single query can match a spoken sentence, a slide, or both at once. Pegasus covers the language side — describing and summarising a clip rather than only pointing at it. What comes back isn't a list of episodes. It's a set of timestamped moments, each playable from the second it starts.
Running alongside that is live speaker diarization. As audio comes through, the pipeline separates it into speaker turns and labels who is talking, so a transcript stops being an undifferentiated wall of text and becomes a conversation with attribution. That distinction changes what a query can ask for: not just “where is pricing discussed,” but “where does the guest — not the host — talk about pricing.” Because diarization happens live rather than as an offline post-process, a stream is searchable by speaker while it is still going out.
Everything around those models is ours: ingestion, diarization, segmentation, ranking, caching, the API, and the interface that makes a result feel like an answer instead of a list of links.
Five stages, each one built so the stage after it has something precise to work with.
The model calls are the easy part. The engineering sits in everything around them.
Semantic search stops feeling like search somewhere north of a couple hundred milliseconds — people go back to scrubbing the timeline. Holding a median in the tens of milliseconds meant keeping hot embeddings cached and shaping the index around the queries that actually get made, not the ones that theoretically could be.
Segment too finely and you lose the thread of an argument; too coarsely and you hand back a ten-minute block, which just moves the searching back onto the user. Overlapping windows plus a re-ranking pass let segments stay small without losing the context that makes them make sense.
Separating speakers is hard enough offline, with the whole file to look at. Doing it live means committing to a decision about who is talking from a short window of audio, while people interrupt each other, laugh over each other, and join mid-sentence — and doing it fast enough that the labels land before the segment is indexed.
Raw vector similarity will cheerfully return four near-identical moments from the same episode. Deduplication and diversity rules across the result set are what make three results feel like three answers instead of one answer repeated.
Content that existed but couldn't be reached is now reachable. Someone half-remembering a conversation can describe it in their own words and land on the exact passage, in the right episode, at the right timestamp — instead of giving up two minutes into scrubbing.
For the people who own these libraries, the effect is that a back catalogue stops behaving like an archive and starts behaving like a product surface. Episodes published months ago keep getting pulled up by searches, because relevance no longer depends on how recent something is or how well it was tagged on the day it shipped.
None of that changed the publishing workflow. New material is indexed as it lands, and the search surface stays current without anyone maintaining a taxonomy by hand.
Podcast and educational search is what's running today. The same indexing infrastructure extends naturally to the rest.
Already working for real clients, every day. Not a pitch, not a promise.