Embeddings

Pure Frame’s two search modalities work differently under the hood: visual search uses vector embeddings, while transcript search uses full-text search. Understanding the difference explains why hybrid search combines both.

Visual embeddings

Every extracted frame is encoded by SigLIP (google/siglip-so400m-patch14-384), producing a 1152-dimensional vector that captures the frame’s visual content. These vectors are stored in a vector database, each tagged with user_id, video_id, and timestamp for isolation and retrieval.

When you submit a text or image search query, Pure Frame encodes it with the same model and finds the nearest frame vectors — this is what makes queries like "whiteboard with a diagram" or a reference image work without exact keyword matching.

Transcript matching (not embeddings)

Spoken audio is transcribed by Whisper (medium) into timestamped speech segments, stored in Postgres. Transcript search does not use vector embeddings — it uses Postgres full-text search (websearch_to_tsquery), a lexical match against the transcribed words.

This matters practically: transcript search rewards queries that share actual words with what was said ("pricing objection" matches a segment containing those words), while visual search rewards conceptual/visual similarity even with no shared vocabulary at all.

Why hybrid search works

Because the two modalities are fundamentally different techniques, they catch different things:

• Visual embeddings find moments by what’s shown — no dependence on anyone talking
• Full-text transcript search finds moments by what’s said — precise on wording, even if the visual content is static

When both modes are active (the default), Pure Frame runs both searches in parallel and merges the ranked results with Reciprocal Rank Fusion (k=60), so a segment can surface because it matches visually, because of what was said, or both.

Model versions

Each video’s model_version field records which embedding model indexed it. When Pure Frame ships a new visual encoder, existing videos can be re-embedded via the frame data already stored — without re-running Whisper transcription — since transcription and visual embedding are independent stages.