Most video platforms treat AI as a post-processing step. The meeting ends, the recording uploads, and some time later — minutes, sometimes hours — you receive a summary, transcript, or set of action items. This is useful, but it misses the fundamental opportunity: the intelligence should be available while the video is happening, not after.
V100 runs a complete AI intelligence pipeline alongside the live media stream. Every frame of video and every segment of audio passes through analysis in real-time. Highlights are surfaced during the meeting. Deepfakes are detected as they appear. Content quality is scored frame-by-frame. Viral potential is calculated as the content is created. All of this runs in Rust at sub-microsecond latency, in parallel with the media pipeline, without adding a single millisecond to the video delivery path.
Six Intelligence Layers Running in Parallel
V100's AI pipeline is not a single model. It is six distinct intelligence layers, each optimized for a different type of analysis, all running concurrently alongside the media server. Some use external AI models (Claude Haiku for natural language understanding). Others use custom Rust-native algorithms optimized for deterministic, sub-microsecond performance.
V100 AI intelligence stack
Layer 1: AI Highlights with Claude Haiku
V100 sends real-time transcript segments to Claude Haiku for semantic analysis during meetings. Haiku identifies five categories of meeting intelligence: questions asked (and whether they were answered), action items assigned (with the responsible person and deadline if mentioned), decisions made, key topics discussed, and overall sentiment shifts. These highlights stream to all participants via WebSocket as they are generated.
The practical impact is significant. A 60-minute meeting typically generates 15-25 highlights. Instead of reviewing a full transcript after the meeting, participants see a structured summary building in real-time. If an action item is assigned and the assignee is not in the meeting, V100 can send an immediate notification. If a decision is made, it is captured and timestamped before anyone forgets the exact wording.
// Enable real-time AI highlights for a meeting
const meeting = await v100.meetings.create({
name: "Q2 Strategy Review",
ai: {
highlights: true, // Claude Haiku analysis
categories: [
"questions", // Track questions asked
"action_items", // Capture assignments
"decisions", // Record decisions made
"topics", // Extract key topics
"sentiment", // Track sentiment shifts
],
webhook: "https://app.example.com/webhooks/highlights",
},
});
// Highlights stream via WebSocket during the meeting
// {
// "type": "action_item",
// "text": "Sarah to send revised Q2 projections by Friday",
// "assignee": "Sarah",
// "deadline": "2026-04-03",
// "timestamp": "00:14:32",
// "confidence": 0.94
// }Layer 2: Active Speaker Detection
Active speaker detection sounds simple: figure out who is talking. In practice, it is a surprisingly difficult real-time signal processing problem. Audio from multiple participants arrives with different latencies, noise levels, and gain settings. Crosstalk (two people speaking simultaneously) is common. Brief interjections ("mm-hmm", "yeah") should not trigger a speaker switch. The system must be responsive enough to track conversation flow but stable enough to avoid jittery switching.
V100's active speaker detection runs server-side in Rust using exponential smoothing with hysteresis. Exponential smoothing weights recent audio energy measurements more heavily than older ones, creating a smooth signal that tracks speaking activity without reacting to transient noise. Hysteresis adds a threshold gap between "become active" and "become inactive" states, preventing rapid toggling when someone is near the detection threshold.
The result drives multiple downstream features: automatic camera zoom to the active speaker, picture-in-picture layout switching, speaker timeline generation for post-meeting review, and speaking time analytics. Because the detection runs server-side, it works regardless of the client platform — web, mobile, desktop, or headless API.
Layer 3: Deepfake Detection on Live Video
Deepfake video is becoming increasingly convincing. In a video conferencing context, deepfakes present a serious threat: an attacker could impersonate an executive, a client, or a government official in a live call. Most deepfake detection operates on recorded video, analyzing it after the fact. By then, the damage — a fraudulent wire transfer authorized by a deepfaked CFO, a deal signed based on a faked identity — is done.
V100 runs deepfake detection in real-time on live video frames. The system does not use neural networks (which would add unacceptable latency and GPU dependency). Instead, it performs statistical analysis on the video signal itself:
Deepfake detection signals
When the system detects anomalies above a configurable threshold, it surfaces a warning to the meeting host or to all participants (based on the meeting's security policy). The warning includes the specific signals that triggered the detection and a confidence score. This is not a binary "fake or real" classification — it is a set of statistical anomaly flags that allow the human to make an informed judgment.
Layer 4: Content Analysis and Quality Scoring
V100 computes three industry-standard quality metrics on every video frame: SSIM (Structural Similarity Index), PSNR (Peak Signal-to-Noise Ratio), and VMAF (Video Multi-Method Assessment Fusion). These metrics quantify the visual quality of the video as perceived by a human viewer. They are essential for adaptive bitrate decisions, encoding optimization, and quality-of-experience monitoring.
Beyond quality metrics, V100 performs scene complexity analysis and motion estimation on every frame. Scene complexity determines the encoding difficulty — a static presentation slide requires far fewer bits than a fast-moving sports broadcast. Motion estimation tracks movement between frames and informs both the encoder (allocate more bits to high-motion segments) and the AI director (switch camera angles during high-motion moments).
// Enable real-time content analysis
const stream = await v100.streams.create({
name: "Product Launch Livestream",
analysis: {
quality_metrics: true, // SSIM, PSNR, VMAF per frame
scene_complexity: true, // Encoding difficulty score
motion_estimation: true, // Inter-frame motion vectors
deepfake_detection: true, // Statistical anomaly detection
viral_scoring: true, // 0-100 engagement prediction
},
});
// Real-time analysis webhook payload:
// {
// "frame": 14832,
// "ssim": 0.967,
// "psnr": 42.3,
// "vmaf": 88.7,
// "scene_complexity": 0.34,
// "motion_magnitude": 12.8,
// "deepfake_score": 0.02,
// "viral_score": 73
// }Layer 5: Viral Scoring — Predicting Engagement Before Publishing
V100's viral scoring engine generates a 0-100 engagement prediction for video content using six weighted factors: emotional intensity (detected from audio sentiment and visual expressiveness), novelty (how different is this content from baseline), shareability (does it contain quotable moments, surprising reveals, or humor), visual appeal (composition quality, lighting, color), audience relevance (based on the configured target demographic), and timing (is the topic trending, is the posting time optimal).
During live streams, the viral score updates every 5 seconds. Content creators can see in real-time which segments of their stream are likely to perform well when clipped and published to social platforms. The system can automatically clip high-scoring segments for review, or — with the multi-platform publishing pipeline — automatically publish them.
Architecture: Parallel AI Pipeline
The critical architectural decision is that the AI pipeline runs in parallel with the media pipeline, not in series. The v100-turn media server processes and routes video frames on the critical path. A separate set of Rust threads runs the AI analysis on copies of the frame data. The AI results are delivered via WebSocket and webhook — they never block media delivery.
Incoming Video Frame
|
|------ [CRITICAL PATH] ------> v100-turn (Rust)
| Decrypt > Route > 0.01ms processing
| Re-encrypt > Deliver < 50ms glass-to-glass
|
|------ [AI PATH] -----------> ai-orchestration (Rust)
(parallel threads, |
zero media impact) |--> Active Speaker (exponential smoothing)
|--> Deepfake Check (entropy + artifacts)
|--> Scene Complexity (spatial frequency)
|--> Motion Estimation (frame diff)
|--> Quality Metrics (SSIM/PSNR/VMAF)
|--> Viral Scoring (6-factor weighted)
|
|--> Claude Haiku (async, transcript-based)
| Questions, action items, decisions,
| topics, sentiment
|
v
WebSocket + Webhook delivery
(real-time to participants)
The on-device analysis layers (active speaker, deepfake, quality, complexity, motion, viral scoring) all run in Rust at sub-microsecond latency per frame. They do not require GPU acceleration. They do not call external APIs. They are deterministic, reproducible, and fast enough to process every frame at 60fps without accumulating a queue.
The Claude Haiku layer is asynchronous by design. Transcript segments are batched (typically 10-15 seconds of speech per batch) and sent to Haiku for semantic analysis. Haiku's response time (typically 200-500ms) is acceptable because highlights are informational — they enhance the meeting experience but do not gate media delivery. The highlights stream to participants as they are generated, with a natural 1-2 second delay from speech to highlight.
Performance: The Numbers
AI pipeline latency breakdown
| Layer | Method | Latency | Runs On |
|---|---|---|---|
| Active speaker | Exp. smoothing + hysteresis | <0.1µs | CPU (Rust) |
| Deepfake detection | Statistical analysis | <0.5µs | CPU (Rust) |
| Scene complexity | Spatial frequency analysis | <0.3µs | CPU (Rust) |
| Motion estimation | Frame differencing | <0.2µs | CPU (Rust) |
| Quality metrics | SSIM + PSNR + VMAF | <0.8µs | CPU (Rust) |
| Viral scoring | 6-factor weighted model | <0.1µs | CPU (Rust) |
| AI highlights | Claude Haiku (async) | 200-500ms | API (async) |
The total on-device AI processing is under 2 microseconds per frame. At 30fps, that is 60 microseconds per second of AI processing — consuming less than 0.006% of a single CPU core's capacity. The AI pipeline is essentially free in terms of compute overhead. This is the advantage of running native Rust algorithms instead of Python-based ML models: the analysis completes faster than it takes to copy the frame data to a GPU.
Why Rust Makes This Possible
The AI pipeline's sub-microsecond latency is a direct consequence of the language choice. Rust compiles to native machine code with zero runtime overhead. There is no garbage collector to introduce latency spikes. There is no interpreter to add overhead. There is no JIT compilation warmup. The first frame through the pipeline is processed at the same speed as the millionth frame.
Rust's ownership model also enables the parallel architecture. The media pipeline and AI pipeline share frame data through zero-copy references — the AI threads read the frame data without copying it, and Rust's borrow checker guarantees at compile time that this is safe. In a garbage-collected language, sharing data between threads either requires copying (adding latency) or careful synchronization (adding complexity and potential deadlocks). Rust eliminates both problems.
This is part of V100's broader 100% Rust architecture. All 20 microservices, including the AI orchestration service, are written in Rust. The entire platform — from API gateway to media server to AI pipeline — runs without a single line of garbage-collected code in the critical path.
What You Can Build With Real-Time Video Intelligence
AI meeting assistants: Build a meeting copilot that surfaces action items, tracks decisions, and generates structured summaries in real-time. The highlights API delivers these as WebSocket events during the meeting, not as a post-meeting report.
Content moderation: Deepfake detection and content analysis can power real-time moderation for live streaming platforms, video social networks, and user-generated content sites. Flag problematic content as it is created, not after it has been viewed thousands of times.
Automated video production: Viral scoring combined with the content analysis layer tells you which segments of a long-form stream are worth clipping and publishing. Combined with V100's transcript-based editing and multi-platform publishing, you can build a fully automated content pipeline from live stream to social media post.
Quality monitoring: SSIM, PSNR, and VMAF metrics give you objective quality measurements for every frame. Build dashboards that alert when quality drops below thresholds, automatically adjust encoding parameters, or provide end-of-stream quality reports for live broadcasters.
Limitations
Deepfake detection is statistical, not deterministic. The system identifies anomalies in video signals that are characteristic of deepfakes. It does not guarantee detection of all deepfakes, particularly high-quality ones that carefully match the statistical properties of genuine video. It is a defense-in-depth layer, not a single point of protection.
Claude Haiku highlights have inherent latency. The AI highlights layer is asynchronous and depends on an external API. Highlights typically appear 1-2 seconds after the corresponding speech. This delay is acceptable for meeting intelligence but is not suitable for applications requiring frame-synchronous NLP.
Viral scoring is predictive, not prescriptive. The 0-100 viral score is a prediction based on content characteristics. It does not account for platform algorithm changes, audience size, posting history, or external events that influence engagement. It is a useful signal, not a guarantee.
Add real-time intelligence to your video application
V100's AI pipeline runs alongside your video with zero latency impact. AI highlights, deepfake detection, content analysis, and viral scoring are all available through the API. Start a free trial and enable any intelligence layer with a single configuration flag.