Streaming Video-LMM Is Not One SOTA Line: Six Development Paths from 2023 to 2026

Series guide. This is the development-history hub for my streaming Video-LMM notes. For a stage-by-stage systems view, see the pipeline survey. For dataset and protocol details, see the benchmark index. The complete reading map is at the end of this article.

Calling a model streaming no longer identifies one technical problem. One paper may process a causal video prefix but retain every historical KV block. Another may keep a fixed cache but permanently forget old evidence. A proactive assistant may know the user’s request before the stream begins, while a delayed question-answering system must decide what to preserve before any question is known. A benchmark may measure the age of remembered evidence without measuring whether the implementation can keep up with a live camera.

This post therefore does not arrange papers as a single SOTA ladder. It follows six intersecting development paths from 2023 to 2026 and asks how the problem definition changed along each one. Before the timeline, four distinctions are needed: when the query becomes visible, which clock a latency number belongs to, which resource is actually bounded, and what form of evidence can still be recovered after compression.

Sources, versions, and publication status in this overview were checked against first-party records on 2026-09-02. Formal proceedings are preferred when available. At that cutoff, MovieChat+, Flash-VStream 2024, Dispider, SAVEMem, CoRDS, NovaCov, CARVE/V-RAGBench, FOLIO, ObjectStream, ThinkStream, RIVER, LyraV, SPOT-Bench, Streaming-Eval, SimpleStream, StreamArena, StreamOPD, and StreamScout remained preprints. LiveVLM v2, StreamMem, VST, ProtoKV, and MERIT had acceptance evidence recorded in the source ledger but not a final proceedings page there. Recent work is used as frontier context rather than as the sole basis for a stable conclusion.

This hub does not compare performance numbers across papers. A number is only meaningful with its backbone and version, sampling rate, write/read budget, hardware, query timing, judge, and exact evaluation protocol; those details belong in the focused paper notes and benchmark index.

First Fix the Setting

Four query-visibility regimes

These regimes are not interchangeable controls. Query-known retention is a useful oracle for an unknown-future-query system, but it does not show that a query-agnostic policy preserved the same evidence. Likewise, persistent-query assistants can build task-specific state from the first frame; that is a different burden from maintaining a reusable index for arbitrary future questions.

Three clocks

Streaming papers often use latency for three different quantities:

A method can do well on one clock and fail on another. For example, a benchmark may pause playback at a question point, giving the model unlimited computation while still calling the protocol streaming. Conversely, a low-latency recent window can be an excellent live perception system while being unable to answer questions about an event from an hour earlier.

Four resource boundaries

The phrase bounded memory also needs an object. This series accounts for at least four stores separately:

Thus, a capped prompt, fixed GPU cache, or bounded query-time top-$k$ does not by itself prove $O(1)$ total storage. Throughout this overview, model-state bounded, active-bounded/archive-grows, request-bounded/store-grows, and recent-only bounded describe different system contracts.

Here, model-state bounded means only that retained video-derived model state has a configured length-independent cap. It does not imply that text or dialogue KV, the raw input stream, application logs, or external evidence stores are also bounded.

Memory carriers determine what can be recovered

The carrier is therefore part of the scientific claim. A memory method should be evaluated on the facts its representation can still express, not only on the number of tokens passed to the final language model.

The Six Development Paths

1. A. Long video to native streaming — Interface: causal encoding and scheduling. The path starts with too many frame tokens for a complete video and turns toward causal, incremental encoding interleaved with dialogue. It leaves open whether finite-clip training remains stable during indefinite, concurrent ingestion and generation.
2. B. Growing archives to bounded online state — Interface: memory write and retention. Exact historical KV is reusable but grows linearly, so the write policy must merge, evict, summarize, or maintain fixed-capacity state before a future query exists. The unresolved question is which future retrieval behavior each irreversible decision destroys.
3. C. Flat tokens to structured state — Interface: memory carrier, write, and index. Event hierarchies, entity records, object slots, graphs, and prototypes give history an address structure. The risk moves to identity drift, boundary and merge errors, OCR, and rare events.
4. D. Single-key to verifiable multi-key RAG — Interface: index, read, and rehydration. Multiple semantic and temporal access paths plus source backlinks make retrieval more testable. The open cost is how much growing index and source storage independently verifiable recall requires.
5. E. Passive QA to proactive interaction — Interface: trigger, answer, and scheduler. A system must learn when to stay silent, when evidence is ready, and how to watch, think, and speak concurrently. Architecture, streaming post-training, and strong recent perception remain confounded.
6. F. Accuracy points to streaming systems — Interface: evaluation harness. Continuous outputs, readiness, asynchronous producers/consumers, and hour-scale interaction expose timing and recent-frame shortcuts. No one score yet captures memory, timeliness, throughput, and evidence failure.

The paths intersect, but they should not be collapsed. A structured memory may still have a growing external store. A proactive system may use only recent frames. A fast compressor may reduce visual prefill without bounding the downstream cache. The rest of the article follows these interfaces rather than the labels chosen by individual papers.

The arrows in these path names describe a change in the problem being asked, not an assumed citation or implementation lineage. I use direct code reuse only when an official source establishes it, as the LiveVLM repository does for ReKV. Elsewhere, chronological neighbors are described as problem inheritance or conceptual adjacency unless the papers provide stronger evidence.

A Compact Timeline of Problem Changes

The dates below are first-public dates, not venue years or dates of later revisions. Chronology establishes when a problem became visible; it does not by itself establish direct technical lineage.

In this curated reading set, the timeline is easier to read as three eras. The selected 2023–2024 papers primarily make long video processable and then causal. The 2025 and early-2026 papers increasingly study reusable KV, fixed active budgets, and specialized write/read policies. By mid-2026, more of the selected work treats structured state, evidence retrieval, response timing, and system counterfactuals as explicit evaluation objects.

Path A: From Long-Video Compression to Native Streaming

The first path changed the input/output contract. Early long-video systems usually assumed that a finite video already existed. Their main problem was representational: too many frames created too many visual tokens. MovieChat introduced dense short-term and sparse long-term visual memory, while MA-LMM made a fixed latent bank update recurrently. These are important precursors, but their sequential processing should not be confused with a live service. The actual user question does not guide their memory write, and merged evidence has no recovery path.

Three adjacent 2023–2024 branches sharpen that boundary. LLaMA-VID compresses each frame to a user-input-conditioned context token and a content token, drastically reducing per-frame input while leaving total tokens proportional to video length. It is therefore not an unknown-future-query retention result. Streaming Dense Video Captioning uses a fixed clustering memory for causal dense captions, an important task-level streaming precursor rather than a general delayed-query dialogue system. The MovieChat+ preprint makes consolidation question-aware, demonstrating the value of a known query but placing it in a less restrictive regime than unknown-future-query retention.

Streaming Long Video Understanding with Large Language Models made the write/read separation clearer. It propagates a state from clip to clip, saves historical snapshots, and selects a bounded subset after a question arrives. That enables reuse across questions, but only the active answer input is bounded: the snapshot archive and its indicators still grow with time.

VideoLLM-online changed the contract more deeply. Frames and language events are interleaved on a causal timeline; the model learns both when to produce language and when to remain silent. Streaming therefore becomes a training and scheduling problem, not merely a wrapper around an offline encoder. Later papers in this reading set explored multi-round memory, proactive triggers, full-frame-rate perception, and offline-to-online adaptation.

One naming collision matters here. StreamChat: Chatting with Streaming Video is the 2024 Liu et al. preprint, built around trained cross-attention and refreshed visual context during decoding. The existing StreamChat note covers the distinct Xiong et al. ICLR 2025 paper, formally titled Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge, which uses a hierarchical memory system and introduces StreamBench. Their shared short name is not evidence of one paper replacing the other.

StreamingVLM later aligned short-window streaming training with inference and kept attention sinks plus recent visual/text KV for stable continual decoding. This is a meaningful answer to long-running model state, but its contract is recent-only bounded: evicted visual evidence cannot be recalled. Infinite runtime is therefore not infinite recall.

Several bridge nodes complete this path without requiring a separate digest for each paper:

The two Flash-VStream entries must remain separate: the 2024 preprint uses STAR memory and VStream-QA, whereas the ICCV 2025 paper introduces a different Context Synopsis/Detail Augmentation architecture. They share a project lineage and short name, not one arXiv version history.

What remains. Native causal ingestion, sensible silence, and stable generation explain how an assistant can remain online. They do not guarantee that an unknown future question can recover old evidence, nor that decoding will never block ingestion under a sustained producer rate.

Path B: From Growing KV Archives to Bounded Online State

Once a Video-LMM produces KV causally, a tempting solution is to preserve all of it. ReKV makes this design explicit: write exact per-layer video KV blocks with a sliding causal window, offload them, then retrieve and reload relevant blocks after the question arrives. The model does not have to re-encode the video for every question, and the GPU read set can be controlled. The cost is a linearly growing RAM/disk archive and index, plus position reconstruction and I/O concerns. The expanded ReKV note audits that storage contract in detail; the bounded-memory survey compares it with fixed-state successors.

The next turn moved irreversible selection into the streaming write path:

The current LiveVLM mechanism is its substantially revised v2/DAC 2026 form; metrics or descriptions from the earlier v1 should not be silently mixed with Vision Sink Bucketing and Position-agnostic KV Retrieval. Its official code states that it is based on ReKV, which supports a direct implementation/code-reuse relation. For most other rows, the safer description is problem inheritance or conceptual adjacency rather than a claimed implementation lineage.

A parallel 2026 preprint branch changes the selection objective. SAVEMem uses fixed pseudo-questions as a semantic retention prior; CoRDS treats retained K/V as a representative and diverse coreset; NovaCov scores marginal set coverage against a bounded historical reference bank. These are distinct objectives. In particular, NovaCov bounds its reference bank, not the total downstream history created by selecting tokens for every frame.

What remains. Temporal redundancy, value norm, generic prompts, attention, geometric coverage, and prototypes are all proxies for unknown future utility. None alone proves that a fixed state preserves the evidence ranking or recall that a future query would obtain from the full historical index.

Path C: From Flat Tokens to Structured Long-Term State

A flat token or KV pool can be compact without being addressable. It does not naturally say which event occurred, which person or object changed state, or whether two similar observations were separate episodes. Structured-memory work changes the unit of history, but also moves failure into segmentation, linking, consolidation, and routing.

StreamForest organizes history as persistent event-level trees while retaining a fine-grained recent window. The turning point is not simply better compression: event boundaries become part of the memory address. If boundaries or merges are wrong, however, a rare detail can be buried inside the wrong branch.

OASIS turns an event hierarchy into an on-demand read policy. It stores summaries, embeddings, keyframes, and recent context; a query begins with a small context and can descend through the hierarchy when more evidence is needed. This bounds the work of one request, not the lifetime event tree or evidence store. A wrong high-level summary can also hide an otherwise correct lower branch. The detailed OASIS note covers that read path.

Two 2026 preprints expose a useful design split. FOLIO writes readable entity/action/state records and keeps visual-evidence backlinks, making state chains more inspectable but allowing the record and evidence stores to grow. ObjectStream instead maintains fixed latent object, change, and recent-state slots. The latter is easier to cap at the model-state level, but identity drift and slot merging can erase small objects, OCR, exact order, or repeated occurrences. ProtoKV supplies a third point in the space: fixed statistical prototypes rather than a readable event or entity store.

What remains. Semantic external memory is easier to inspect and can preserve source backlinks, yet it usually grows and depends on a reliable writer and router. Fixed latent state is easier to budget, yet is less reversible. Structure changes where errors happen; it does not remove information loss.

Path D: From One Retrieval Key to Verifiable Multi-Key Video RAG

Caption retrieval is an attractive long-term baseline because text is cheap to store and compare with a language query. rLiVS combines a bounded recurrent visual state with a searchable caption history. That separation is useful, but any fact omitted by the caption writer—small text, an exact count, a background object, a failed action, or precise event order—cannot be recovered from the text index alone. The visual state may be bounded while the caption store grows.

Several systems widen the address space in different ways:

• AdaVideoRAG routes a query among text, visual, and graph databases according to the estimated problem type. It is an offline long-video RAG reference, not a native causal-streaming result, and all three databases grow with the source.
• ViG-RAG adds temporal and semantic graph relations, making time and confidence part of retrieval. Its whole-video graph and indexes precede the query, so it is also an offline long-video reference. Graph construction, retrieval, source selection, and answer generation remain separate possible failure points.
• OASIS and StreamRAG bring event construction and routed reads into an online write process. A bounded read still sits over a growing event, caption, or evidence store.
• MERIT gives one episode multiple semantic keys and expands temporal neighbors only around hits. Its episodic memory is built query-agnostically before an unknown query, not demonstrated here as a native online-write pipeline. Multiple keys improve addressability without making the episode store fixed or guaranteeing that generation used the retrieved source faithfully.

The long-video preprint V-RAGBench and CARVE uses query–evidence–answer structure to separate what should be retrieved from how retrieved context is used. It is evidence for independently testing RAG, not for native streaming ingestion. That separation is more diagnostic than answer accuracy alone: a model can guess the answer from language priors after retrieving the wrong clip, or retrieve the right clip and fail during generation. The very recent StreamScout preprint extends the open question from what to read to how deeply to read, escalating from a recent glance to broader historical search. Its maturity makes it a frontier direction, not yet a stable conclusion.

What remains. Multi-key and adaptive retrieval increasingly bound evidence read per request, not total retained information. A convincing system needs source backlinks, evidence-level recall/ranking metrics, an explicit read budget, and separate accounting for its index and raw evidence store.

Path E: From Passive QA to Readiness, Thinking, and Near-Full-Duplex Interaction

Reactive streaming QA assumes that a question arrives at a predefined point and the model answers immediately. Interactive assistants need more states: remain silent, trigger a response, wait for future evidence, update reasoning while watching, or continue ingesting frames while language is generated.

VideoLLM-online provides one causal-interleaving formulation. Later papers in this reading set explore other interfaces: the Dispider preprint separates perception, decision, and reaction, allowing a lightweight component to monitor the stream while a larger module responds. StreamBridge made the activation model explicit while adapting an offline Video-LMM to proactive, multi-round use. These systems introduce a new failure axis: a trigger can be wrong even when the eventual answer would be correct.

The ICCV 2025 StreamMind paper explores a neighboring event-gated, full-frame-rate dialogue branch. It should not be confused with the different two-tier StreamMind system later reported inside StreamArena.

Memento studies an all-day proactive assistant with a request registered before the stream and uses that request to shape dynamic memory. This is a strong persistent-query result, not an equivalent test of unknown-future-query retention. The Memento note discusses that distinction in more detail.

StreamReady then makes evidence sufficiency measurable. A registered request waits for later evidence, and Answer Readiness Score penalizes responses that are too early or too late. This targets $\Delta_{ready}$ in video time. It does not show that a service has low $\Delta_{sys}$, and its query-known filtering is an oracle relative to query-agnostic retention.

Streaming reasoning forms another branch. VST updates reasoning while the video plays, while the parallel ThinkStream preprint follows a Watch–Think–Speak loop with an evolving reasoning state. Both can move work out of the final response burst; neither automatically supplies a verifiable visual evidence chain. Reasoning-compressed history can preserve an early misinterpretation just as a visual compressor can preserve the wrong token.

LyraV, a preprint at the cutoff, moves scheduling toward frame-token synchrony, controlling whether to continue an answer, begin another, or remain silent while pacing generation across incoming frames. Near-full-duplex is the conservative description: a general, backlog-free full-duplex service still requires matched producer-consumer measurements.

What remains. “Proactive,” “ready,” “thinking,” and “full-duplex” name different abilities. They need separate trigger, evidence, reasoning-fidelity, queueing, and latency tests rather than one interaction score.

Path F: From Query-Point Accuracy to Streaming-System Evaluation

StreamingBench evaluates timestamped questions on causal video prefixes. OVBench separated past, current, and future understanding, while the independent OVO-Bench organized backward tracing, real-time understanding, and forward active responding. These are important causal evaluations, but an aggregate can still be dominated by recent perception and usually does not reveal wall-clock backlog.

The next wave decomposes the measurement:

Several further collisions should remain explicit. The standalone StreamingBench, StreamBench inside the Xiong et al. StreamChat paper, and the 2026 StreamArena preprint are three different objects; StreamBench has no independent paper identifier. ACL 2026 StreamingEval: A Unified Evaluation Framework towards Realistic Streaming Video Understanding is not the hyphenated Streaming-Eval component in the preprint Harnessing Streaming Video in the Wild. Likewise, the StreamMind system reported with StreamArena in 2026 is not the ICCV 2025 paper StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition. Finally, SPOT-Bench belongs to Don’t Pause! Every prediction matters in a streaming video, while LyraV belongs to the separate Don’t Pause: Streaming Video-Language Synchrony for Online Video Understanding; the similar titles do not establish a shared method lineage.

The strongest evaluation is therefore a suite, not a synthetic leaderboard. It should pair a memory-science protocol stratified by $\Delta_{mem}$ with a producer-consumer harness reporting input rate, sustainable processing rate, queue depth, frame age, dropped frames, and p50/p95/p99 latency. Storage must be reported for active GPU state, the searchable index, archives, and retained raw evidence. Differences in backbone, sampling rate, budget, hardware, query timing, and judge version prevent most cross-paper numbers from being strict rankings; the benchmark index keeps those protocol details separate.

What remains. No single benchmark in this reading set simultaneously tests strict arrival-time causality, hour-scale multi-evidence recall, grounded retrieval, full-duplex ingestion and answering, fixed end-to-end storage, backpressure, and tail latency. Claims should name the subset actually tested.

A Cross-Cutting Counterfactual: Architecture or Training?

Complex memory should not be credited for gains that a stronger current-frame model or streaming-specific post-training can produce. The SimpleStream preprint deliberately provides a recent-only control with a small frame window. The later StreamOPD preprint keeps that recent-only inference path unchanged and changes the post-training recipe; its optional ST-CueGate uses teacher-only spatio-temporal cues to reweight training, not to add a deployment-time memory module.

Together they motivate a matched $2\times2$ control rather than the conclusion that long-term memory is unnecessary:

The backbone, frame policy, recent exact window, question timing, and evaluation protocol must be held constant. Only then can an experiment distinguish memory architecture from training and recent-perception quality. My current SimpleStream note is intentionally a short control summary; F7 will expand both sides of this comparison.

My Takeaways

1. Streaming is a contract, not an adjective

A useful method card begins with query visibility and arrival-time causality, not with the word streaming in the title. Persistent-query monitoring, query-known readiness, unknown-future-query recall, and offline long-video RAG answer different scientific questions.

2. Write, read, and archive policies should be audited separately

ReKV makes the read set small while its exact archive grows. Fixed-state methods make a harder write-time decision but cannot recover deleted evidence. RAG systems may make one read small while their indexes and source stores grow. One number called “memory” hides all three policies.

3. Recent perception and distant recall need different channels

A fixed high-fidelity recent window is a strong control and often the right path for current perception. Far history should be retrieved or represented at lower bandwidth only when needed. Mixing all history and the current frame in one attention pool can increase cost and interference without preserving exact old evidence.

4. Structured and latent memory trade different kinds of error

Event and entity records are readable and can point back to evidence, but their writers and routers can hallucinate, merge identities, or hide branches. Latent KV, slots, and prototypes are easier to keep compact, but they are opaque and usually irreversible. A hybrid is promising only if each interface is evaluated independently.

5. Evidence retrieval should become a first-class metric

Answer accuracy cannot reveal whether a method preserved the right history or guessed from language priors. Evidence Recall@$k$, ranking quality, temporal overlap, and agreement with a full-history oracle make the memory layer more falsifiable. They also separate writer failure, retrieval failure, and generation failure.

Open Hypothesis: Preserve Future Query-to-Evidence Behavior

The most interesting open problem, in my view, is not another token-importance score. It is this:

Given a single-pass video stream, an unknown future query distribution, and a fixed online index/evidence budget, maintain a coreset that preserves as much as possible of the full-history index’s future query-to-evidence behavior.

“Behavior” should be operational rather than metaphorical: top-$k$ membership, ranking order, boundary margin, evidence recall, and temporal localization for held-out future queries. The bounded structure may store several keys per episode and sparse backlinks to high-risk visual evidence, but the budget must charge for the keys, backlink metadata, and the retained frame/clip bytes those backlinks address. A small pointer does not make its evidence store free or bounded.

This hypothesis is falsifiable. Under a matched backbone, FPS, recent exact window, training recipe, and legal query-arrival protocol, compare the bounded index with a full-history oracle. If a new selection objective improves visual coverage but fails to preserve retrieval agreement or long-delay answer quality, then it has not solved the proposed problem. If it preserves retrieval but makes online updates, I/O, or tail latency unsustainable, it has not solved the streaming-system version either.

This is deliberately presented as a research direction, not a consensus result. Semantic priors, K/V coresets, set-wise visual coverage, multi-key records, and adaptive reading already cover important neighboring territory. The remaining claim must be narrower and tested against those baselines.

Series Map

Overview maps

• Long Streaming Video Understanding Pipeline — architecture and pipeline stages.
• Benchmarks for Streaming Video Understanding — evaluation settings and protocol-specific numbers.

Memory, KV cache, and compression

• ReKV
• StreamKV
• InfiniPot-V
• StreamMem
• LiveVLM
• StreamingTOM
• STC: hierarchical token compression
• MuKV
• WeaveTime

Retrieval and structured state

• rLiVS
• AdaVideoRAG
• ViG-RAG
• OASIS

Native streaming, interaction, and controls

• StreamChat: Xiong et al., ICLR 2025, arXiv 2501.13468
• StreamingVLM
• Memento
• SimpleStream
• Qwen3-VL: backbone note

Thematic articles

• Bounded streaming-memory evolution — published in F2.
• Set-wise streaming-memory selection — planned (F3).
• Structured long-term state — planned (F4).
• Verifiable multi-key Video RAG — planned (F5).
• StreamArena and asynchronous evaluation — planned (F6).
• Architecture versus streaming post-training — planned (F7).

This overview is a map, not a final taxonomy. The boundaries will move as preprints mature, evaluation protocols become stricter, and methods expose more complete storage and systems accounting. The underlying comparison contract— query visibility, clocks, resource boundaries, evidence recoverability, and matched controls—should remain useful even when the paper list changes.