From ReKV to Retrieval-Behavior Preservation: What Should a Streaming VLM Remember?

Series guide. This article is the bounded-memory thematic survey. Start with the development-history hub for the six-path map, read the ReKV note for the exact-archive baseline, and use the pipeline survey for the stage-by-stage architecture.

A streaming Video-LMM has to make decisions before it knows which detail a future user will care about. ReKV avoids write-time eviction or compression by storing every generated visual KV block, then retrieves a small subset after the question is known. This does not recover unsampled pixels or information absent from those model states. Fixed-state methods make the opposite trade: they evict, merge, or summarize history during the stream so that memory no longer grows with video duration.

That sounds like one progression from large to small memory. It is actually a change in the scientific contract. ReKV asks how to make a small read over a large recoverable exact-KV archive. Bounded-state methods ask which evidence can be discarded before an unknown question exists. Their success cannot be judged by the same memory number, because write state, retrieval index, archive, and answer-time context are different resources.

This article separates those resources, compares the retention priors used by InfiniPot-V, StreamMem, LiveVLM v2, HERMES, and ProtoKV, and ends with a research proposal: evaluate whether a bounded state preserves the future query-to-evidence behavior of a full-history oracle.

Sources and publication status were checked against first-party records on 2026-09-03. ReKV, InfiniPot-V, and HERMES have formal proceedings records. LiveVLM v2 is the substantially revised DAC 2026 version, not its 2025 v1 mechanism. StreamMem is an arXiv paper whose author page records CVPR 2026 VidLLMs workshop acceptance. ProtoKV appears on the official ICML 2026 accepted-paper list, while its public code repository remains a release-in-progress placeholder. Recent acceptance status and implementation maturity are therefore not treated as equivalent to an established, reproduced result.

Begin with the Actual Query Contract

All six primary methods process earlier frames before the eventual question is available. A question arriving at time $t$ may use the causal prefix $V_{\leq t}$, but it cannot retroactively guide decisions already made at time $t-d$.

This is stricter than either of the following:

• offline query-aware compression, where the whole video and question can be inspected together;
• persistent-query monitoring, where a registered request guides every future memory update.

The distinction matters because a query-aware top-$k$ selector can use the actual question to estimate relevance, whereas an unknown-future-query memory must commit to retention before that signal exists. It may use visual redundancy, generic prompts, attention patterns, or learned structure, but not the future question itself.

HERMES needs one extra qualification. Its fixed local/global guidance prompts are pseudo-queries rather than the actual future question; in a multi-turn stream, they may also include the previous conversation. The next unseen question is still unavailable, but later retention can be conditioned on past interaction.

Write, Read, and Archive Are Three Policies

The word memory often hides three independent decisions:

1. Write policy: what representation is produced, retained, merged, or evicted as a new stream unit arrives?
2. Read policy: which part of retained state enters the context for one question?
3. Archive policy: what historical evidence remains recoverable after the active window moves on?

ReKV and fixed-state methods occupy different points because they answer the third question differently:

ReKV
  write: keep exact generated KV
  read:  retrieve a fixed top-r subset
  store: exact archive + index grow with time

bounded-state methods
  write: evict, merge, or summarize under a fixed model-state budget
  read:  use all or part of that fixed state
  store: no complete historical KV recovery path

Offloading, compression, and retrieval are not synonyms. Offloading moves state between devices. Compression changes what state represents. Retrieval changes which retained state a request activates.

Resource Accounting

The table below charges each resource separately. Model-state bounded means that persistent video-derived model state has a configured, length-independent upper bound. It does not mean every intermediate tensor has the same shape, nor does it automatically bound dialogue KV, raw video, application logs, temporary compression tensors, or numerical counter width.

Two subtleties follow from this accounting.

None of the fixed-state rows establishes a bound on raw-video retention, application logs, or dialogue history; those stores are outside their reported video-KV budgets. Conversely, ReKV’s conceptual method does not require a raw frame archive, although its released benchmark evaluator preloads sampled frames into a host-side array. Implementation buffers and algorithmic archives must not be silently treated as the same object.

First, a fixed answer context is not proof of fixed total storage. ReKV can load 64 frames for every question while its archive adds a new set of full per-layer KV tensors for every sampled frame. In its LLaVA-OneVision-7B FP16 setup, the authors calculate 18.8 GB of new KV per hour at 0.5 FPS. That is a growth rate, not a capacity limit; the complete protocol and hardware context are in the focused ReKV note.

Second, a fixed compressed state is not lossless. InfiniPot-V, StreamMem, LiveVLM v2, HERMES, and ProtoKV have no complete old-KV archive from which a discarded detail can be retrieved later. Their model-state contract is stronger, but their evidence contract is irreversible.

The active-bounded/archive-growing bridge

Several related systems compress or gate reads without crossing into fixed lifetime state:

MuKV’s reported memory-token count is measured per 300 frames; its paper and public implementation do not define a lifetime eviction rule over all segments. WeaveTime’s PCDF-Cache decides whether and how much to read, not how much to store. StreamKV’s compression ratio reduces the slope, but each new segment still contributes retained state. All three are best described as active-bounded/archive-grows.

What Each Bounded Method Chooses to Preserve

A future query is unavailable, so every writer substitutes a proxy objective. The methods differ less by whether they compress than by what they treat as a reasonable forecast of future evidence utility. The final column below records mechanism-level audit inferences unless a sentence explicitly attributes a finding to the paper; it should not be read as a list of experimentally proven failures.

These objectives are conceptually adjacent, not interchangeable estimates of one ground-truth quantity. TaR asks whether a spatial patch repeats. VaN asks whether a value vector is large. A chat template asks what generic content the model attends to before an assistant response. VSB combines learned attention with temporal coverage. HERMES assumes different layers benefit from different retention biases. ProtoKV changes the representation from instances to running summary statistics.

Among the five fixed-state methods above, the official LiveVLM repository gives direct implementation-lineage evidence: it states that the code is based on LLaVA-NeXT and ReKV. That supports direct implementation reuse, not a claim that every LiveVLM idea or every other bounded method descends technically from ReKV. Separately, in the archive-growing branch, the WeaveTime paper and repository explicitly reuse the ReKV codebase for historical KV memory. No equivalent direct-code claim is made here for StreamKV or MuKV.

Recent Perception and Distant Recall Compete

Why do almost all of these systems protect a recent window? Current perception and distant recall impose different requirements.

Recent frames need high spatial detail, stable local motion, and low latency. Distant history needs addressability and enough preserved evidence to answer a question that did not yet exist. If both are placed in one undifferentiated attention pool, three things happen:

1. old tokens increase prefill and attention cost;
2. numerous historical distractors compete with the current scene;
3. aggressive global compression can spend capacity on frequent old patterns while erasing a small but decisive recent object—or the reverse.

ReKV avoids write-time competition by storing history outside the active heavy KV window, then creates read-time competition among retrieved blocks. Its own ablation reports that increasing retrieved frames can help and then plateau as irrelevant context accumulates. LiveVLM v2 narrows the read again inside fixed state. WeaveTime first tries the present and opens historical retrieval only when a language-model uncertainty heuristic fires. Fixed-state methods instead reserve recent exact capacity and represent old history at lower bandwidth.

This suggests a two-channel design principle, not a theorem:

high-fidelity recent channel
    +
lower-bandwidth addressable far channel
    +
an evidence-sufficiency decision about whether the far channel is needed

The difficult part is the third line. Predictive entropy can reflect linguistic uncertainty rather than missing visual evidence, and a model can be confidently wrong about an old event. Likewise, always injecting far summaries may be cheap but can still interfere with current perception.

Position Is Part of the Memory Contract

KV cannot be removed and concatenated as if positions would repair themselves. Some implementations correct keys that already contain a rotary transform; others, including ReKV, retain projected pre-RoPE keys and apply new compact positions when reading. Either design needs an explicit policy for the indices, gaps, and axes that survive.

The policies are meaningful engineering choices, but none makes compression free. Preserving order, temporal distance, spatial coordinates, and standard attention compatibility simultaneously remains difficult when old token instances have been deleted.

How the Evaluation Evidence Differs

I do not combine accuracy numbers from these papers. The backbones, sampling rates, cache budgets, frame grouping, judge versions, question timing, and comparison constraints differ. A valid comparison needs matched write and read budgets as well as a matched model.

The evaluation evidence also answers different questions:

• ReKV evaluates relevant-frame recall on QAEgo4D as an intermediate signal and reports controlled RVS serving measurements. This makes retrieval failure more visible, but it does not test a fixed total state.
• InfiniPot-V, StreamMem, and LiveVLM v2 test bounded caches on combinations of RVS, OVO-Bench, StreamingBench, and offline long-video benchmarks. Their results support the authors’ configured-budget claims, but they do not provide a matched sweep over how long decisive evidence must survive after it leaves the recent window.
• HERMES evaluates direct answer-time cache reuse on StreamingBench real-time, OVO real-time/backward, RVS, and offline tasks across several backbones. It has no explicit delayed-query sweep, and its sensory/working/long-term layer names remain an interpretation of observed attention patterns.
• ProtoKV continues ingesting the stream after a dataset question’s original timestamp $t_0$, then asks that question at $t_0+\Delta$ for delays up to 30 minutes. It uses the intersection of examples valid at every delay: three retrospective RVS categories—whether-something-happened, order-judging, and what-event-order—and OVO-Bench backward-tracing. StreamingBench real-time is excluded because later video may change its answer. The authors interpret the extra post-query updates as a proxy for post-evidence update pressure, but the protocol does not know the last decisive-evidence timestamp. It is therefore a controlled query-delay intervention, not an exact evidence-age measurement.

ProtoKV’s protocol is the closest in this group to asking whether fixed state survives delayed queries, but it still measures answer behavior after lossy summary updates. It does not compare the bounded state’s evidence ranking with a full-history index. That missing counterfactual motivates the next section.

Open Hypothesis: Preserve Future Query-to-Evidence Behavior

The following is my proposed evaluation target, not an established field consensus:

Given one causal pass over a video, an unknown future query distribution, and a fixed online state budget, preserve as much as possible of the full-history oracle’s future query-to-evidence behavior.

Let $I_T$ be an index over all legal evidence observed by time $T$, and let $C_T$ be a bounded state produced from the same stream without seeing future queries. For a held-out query $q$, compare the evidence behavior of $R(q,I_T)$ and $R(q,C_T)$ before comparing final answers.

The target should include at least:

• top-$k$ evidence membership and Recall@$k$;
• ranking agreement or NDCG under matched candidates;
• temporal localization and event-boundary overlap;
• margin preservation near the retrieval cutoff;
• answer quality with retrieved evidence fixed, separating retrieval from generation;
• online update time, index memory, archive bytes, I/O, and tail query latency.

A useful experiment would hold constant the backbone, FPS, recent exact window, question-arrival protocol, answer context length, and judge. It would then compare:

1. a full-history exact-KV or evidence oracle;
2. recent-only state;
3. independent-score retention;
4. summary/prototype state;
5. the proposed bounded method.

The oracle is not a deployable target; it is a diagnostic reference. If a method improves generic feature coverage but fails to preserve the oracle’s evidence ranking for unseen questions, then “coverage” has not established the stronger claim. If retrieval behavior survives but updates or tail latency fall behind a live producer, the streaming system claim still fails.

This formulation also prevents a bookkeeping shortcut. A bounded vector index with pointers into an unbounded raw-video store is not bounded total storage. The pointer bytes and the evidence bytes it addresses must be reported separately.

My Takeaways

1. ReKV is the clean exact-archive baseline. It bounds active heavy KV and per-question reads, not the full historical index or archive.
2. Fixed state changes the failure mode. InfiniPot-V, StreamMem, LiveVLM v2, HERMES, and ProtoKV stop lifetime KV growth by making irreversible write-time decisions.
3. There is no neutral query-agnostic importance score. Redundancy, value norm, generic-prompt attention, visual attention, layer depth, and prototype assignment preserve different kinds of evidence.
4. Recent perception deserves protected capacity. Far history should enter at lower bandwidth and only under an explicit evidence-sufficiency policy, but today’s routers are still proxies.
5. Position handling belongs in every method card. Compact state can retain order while losing distance, or retain a recency anchor while losing episode structure.
6. Answer accuracy is too late in the pipeline. Evidence-level agreement with a full-history oracle would make retention failures more falsifiable.

Limitations of This Review

• This is a source and implementation audit, not an independent reproduction. Reported experimental conclusions remain author-reported.
• The primary comparison is about video-derived KV state. Structured event memory, caption stores, raw-video RAG, and proactive scheduling have different carriers and are treated elsewhere in this series.
• Public repositories do not always implement every paper-level storage tier or expose a complete serving stack. Resource claims here use the narrower of the visible implementation and the documented design.
• ProtoKV is recent and its public code is not yet a complete reproduction path; its delayed-query findings should be treated as frontier evidence.
• Fixed-shape state can still have numerical or systems limits, including counters, temporary update buffers, dialogue state, and producer backlogs that the paper’s main cache budget does not measure.

Open Questions

• Which held-out future-query distribution is broad enough to test reusable memory without leaking evaluation questions into the retention policy?
• Can a bounded state preserve rare OCR, exact counts, failed actions, and repeated-event boundaries instead of only dominant semantics?
• How should evidence necessity be estimated when language-model confidence is poorly calibrated to missing visual history?
• Can position policies preserve temporal distance and multi-axis spatial structure without reintroducing a growing cache?
• What is the right hybrid between exact recent KV, compact far summaries, and sparse verifiable source backlinks—and how should every byte be charged?
• Can matched systems experiments separate memory architecture from streaming post-training, recent-window quality, and asynchronous runtime design?

The useful next step is not to declare one retention heuristic the winner. It is to make the lost behavior observable: what the full history would have retrieved, what the bounded state retrieves instead, which evidence disappeared during writing, and whether the complete system still keeps pace with the stream.