WeaveTime

Paper: WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs

PDF: CVPR 2026 Open Access

Supplement: CVPR 2026 Supplementary Material

Project: WeaveTime

Code: zhangyl4/weavetime

Core Idea

WeaveTime can be summarized as:

ReKV-style historical visual KV memory + temporal-order SFT + uncertainty-gated coarse-to-fine retrieval.

It does not introduce a new compressed KV format or a bounded long-term memory. Instead, it improves two parts of a ReKV-style streaming system:

• temporal interpretation: teach the Video-LLM to distinguish event order rather than treat video as an unordered bag of visual evidence;
• memory access: answer from the current/recent context when possible and retrieve historical KV only when the model appears uncertain.

The individual ingredients are familiar. The paper’s main contribution is their integration and evaluation under a causal streaming VideoQA protocol.

Time-Agnosticism in Video-LLMs

The paper describes a common Video-LLM failure as Time-Agnosticism: the model recognizes useful objects and actions but does not reliably represent their chronological order.

The authors support this diagnosis with two controlled tests.

First, shuffling video frames causes surprisingly little degradation and sometimes even improves accuracy, including on several temporal tasks. Humans show a different pattern: temporal and action reasoning degrade when order is removed and recover when timestamps are available.

Second, moving a ground-truth answer window to different relative positions changes model accuracy systematically. Short videos show preferences for particular regions, while longer videos exhibit a strong early-context bias. This suggests that the model often relies on positional shortcuts instead of locating evidence from the question.

In streaming settings, the paper separates the consequence into two failures:

• Temporal Order Ambiguity: semantically similar historical segments are confused even when their event directions differ, such as entering versus leaving a room.
• Past-Current Focus Blindness: the model retrieves old evidence when the answer is visible now, or remains focused on the present when the question requires history.

WeaveTime assigns one component to each failure: SOPE teaches order, while PCDF-Cache controls access to the past.

Pipeline

Incoming frame or segment
        |
        v
Sliding-window Video-LLM encoding
        |
        +-- recent KV stays in the local window
        |
        +-- older visual KV enters a growing historical memory
        |
        v
Question arrives
        |
        v
First forward with current/recent context
        |
        v
Estimate predictive uncertainty
        |
        +-- low uncertainty  --> answer without long-term recall
        |
        +-- high uncertainty --> coarse historical search
                                  --> token-level reranking
                                  --> load selected KV
                                  --> answer with expanded context

Method

1. ReKV-Style Historical KV Memory

The memory substrate is inherited from ReKV. Each incoming frame $f_t$ is encoded with only a local token window of length $C$:

The resulting KV is appended to the historical memory:

These are in-context visual KV states produced by the LLM, not independent CLIP embeddings or captions. Because every frame is encoded with a recent sliding window, its KV may also contain information from nearby preceding frames.

Like ReKV, old KV blocks can be offloaded from GPU memory to RAM and loaded again after a question arrives. The active GPU context is bounded, but the total historical storage still grows with stream duration.

2. SOPE: Learning Temporal Order

Streaming Order Perception Enhancement (SOPE) is a training-time scheme built around a Temporal Reconstruction auxiliary task.

The input video is divided into timestamped segments. The visual segments are then shuffled while timestamp labels remain explicit. A simplified example is:

t=0.3-1.2s  <visual content from 1.2-2.1s>
t=1.2-2.1s  <visual content from 2.1-3.0s>
t=2.1-3.0s  <visual content from 0.3-1.2s>
t=3.0-3.9s  <visual content from 3.0-3.9s>

These video segments are shuffled.
List each segment's true time range.

The model first generates the correct timestamp order and then answers the original video question in the same conversation. This remains ordinary autoregressive instruction tuning: there is no temporal prediction head or specialized loss.

The reported SOPE setup uses 30K sampled offline video-instruction examples, one epoch of training, a fully trainable connector, and LoRA on the LLM with rank 128 and scale 256. It does not require a stream-tailored instruction dataset.

SOPE should be interpreted as behavioral temporal supervision. It trains the model to use timestamps and event progression, but it does not redesign RoPE or guarantee that retrieved, non-contiguous KV blocks preserve their original temporal distances.

3. PCDF-Cache: Look Now, Recall if Needed

The Past-Current Dynamic Focus Cache (PCDF-Cache) first answers from the current/recent KV window:

It then compares predictive entropy $H_t$ with a threshold $\delta$:

The experiments use $\delta=0.6$.

The paper describes this as computing the uncertainty of the initial answer. The public implementation is more specific: it applies the LM head to the hidden state at the last query position, computes normalized entropy over the top-10 logits for each layer, and averages a configurable window of final-layer entropies. A low-entropy path reuses the first forward pass; a high-entropy path performs another forward with historical retrieval.

This is cheaper than fully generating an answer twice, but it is still a heuristic. Next-token uncertainty is not the same as estimating whether historical visual evidence is necessary.

4. Coarse-to-Fine KV Retrieval

If the uncertainty gate opens, WeaveTime uses two-stage retrieval.

The coarse stage averages the token keys of each historical frame or block and compares this representative vector with an averaged query key using cosine similarity. This cheaply removes most of the history.

The fine stage performs ColBERT-style late interaction over the remaining candidates. For frame $i$, the paper defines:

Each query token can therefore match a different visual KV token instead of forcing the question into one average vector. The final top-$K$ historical blocks are loaded for answer generation, with the experiments limiting recall to 64 frames.

The public code prefilters roughly $3K$ candidates with average similarity before fine scoring. Some current code paths reduce the token-level MaxSim scores with a maximum or mean rather than the sum printed in the paper, so reproductions should verify which implementation path is active.

Where the Components Come From

The method is easier to evaluate when its direct sources are separated from its streaming-specific integration.

Explicitly Cited Sources

The connection to MemVR is especially direct. MemVR detects high uncertainty at intermediate layers and re-injects visual evidence to reduce hallucination. WeaveTime changes the retrieved evidence from current-image features to historical video KV:

MemVR:     high entropy --> re-inject visual tokens
WeaveTime: high entropy --> retrieve historical visual KV

The control policy is therefore not new by itself; the change is where the extra visual evidence comes from and how it enters the model.

Closely Related but Uncited Precedents

The following papers are not cited by WeaveTime, so they are conceptual precedents rather than evidence of direct influence.

Shuffle and Learn introduced temporal-order verification as a self-supervised video pretext task in 2016. It shuffles frames and trains a CNN to judge whether they are chronologically ordered. SOPE adapts the same underlying idea to a Video-LLM: shuffle larger segments, expose timestamps, and generate the recovered permutation as text before QA. The generative SFT formulation is different, but the temporal-learning principle is established.

Paper: Shuffle and Learn: Unsupervised Learning using Temporal Order Verification

TimeChat already combines timestamp-aware visual representations with temporal instruction tuning for long-video grounding and reasoning. SOPE adds corrupted ordering rather than simply binding content to correct timestamps, but explicit timestamp supervision for Video-LLMs predates WeaveTime.

Paper: TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

The PCDF control policy also resembles adaptive RAG. FLARE retrieves documents when upcoming text contains low-confidence tokens; Self-RAG learns reflection tokens that decide whether retrieval is necessary; Adaptive-RAG routes queries among no-retrieval, single-step, and multi-step strategies. WeaveTime uses a much simpler entropy threshold and retrieves historical visual KV instead of text documents.

Papers: FLARE / Self-RAG / Adaptive-RAG

From this perspective, WeaveTime’s primitive-level novelty is limited. Its contribution is a specific composition:

temporal-order pretext task
    + ReKV historical memory
    + uncertainty-based retrieval routing
    + two-stage dense retrieval

Evaluation

Main Results

On the real-time subsets, the reported average scores are:

The largest gains appear in temporally sensitive categories such as Action Recognition, Action Perception, and Event Understanding, which is consistent with the purpose of SOPE.

What the Ablations Show

Starting from LLaVA-OV-7B + ReKV:

Timestamp supervision alone is not enough and can hurt because of the offline-to-streaming distribution mismatch. The temporal re-ordering objective is the part that makes the small SFT dataset useful.

Adding the whole PCDF-Cache produces another clear improvement. However, the ablation does not fully separate the entropy gate from coarse-to-fine reranking or compare against an always-retrieve C2F system, so the contribution of each PCDF subcomponent remains ambiguous.

The independent C2F gains over ReKV are relatively small:

This suggests that the final improvement comes from the combination of temporal SFT, present-versus-history routing, and retrieval rather than from MaxSim alone.

My Takeaways

WeaveTime is best understood as a read-policy improvement over ReKV, not a new way to store historical KV.

Its useful conceptual shift is to split historical memory access into two decisions:

1. Does this question require the past at all?
2. If it does, which historical KV blocks should be loaded?

ReKV mainly addresses the second question. WeaveTime adds a simple answer to the first one and trains the model to interpret the retrieved evidence in temporal order.

The most interesting open direction is therefore not another MaxSim variant. It is learning a better estimate of historical evidence necessity. A language-model entropy score may be high because a question is linguistically ambiguous and low even when the model is confidently missing old visual evidence. A stronger router should jointly model:

• the temporal dependency expressed by the question;
• the sufficiency of the current visual observation;
• the availability and expected value of historical evidence;
• the latency and memory cost of retrieval.

Limitations / Open Questions

• Historical storage remains unbounded. PCDF bounds active retrieval, not the total CPU or disk KV memory accumulated over an infinite stream.
• Entropy is not evidence sufficiency. A model can be confidently wrong, and next-token uncertainty may not indicate whether history is needed.
• The PCDF attribution is incomplete. The paper does not cleanly isolate entropy gating from C2F scoring and retrieval frequency.
• Position integrity is not solved. SOPE teaches temporal behavior but does not preserve original distances among retrieved, non-contiguous KV blocks. This inherits the positional compromise of ReKV-style retrieval.
• Fine retrieval still stores detailed keys. Token-level reranking improves precision but increases CPU memory and transfer cost as history grows.
• SOPE may exploit local action cues. Reconstructing a short, coherent action sequence does not guarantee robust reasoning over sparse events separated by hours.
• The main backbones are 7B models. Transfer to newer VLMs with more complicated multimodal RoPE and tokenization remains unclear.
• Paper and code details differ. Entropy aggregation and MaxSim reduction are more specific in the public implementation than in the paper and are not identical across all code paths.

The paper is still valuable as a clean systems result: familiar components can become useful when the memory carrier, temporal supervision, and retrieval controller are aligned with the actual streaming protocol. But its contribution is primarily integration-level rather than a new memory or retrieval primitive.