The Two Clouds of AI Agents: Streaming Interaction with the Environment, and Autonomously Learning from Experience
🌐 Languages: 中文 · English
[This article is adapted from the author’s keynote at Flink Forward Asia 2026 (Shenzhen, June 26–27), “The Two Clouds of AI Agents: Streaming Interaction with the Environment, and Autonomously Learning from Experience.”]
It’s a great honor to be here at Flink Forward Asia to give this talk. My topic today is “The Two Clouds of AI Agents.” You may have heard the phrase before: in 1900, Lord Kelvin said there were two dark clouds hanging over the sky of physics, and out of them eventually came relativity. Today I’ll borrow that metaphor to talk about two major challenges facing agents: first, how an agent interacts with its environment in a streaming, real-time way; and second, how an agent autonomously learns from its environment—that is, how, like a person, it interacts with the environment continuously and accumulates experience and knowledge from its successes and failures.
These two clouds correspond to two major challenges for agents. The first cloud is streaming interaction with the environment: how an agent interacts with its environment in a streaming, real-time way. The second cloud is autonomously learning from experience: how an agent, like a person, continuously interacts with the environment and accumulates experience and knowledge from its successes and failures.
A claim: interaction is streaming event processing
Why give this talk here at Flink? When Junhua first invited me, I too wondered what agents really have to do with Flink. On reflection, I found the connection runs deep—the architecture we’ve built shares the same origin as Flink’s idea of unified batch and stream processing.
The traditional ChatGPT is question-and-answer, essentially a Request / Response microbatch model: the user asks a question, and once the input is complete, it computes an answer. But an agent that truly interacts with the world in real time needs true streaming—per-event processing with event-time semantics.
Two examples. First, as I’m speaking now, if you interrupt me, I have to stop immediately and listen to you, rather than reciting my whole 20 minutes in one breath. And while you’re speaking, I have to keep thinking in the background—I can’t wait until you finish before starting on my next sentence. If you insisted on waiting for the other party to finish before thinking, you’d get the effect of heads of state negotiating on TV through interpreters: every sentence has to wait for the translation before anyone can react. Second, the OpenClaw that’s so popular today—compared with the earlier Manus, the biggest difference many people summarize as a sense of being “alive”: it proactively comes to you and says “there’s an email you should deal with.” How does it know? A traditional agent only checks your email after you say “check my email for me,” and the email may have been sitting in the inbox for two days; OpenClaw, on the other hand, proactively initiates every four hours, backed by all sorts of event-driven mechanisms—exactly like Flink’s event-time event processing—so it can sense what’s happening in the world in real time.
Talk structure: perception, cognition, execution
For the streaming-interaction part, I’ll break it into three stages: perception, cognition, and execution:
- Perception: how the model perceives the various things happening in the world—including human events, events in the external world, and events produced by the agent interacting with other people.
- Cognition: how to interact with a person in real time while thinking deeply in the background.
- Execution: how to reuse the agent’s experience in real time.
The rest of the talk follows this order.
Part 1 · Perception: the world enters the model = Flink’s Source
Letting an agent perceive the world in real time, like a person, is essentially the Flink problem of connecting an external data source and getting the serialization right.
Take the simplest example: a Computer Use Agent. Anyone who has used one knows its typical workflow: take a screenshot, think, then perform one action—click a button or press a key—then take another screenshot, think again; it’s a loop like that. The problem is that such a loop takes roughly three to five seconds. This brings a big limitation: it simply cannot watch video—the kind of real-time video that several earlier speakers demonstrated can’t be handled with this “screenshot–think–act” method; it can’t take part in a real-time voice meeting either; and if I click a button and a warning box pops up, if that box disappears after one second, it’s as if the agent never saw it, because it simply couldn’t look in time.
Paper · AOI: Agent-Computer Observation Interfaces
arXiv:2606.29472 · Website · GitHub
Our approach to this problem is to turn the whole of observation into an event-triggered processing flow, which in principle is exactly like Flink’s stream processing—so we call this observation interface AOI (Agent Observation Interface). Concretely: split the real-time input video into images by keyframe, feeding those frames in only when the picture changes, and feeding just one frame when there’s no change; speech is first understood into text by a multimodal model before being fed to the model. This way, although our agent can only see a single snapshot of about five seconds per action (that’s just how slow current model inference is), it can still continuously watch video, listen to speech, and interact in real time with the animated elements on screen. This corresponds exactly to Flink’s idea of event-triggered, on-demand processing: you spend compute only when something actually changes, when an event actually occurs.
Paper · Sema: Semantic Transport for Real-Time Multimodal Agents
arXiv:2604.20940 · Website · GitHub
There’s another problem in perception: how the data is transported to the model. I don’t know if you’ve noticed the live captions on either side of the hall—it seems basically every “Flink” was recognized wrong, and many proper nouns too. Why do live captions like these usually have so many errors? The root cause is that their observation space is too narrow: they only see the current short phrase and do speech recognition based on that phrase alone; and behind them isn’t a very large model, so they lack world knowledge, domain knowledge, and the context of the whole talk—naturally accuracy is low.
Underneath this is a more fundamental issue: today many transmission and recognition systems already have a model, not a human, as the receiver, yet they’re still designed on the old assumption of “for a human to see, for a human to hear.” This is exactly the starting point of Sema—when the receiver is the model, we can transport around the semantics the model actually needs, rather than the redundant detail prepared only for human eyes and ears.
Part 2 · Cognition: real-time and intelligence are nearly orthogonal
After perceiving the world, the next step is cognition. Here there’s a core tension: real-time and intelligence are almost two orthogonal axes. Models that can respond in real time tend to reason shallowly; while the SOTA large models that reason very well are slow on a single response and can’t be real-time. Getting both real-time and intelligence at once is hard with a single model.
Our approach is to split fast and slow: a small foreground model handles fast responses and maintains the real-time interaction rhythm; a SOTA large model in the background handles deep thinking and deep planning. This is really Flink’s unified batch and stream, realized on agents—the foreground stream processing seeks low latency, the background batch processing seeks reasoning depth, and the two paths work together.
Concretely, in a conversation, Interactive ReAct has two forms. The first is “Think While Listening.” In traditional ReAct, the agent only begins thinking, calling tools, and answering after the user has finished an entire utterance; with Think While Listening, thinking and tool calls kick off in parallel while the user is still speaking—the user describes a need (say, the kind of candidate they’re looking for), and in the background the system is already analyzing the requirement and retrieving material in parallel, so that by the time the user stops speaking, the answer and tool results are often already ready.
The second form is “Speak While Thinking.” It’s like how I’m giving this talk: I don’t think through all 20 minutes first and then recite it in one go; instead, as I present each slide, I think about how to present the next one. An agent works the same way—the fast foreground model first picks up the thread and maintains a natural conversational rhythm, while the slow background model keeps thinking deeply and then feeds in the more complete content afterward. This way, the answer can be given immediately and still unfold naturally.
Paper · The Latent Bridge: A Continuous Slow-Fast Channel for Real-Time Game Agents
arXiv:2606.24470 · Website · GitHub
The fast and slow models I just described can be used with a similar method not only in real-time speech, but also in Computer Use—operating a computer’s graphical interface. Here we study it with an action game (Ms. Pac-Man)—playing a game is a very typical scenario: it requires you to dodge obstacles at the hundred-millisecond level, a fast reflexive action, and also to slowly think through strategy in the background, or you’ll never get out of the maze.
So what channel exactly should the fast and slow models use to communicate? That’s precisely the question this work investigates. We compared several approaches: one is to use only the fast model alone; one is a text channel, feeding the text the slow model generates to the fast model; and one is a continuous latent channel—passing the slow model’s thinking to the fast model as a vector. To make this middle channel the only variable, we freeze both the fast and slow models and change only the channel between them.
The conclusion: neither a SOTA slow thinking model nor a fast-responding model alone can play the game well; only by making the fast and slow models cooperate can you get both fast reflexes and long-horizon planning. And as for the channel that connects them, text and latent each have scenarios where they fit—for different tasks you can use different methods, and it is not the case that the higher-bandwidth latent channel is always better. In other words, the key is not the channel’s bandwidth, but whether slow thinking is actually useful for the task; judge that first, then decide which channel to use—that’s the right order.
Part 3 · Memory: where experience lives, and how it is retrieved
Next is the second cloud: autonomously learning from experience—that is, memory—how an agent stores and reuses the experience it accumulates during interaction. In Flink terms, this is its most core capability of stateful stream processing (state backend + checkpoint). In this part I’ll cover three works, corresponding to three mechanisms: the first turns textual memory into code, which is more structured and more convenient for aggregate reasoning and conflict detection; the second stores memory in an embedding-like form, placed into hash slots; the third internalizes a workflow into code, turning it into a domain-specific small model that automatically executes the previously slow workflow. Let’s go through them one by one.
① User as Code: represent user memory as code
Paper · User as Code: Executable Memory for Personalized Agents
arXiv:2606.16707 · Website · GitHub
Let’s start with the first work, User as Code. Begin with an example: suppose a user has 100 flight records and asks you—how many flights did I take in 2025, how many in 2026, how many trips to Europe, how many to Japan? If you store memory in Markdown, the model has to load the entire history into context and then, during thinking, count the flights one by one using reasoning tokens. This is quite error-prone—although today’s SOTA models already count fairly well, the error rate is still non-trivial.
Naturally we then wonder: if we store it in a database, in a structured way, wouldn’t it count well? But there’s a problem: a database is a fixed data structure. Today I designed a schema for flights, but tomorrow there’s bank-card information, what the user ate each day… the information is so voluminous and varied that the data structure can’t generalize—I can’t design in advance a universal structure able to store all information.
So it suddenly occurred to us: the model’s strongest ability is precisely writing code. It’s now widely agreed that an agent’s most powerful ability is coding, so why not let it design the data structure itself by writing code? That’s the starting point of User as Code—represent user memory as a piece of structured code the model writes itself, rather than a pile of Markdown text. This way you can do precise aggregate queries like a database (count flights, tally by destination) without fixing the schema in advance.
Once memory becomes code, there’s another benefit: you can attach constraints and validation logic that actually run. Structured memory plus a piece of code lets an interpreter deterministically stitch together many records to compute, do conflict detection, and proactively raise alerts—something plain-text memory can hardly do (with text you can only have the model count and compare during thinking, which is both slow and error-prone).
The experiments bear this out: once memory is turned into code, aggregate queries like counting flights above can be answered quickly and accurately, and on standard memory benchmarks it approaches the upper bound of “stuffing the entire conversation directly into context.”
① Programmable KV: compile memory and Skills into the KV Cache
Paper · Models Take Notes at Prefill: KV Cache Can Be Editable and Composable
arXiv:2606.17107 · Website · GitHub
As user memory and Skills get longer, re-prefilling them every time drags down TTFT (time-to-first-token). Programmable KV’s core insight is—the KV Cache is not a pile of frozen intermediate products, but a “memo of conclusions” the model writes during prefill: at prefill time, the transformer has already written the “conclusion derived from some field” onto downstream aggregation / separator tokens; at decode time the model reads these “memos,” not the fields themselves.
Move one: compile memory and Skills into the KV Cache (Composable). The pain point is “memory re-read every turn”: reusable prefixes like skills, long Markdown, and user profiles are re-prefilled every turn—an O(L²) cost, and TTFT explodes with length.
Our approach is pre-compilation + RoPE relocation: compile a piece of skill/rule just once, and when needed, splice its cached KV into any context by “relocating” it with RoPE—because prefill caches keys with absolute positions, you just rotate the key to the target position. This turns O(L²) recomputation into O(L) concatenation, behaving almost identically to recomputation, and the longer the skill, the larger the gap between “full recomputation vs. pre-compiled concatenation.”
Move two: edit fields in the KV Cache with a sticky-note (Editable). The low-level fields in memory change—account balance, time, weather—but we don’t want every field edit to invalidate the whole cache and force a full recompute.
Why can’t we just directly edit the KV of the corresponding token in the KV Cache? Because, as noted, at prefill the model wrote the “conclusion based on that field” onto downstream aggregation tokens (PREFILL — the model writes notes); if you only refresh the field’s own KV, those stale downstream “memos” remain, and the decision still reads the old conclusion.
So we use a sticky-note revision (erratum): rather than editing the prefix, we append a conspicuous correction at the end of the context—[erratum] change X to Y. It is append-only, so it’s fully compatible with prefix caching. In Flink / DB terms, this is the classic database form of append-only log + periodic checkpoint = changelog + checkpoint. In an online-serving measurement: prefix-cache hit rate is 98.5% vs 1% (editing the prefix directly misses almost entirely), throughput up to 14.5×, and p90 TTFT can drop 53×.
② User as Engram: write user facts into model parameters
Paper · User as Engram: Internalizing Per-User Memory as Local Parametric Edits
arXiv:2606.19172 · Website · GitHub
The two previous ideas (User as Code and Programmable KV) both put memory in the context; the second mechanism writes the user’s factual memory directly into model parameters. The prior here is DeepSeek’s Engram memory work published this year. Engram is, in plain terms, an external, ever-growing hash table: the model learns on its own when to query which slot—hash the last few tokens, look up the corresponding slot—much like a continuously growing bank of external RAM. Right after DeepSeek’s Engram paper came out, we added Engram to our model and inserted the user’s factual knowledge into it.
Before this, plenty of work had already tried using LoRA to store user memory into model parameters. But LoRA has a well-known problem: it can extract knowledge but struggles to do complex reasoning over that knowledge; and it modifies weights globally, so it’s prone to pollution and hard to isolate.
Using Engram this way has several clear benefits over post-training (e.g. LoRA). First, it doesn’t need complex gradient descent; the update is very simple. Second, it has additivity—a user has their own personal memory, family memory, and company memory, and generally, when hashes don’t collide, you can just put them all in and stack them together. This amounts to: at the bottom we train a shared “skill for reading memory” (how to use this memory block), and at the top we insert each user’s factual memory into different slots. This naturally supports multi-tenancy while storing and retrieving in a vectorized way.
It also has a benefit plain-text memory cannot offer—it can store multimodal user memory. For instance, today you met ten people and took ten photos, and later you need to recognize those ten people again; it’s hard to describe every face clearly in plain text, whereas this parametric, vectorized storage-and-retrieval is naturally suited to such multimodal user memory.
③ PreAct: compile repeated work into a cache
Paper · PreAct: Computer-Using Agents that Get Faster on Repeated Tasks
arXiv:2606.17929 · Website · GitHub
Everything so far has been declarative memory (facts); the third work, PreAct, targets procedural memory—how to do something, i.e. the mechanism of skills. We often have repeated work, especially in Computer Use where the agent operates a computer and repeatedly performs the same actions. It’s like a foreigner using Alipay for the first time—they might click around forever without knowing how to pay; but after a few uses they’re surely very familiar. We want the agent to have this “the more you use it, the more fluent and faster you get” ability too.
Agents already have quite high success rates on Computer Use, but they often re-think from scratch on every frame they see, wasting a lot of tokens and being fairly inefficient. So we compile the trajectories the agent has executed before into a small program (a deterministic state machine) and let it execute that program repeatedly. As long as the process is reproducible and replayable, we can complete it much faster.
Take a concrete example—creating a new contact. On the first run, the agent dutifully goes through the full “look at screen → reason → act” flow to finish the task, and at the same time compiles the whole process into a state machine: each state is a set of assertions about UI elements plus an action. The next time it meets a similar task, it can replay directly through this state machine step by step, no longer needing to call the large model at every step.
So PreAct’s execution policy is: on a hit, take the fast path and replay the compiled trajectory directly—deterministic and nearly free; if it detects an anomaly—say, after clicking a button the expected next page doesn’t appear, unlike at compile time—then fall back to the ordinary VLM slow path and think carefully again.
This is a bit like a low-budget miniature world model: the core of a world model is continuously predicting “what will happen after I take this action”; if it matches expectation, keep going to the next action without re-thinking; if it doesn’t match, fall back and think carefully. This is how we achieve cache reuse of already-executed trajectories. This is also consistent with the fast/slow split in the cognition part earlier—hit takes the fast path, miss takes the slow path.
Back to the two clouds of the agent
Let me wrap up. In the preceding stages we covered several techniques for “streaming interaction with the environment” and “autonomously learning from experience.”
On perception, we extended the space in which the agent perceives the world from a turn-by-turn space to a streaming-processing space—which is exactly Flink’s core idea of unified batch and stream. This lets it continuously watch video, continuously listen to sound, continuously watch animations, and interact with the environment in real time. On cognition, we make full use of fast–slow model cooperation: the fast model interacts with the world in real time, while the slow model does deep thinking in the background.
In the autonomous-experience-learning part, we introduced three mechanisms: the first turns textual memory into code, which is more structured and more convenient for aggregate reasoning and conflict detection; the second stores memory in an embedding-like form, placed into hash slots; the third internalizes a workflow into code, turning it into a domain-specific small model that automatically executes the previously slow workflow.
From these works we can see: what decides a system’s performance is often its interaction and representation, not the model’s ability itself. Our models today, whether vision or large language models, are actually already very capable; but much of the time we simply haven’t given the model the right context, or the right modality, so the model can’t deliver real-world effect. I think this is exactly the key value the Flink framework brings—it can process these different modalities into forms convenient for the model, letting the model deliver maximum value in real business scenarios.
Thank you!
Related papers and websites
Works covered in this talk (for more, see 01.me/research):
| Work | arXiv | Website | Code |
|---|---|---|---|
| AOI · agent observation interface | 2606.29472 | website | GitHub |
| Sema · semantic transport | 2604.20940 | website | GitHub |
| Latent Bridge · slow-fast channel | 2606.24470 | website | GitHub |
| User as Code · executable memory | 2606.16707 | website | GitHub |
| Programmable KV · editable & composable KV Cache | 2606.17107 | website | GitHub |
| User as Engram · parametric user memory | 2606.19172 | website | GitHub |
| PreAct · compiling procedural memory into a cache | 2606.17929 | website | GitHub |
The Interactive ReAct work is coming soon.