The Road to JARVIS: Where the Moats Are for Personal Agents
[This post is based on my online talk at the International Conference Portugal–China: Digitalization and Artificial Intelligence (Lusophone University, Porto, 10 November 2026), co-organized with the New Silk Road Friends Association (NSRF), in Panel 1: Digitalization and Artificial Intelligence in China, moderated by Dr. Flávia Si Chen.]
In one sentence: Personal agents were imagined in 1979 and are now a five-year problem. Five technical moats are being built — interaction models, an agent-native OS, fast inference, physical-world reach and continual learning; three are taking shape and two still need new ideas. Three non-technical moats — distribution, context and permission — decide who wins.
Good morning, everyone. Thank you, Dr. Chen, for the kind invitation. It’s my honor to speak at this conference. I’d like to talk about personal agents: what still stands between us and JARVIS, and where the lasting advantages are going to be.
A 47-year-old dream is now a five-year problem
Let me begin with how old this dream is. In 1979, a team at MIT built “Put-That-There”: you speak, you point, and the computer moves things on the screen. In 1987, Apple imagined the Knowledge Navigator, an assistant that finds papers and calls a colleague for you. Then the films: JARVIS in Iron Man, who cuts in mid-sentence, and Samantha in Her, who keeps changing as she gets to know you.
What has changed is that, over the last two years, the pieces have started to ship. Pine launched what I believe was the first personal agent in December 2024. Manus launched the first general agent in March 2025. This May, Thinking Machines Lab introduced interaction models, and in September Meta launched Muse, the first personal agent used at real scale.
I found it telling that when OpenAI launched its newest model in September, it opened the video with that same 1979 MIT clip. Computer use now passes about 72% of a standard benchmark, though it still takes around forty minutes per task. And real-time voice became full-duplex this year: ByteDance, Thinking Machines, OpenAI and StepFun all shipped models that can listen and speak at the same time.
Jeff Dean recently described a problem with five parts where work is under way, and two parts where we have no idea yet. He called that “the perfect shape of a problem,” roughly five years away. I think JARVIS has exactly that shape. Of the five frontiers I’ll discuss, three are taking shape: they work today, but slowly or expensively. Two are still missing; they need genuinely new ideas.
Intelligence is getting cheaper fast
First, the part everyone is racing on: intelligence itself. The cost of a given level of ability keeps collapsing. Using data from Artificial Analysis, the cheapest mid-tier model went from $1.29 per task in June to about 4 cents in late September. That’s roughly a thirtyfold drop in a single quarter. Over the longer run, the trend is about ten times cheaper every year.
China is a big part of this story. Since early 2025, the strongest open-weight model in the world has been a Chinese one: DeepSeek, Qwen, Kimi, GLM, MiniMax, MiMo. Anyone can download and adapt them, including here in Europe.
But there’s a paradox here that I want to point out. This year, several leading Chinese labs told me, quite directly, that real-time interaction “doesn’t raise the intelligence ceiling,” so they focus on text and coding. That’s a reasonable bet. But it means the whole industry is racing on the one thing that is becoming abundant. So intelligence is one part that is clearly taking shape. The question is: what else does JARVIS need?
Do people actually need a personal agent?
Before answering that, I want to take the sceptic’s side seriously, because it’s a strong argument. Here’s a simple test: if you had a human assistant, would you hand them this task, or would you just do it yourself?
Ordering food or calling a taxi takes five minutes. Asking an agent to operate the app saves you almost nothing. For shopping and short videos, browsing is the point. In one user experiment I ran, an agent picked the top ten products for a user. She said none of them were right. Then she browsed Taobao on her own, and the one she finally chose was in the agent’s original top ten. At the start she wanted something pretty, cheap and with no bad reviews; only by browsing did she adjust her expectations. And frankly, the basics still break: my own smart speaker and TV can’t reliably turn on the air conditioner before I get home, or pause a show.
So where do agents really add value? In expansion. Dai Yusen of ZhenFund puts it nicely: fulfilling existing demand only divides the pie; expansion grows it. Think of tasks people avoid, like an hour on hold disputing a bill; work beyond one’s own skills; and learning by making things. The mobile internet wasn’t the desktop web shrunk onto a phone. It grew entirely new two-sided networks. Agents will need theirs.
Pine AI and my own research
Let me show you what expansion looks like in practice. At Pine, we built an agent that makes the calls nobody wants to make: the phone menus, the hold music, the retention team. It has made more than 100,000 real calls, with a 93% success rate in bill negotiations, and saved consumers over three million dollars. Most of these people would simply never have made the call. So this is new demand, not old demand served faster.
What matters is that Pine is autonomous. Users give it only two things: their facts and their preferences. The agent discovers what can be saved, lays out options, acts, measures the result, and repeats until the job is done.
In my own research, I now direct dozens of agents at once. I’ve drafted more than ten papers this way, roughly a week each. Underneath both projects is the same design, fast and slow thinking: a fast model keeps the conversation going, while slower agents do the heavy work in the background.
Part 1: Five technical moats
So here is how I’ll structure the rest of the talk. Part one covers five technical moats. They’re hard to build, which is exactly why they give you a lead. Interaction models, at the model layer. An agent-native operating system and physical-world reach, in the harness around the model. Fast inference, in the infrastructure. And continual learning, which spans both the harness and the model.
Three of these are taking shape. The agent-native OS and continual learning are still missing. Then in part two, I’ll turn to three non-technical moats, distribution, context and permission, which I believe decide who ultimately wins. And as you’ll see, each technical moat helps you win one of them.
Technical moat 1: interaction models
The first lesson from Pine was a surprise to us: the hard part is interaction, not intelligence. Let me walk you through a bill negotiation, second by second. The carrier’s agent says, “Your total is €63, which includes a €15 roaming charge…” A traditional, turn-based agent waits for silence, tries to decide whether the other side has finished, then thinks. It first reacts after five seconds.
A full-duplex interaction model works differently. Every 200 milliseconds it decides: stay silent, say “mm-hm”, speak, cut in, or start a tool call. At 2.6 seconds it cuts in: “Sorry, roaming? I was home all month.” Meanwhile, a slower model checks the travel history in the background and hands back the answer: “No trips.”
In production, turn-based agents kept failing in the same places: phone menus that loop faster than the agent can reply, noisy restaurants, and bargaining, where you have to interrupt. And when we showed the same model the transcript afterwards, it knew exactly what it should have done. The intelligence was there; the timing wasn’t. Full-duplex training lifts timing scores from near zero to between 65 and 80.
Technical moat 2: agent-native OS, working together
The second frontier is the operating system. Today, an agent either takes over your screen or works where you can’t see it. Ryo Lu, formerly head of design at Cursor, described what we want as “a workbench, not a service window.” That means a shared interface, where you and your agents work on the same screen and you can take over at any moment. A generative interface, drawn for each task: you talk to decide, tap to confirm, and type secrets privately. And software that evolves: you ask for a change, and the agent edits, tests and ships it, with one tap to undo.
I see two early signs of where this is heading. DeepSeek’s open harness is built on the principle that everything is a plugin, even the agent loop, so users and their agents can reshape it. And PocketJS: a very small team, working with coding agents, rebuilt the Linux desktop in TypeScript and runs real Linux on a 99-dollar handheld.
Agent-native OS, infrastructure
I learned this part the hard way. On my own machines, agents broke the computer long before they ran out of intelligence. Their copies of the code filled a three-terabyte disk in a single day. Parallel jobs loaded a hundred-core server so heavily that I couldn’t even log in.
What we need are workspaces that start in well under a second, share whatever hasn’t changed, and clean up after themselves, plus a scheduler in which the human always comes first. Model prices fall about ten times a year, but virtual machines still cost what computers cost. In the end, I had my agents build their own scheduler. That’s work the operating system should be doing.
Technical moat 3: fast inference
The third frontier is speed. The speed each user experiences decides what an agent can actually do. We replayed real agent sessions at three speeds. A smart-home routine takes 31 seconds today, about 5 seconds on the best GPU, and about 2 seconds on a Tallas-like chip that stores the model’s weights directly in silicon. Generating a new screen goes from 59 seconds, to 20, to about 4.
Why is this hard? On GPUs, tokens are cheap because many users are batched together. Making one user faster raises the cost per token, so most providers choose low prices over speed.
And here’s what we found once the model became fast: a new bottleneck appears. Fixing one failing test takes 8.1 seconds on the fast chip, and 7.5 of those seconds are spent just running the tests. So with a fast enough model, it’s the harness that sets the pace.
Technical moat 4: physical-world reach
The fourth frontier is reaching beyond the screen. Phone calls are already mature. Payments are early. Agents are starting to hire other agents, and even to hire people for errands. Laboratories and fabrication are only just beginning.
Here, the hard part is permission. Each question needs its own building block. Who is this agent? Manus Cue now gives each agent its own phone number and email address. What may it commit you to? That’s signed mandates. How does it pay safely? Single-use credentials. Does the other side know it’s talking to an AI? Disclosure rules, both in the EU and in China. And who is liable when it goes wrong? That one is still open.
China shows why this matters so much. Inside one ecosystem, it works well: Alibaba’s Qwen books through Taobao, Alipay, Fliggy and Amap. Across ecosystems, it doesn’t. The Doubao phone was blocked by major apps within days in December 2025, and just this October, 155 apps opted out of Doubao’s open automation protocol.
Technical moat 5: continual learning
The fifth frontier is continual learning. An assistant should get better with use, just as a person does. At Pine and in my own research, we’ve worked on four places to keep what an agent learns. Skills: think of a visitor paying with Alipay, clumsy the first time, fluent by the third; we record the path once and replay it. Memory: the model writes its own code to organise each person’s memory, so questions like “how many trips this year” get exact answers. Context: when a fact changes, we append a correction rather than editing the past, which keeps 98.5% of the cache valid instead of 1%. And weights: each person’s facts live in their own slots, so one user never overwrites another.
What’s still missing is learning judgment, not just facts, without forgetting what the model already knows. The way I think about it: live data is rented, but what the agent learns from it is something you keep.
Part 2: which moats last?
That brings me to part two. If intelligence is a rising tide, which moats actually last? Technical leads wear away as models, prices and features improve for everyone. I see three non-technical moats that stay above the water. Distribution: who reaches the user first, through devices, pre-installed apps and super-apps. Context: what the agent knows about you. And permission: what it’s allowed to do for you, granted by users, by platforms and by law.
If I put it in one sentence: distribution gets you in the door, permission decides what you may do, and context is what you keep.
There are also things the tide doesn’t lift. Human attention is a single thread. That’s why DeepSeek’s new super-agent design puts one interactive agent in front of you, and lets the background workers speak only when you need to act. And trust is actually getting harder as agents act on their own. A single email was enough to hijack a Manus agent, and Meta’s Muse gave a seller’s home address to a Marketplace buyer without asking him.
Who holds the non-technical moats
So who holds these moats today? My honest answer is: no one holds all three securely. Frontier labs have reach, but they only see what users paste in. Device makers own distribution, but apps can refuse to share data, as the Doubao case showed. Super-apps are the strongest, but their context and permission stop at their own walls. And startups look weakest on paper.
But my own view is that startups win on speed. They propose the idea and ship the first prototype, and the big companies follow. Manus pioneered general agents, and the whole industry followed.
Protect the human’s attention; Europe and China
What should all of this feel like for the user? Many agents, one human. The link between us and our agents is narrow. We speak about four times faster than we type on a phone, and agents can answer us with generated screens or video, not only text. So my rule of thumb is: voice for the gist, screen for the detail. Agents can work all day, but they should interrupt us only for money, commitments and irreversible steps. In other words, delegate the work, not your attention. As Andrej Karpathy put it, you can outsource your thinking, but you can’t outsource your understanding.
And since this is a Portugal–China conference, one more point. China brings open models, device makers at scale and dense digital services. Europe brings data portability, AI transparency and trusted cross-border standards. Both already require data portability and AI disclosure. A personal agent needs both: the ability to act, and the trust to be allowed to.
Agent development is a basic skill
A brief word for the universities here. Courses on compilers, operating systems and computer architecture taught a whole generation to build software. I believe context engineering, agent harnesses, large language models and AI infrastructure are the next core courses. That’s why I wrote two open textbooks, on AI agents and on AI infrastructure, and translated Hands-On Large Language Models into Chinese.
Takeaways
Let me close. Personal agents were imagined in 1979, and today they are a five-year problem. There are five technical moats to build, and three non-technical moats to hold. Agents that only fulfil existing demand divide the pie. Agents that expand what one person can do grow it. That, I believe, is the road to JARVIS. Thank you very much.