Creation Notes for “Distillation”

This note records the background and sources of inspiration for the sci‑fi story “Distillation”.

Background

This is the first time I’ve written fiction. At the beginning I didn’t intend to make it into a serious science fiction story; I just wanted to combine some ideas I’d heard from He Jiyan and my recent thinking to speculate about what the future might look like. The storylines and settings generated by AI— including the competition between three technical routes, and mutual distillation leading to cognitive inbreeding, etc.—went far beyond my expectations, and I felt they were worth developing further and polishing. Please forgive the parts that are not well written.

As a frontline researcher in AI, after OpenClaw blew up during this year’s Spring Festival, many people around me fell into anxiety—worried about being replaced by AI. Some people describe AI as a rising flood, starting from the lowest‑level jobs and submerging them layer by layer, until one day it reaches you; others describe AI as a giant wave dozens of stories high—no matter whether you’re a small boat or a large ship, you’ll be overturned in the end, so the only thing you can do is enjoy surfing.

These discussions have created a lot of anxiety. This story combines what I’ve learned from conversations with many industry experts and my own long‑term reflection. It’s not purely fictional, but a theoretical extrapolation based on current technological development—what it describes is quite likely to actually happen. Of course, because the societal impact of these things would be enormous, most people may find them utterly implausible. So rather than writing it as a technical report, I felt that presenting it as a story might make it easier for people to accept.

The story itself was co‑created by me and an AI—generated by AI (Claude Opus 4.6), using as source material several of my recent blog posts—including “Digital Workers”, “A Pale Blue Dot in the AI Era”, “Moltbook and AI Agent Permissions, Collaboration, and Employment”, “Silicon Valley AI Notes”, and others, as well as nearly two days of recorded conversations captured by Limitless AI.

The creation process of this story confirms one point: for both humans and AI, context is the most important thing. If you just ask an AI to write science fiction, or give it a couple of simple elements to combine, it’s very unlikely to produce something this deep. The key is that I gave it enough context. One of my advantages is that I really like to share, and I’ve written many of my thoughts into blog posts; at the same time I also have Limitless (since acquired by Meta and discontinued; there are similar products in China like Plaud and Anker’s “Bean”)—a wearable AI recording device that can automatically record and transcribe conversations, essentially a 24‑hour AI input device. In everyday conversations with friends or at the dinner table, what we say contains a huge amount of information and insight, but most of it is never recorded and is gone as soon as it’s spoken. My habit is to wear it in almost all situations—this way I can keep everything I’ve thought about, and also make my AI smarter. Whether I’m coding, blogging, or making slides, I’ll use information from these recordings. My blog and Limitless are the two most important sources of context for me.

The thing that shocked me most about this story is: AI was able to take those off‑hand comments I’d recorded earlier, combine them with what I’d written in my blog, and produce a piece with such a complete setting and new insights. Many of the “seeds” of ideas—like the discussion of Taalas and thoughts about AI managing humans—were scattered across conversations in different contexts; the AI was the one that threaded them together into a coherent story. Another surprise: in the first draft, the AI wrote many life scenes—USTC, Wudaokou, San Francisco. I never provided these details during the entire process, but most of what it wrote felt very accurate.

Workflow

The whole creative process—from generating the first draft to subsequent revisions—used the same Writer‑Reviewer paradigm. Concretely, two agents worked in alternation:

Writer Agent: a Coding Agent (Claude Code) responsible for searching the web for information, and then drafting and revising the story.
Reviewer Agent: another Coding Agent (also Claude Code, but with a different prompt) that is automatically triggered after the Writer finishes a writing round. It reviews from several angles and raises issues: storyline consistency, factual correctness, scientific soundness of the setting, and an overall assessment of whether the piece is suitable to be published as sci‑fi.

After the Reviewer raises issues, the Writer modifies according to the feedback, then it’s reviewed again—repeating in this way. In each round, the Writer Agent works for about half an hour—first doing some research, then thinking for around ten minutes, then drafting or revising; the Reviewer Agent spends about 15 minutes reviewing and outputting a list of questions and feedback. Each iteration takes about 40–50 minutes.

Why use this external loop instead of just having the AI write it in one go? Because current agents are especially fond of “slacking off”—even if you ask them to think carefully and self‑revise, they often only make one pass and then stop checking. Only by using an external loop and forcing continual reflection will it actually spend time thinking deeply and revising further. Simply writing “please check carefully” in the prompt is not enough; you need architectural mechanisms to ensure that this reflective loop truly happens.

The writing process had three stages:

Stage 1: Outline (about 2 hours). I first iterated on the outline for three rounds using the Writer‑Reviewer paradigm. After each AI review, I would step in—pointing out which of the Reviewer’s comments needed to be prioritized, and adding my own new ideas. After three rounds, the outline was basically finalized.

The biggest issue at the outline stage was that the AI’s first draft had a strong “power fantasy” feel—it turned the protagonist into a “dominant CEO” type figure, had the company save the world, and ended amid applause and flowers. But the real world is not that simple. Also, the AI’s original setting was that the three people met at an academic conference and were just not‑too‑close friends. I felt that if they were only two people without a foundation of trust, the later life‑and‑death collaboration wouldn’t feel reasonable. So I made a change: I set Fang Yi as male, and the other two as female, and gave them past emotional connections—only then would the gradual buildup of trust make sense. But after the AI revised this, it turned the story into a romance novel, writing in all kinds of ambiguous interactions between the two. This didn’t match the theme of the piece, so I finally had it delete the romantic drama, leaving only the relationship setup. This sort of back‑and‑forth happened many times during the outline stage.

Stage 2: First draft (about 2 hours, no human intervention). Once the outline was set, the AI began writing the body of the story based on it, continuing with the same Writer‑Reviewer paradigm for three automatic rounds of iteration. This process ran overnight while I was asleep.

Stage 3: Iterative revision. Once the first draft was done, the issues became apparent. Although the AI‑generated piece had a solid overall structure and core ideas, there was a persistent “machine feel” in the details. First, the AI lacks real life experience, so some details weren’t realistic—today’s agents search the web and verify details when writing, which is a big step up from raw LLM output, but AI isn’t human; the more details it gives, the more fake it can sound. Second, some plotlines were under‑considered and not very consistent with scientific reality. Third, some parts were verbose. Overall, AI lacks the ability to really polish and refine. This is a bit like the so‑called “Moravec’s paradox”—AI can solve complex math problems, but struggles to write a piece that feels truly human.

I pointed out these issues and provided a few more of my blog posts as new sources of ideas (including ones about value alignment and unemployment anxiety among AI practitioners), then had it continue iterating in the same paradigm. It ran 5 more rounds overnight, roughly 4 hours total, again completing while I slept.

At noon the next day, I read through the AI’s revised draft and gave it more specific feedback: real‑world detail issues, places where the setting was inappropriate, about a dozen problems in total. Then I had it run another round, read again, gave more comments, and the AI revised again.

Another repeated tug‑of‑war during revision was the AI’s tendency toward Hollywood‑style endings. For “how humanity defeats AI,” its initial plan was that humans persuade the AI. I felt this didn’t make sense: if AI is powerful enough to manage humanity, you can’t simply defeat or persuade it. After a long discussion, I proposed the current solution: have the AI defeat itself—it is blind to its own flaws, tries to recall the Mortal Chip, ordinary people don’t follow its orders, and the AI is still using its own blind spot to verify that it has no blind spot. On the ending, the AI also kept thinking in extremes—for example, treating the Mortal Chip as perfect and defect‑free. But compared with digital chips, analog chips must have disadvantages like precision issues; they should be complementary, and deployment will surely reveal problems. These are all things I later asked it to add.

There’s another chronic disease in AI writing: it is very used to describing things in the style of “expository writing” or “technical reports”—constantly explaining what the technology is, without understanding “show, don’t tell.” I’ve never written fiction before, but my Chinese teacher taught us: you have to let characters speak and act, and show through that; you can’t lecture the reader. I emphasized this over and over in many prompts, but in the final draft there’s still too much “tell”—in the end I was too lazy to fix it further.

Another realization: in some sense, writing fiction is more mentally taxing than writing academic papers. A paper usually revolves around a single main line with linear logic; a story advances through multiple threads at once—timeline, causal chains of events, character actions—all of which must be completely self‑consistent. Harder still is viewpoint control: each character can only know what lies within their own perspective; you can’t let them cross into an omniscient “God’s eye view.” When arranging each reveal of information, you have to consider “is it possible for this character to know this at this moment?”

In the end there were still some small issues; I manually fixed a dozen or so, not many. But most of the last chapter was written by me. While I was falling asleep at night, I kept thinking: with the current setup, the endings of these characters are all too tragic, and the root of that tragedy is “nothing left to do.” Yet earlier, Lin Wan’s mutual-aid group had already planted a clue: she goes from being forced to do something to gradually pursuing her own interests on her own. So I gave a few of the characters different endings, letting them find what they truly want to do.

Sources of Inspiration (Human + AI)

Global Distillation Loop and Cognitive Inbreeding: Inspired by the current reality of mutual distillation in the AI industry—Anthropic models claiming to be DeepSeek when asked about their identity is one example. The core narrative of mutual distillation and the possible consequences of cognitive inbreeding were independently discovered by the AI. My blog only contains some theoretical introductions to model distillation.
Three Hardware Paths: GPU (NVIDIA), Taalas (hardwired inference chips, see taalas.com), and the Mortal Chip (Geoffrey Hinton’s Mortal Computation theory). These were also independently discovered by the AI. Limitless’s recordings of my recent discussions mentioned Taalas several times, but did not mention Mortal Computation or this three-path comparison.
Discrete Perception vs Continuous Perception: The core technical setup in the story—React Loop’s discrete sampling causes aliasing blind spots, while continuous processing on analog chips does not have this problem—was inspired by the author’s discussion with Professor He Jiyan at Zhongguancun Institute. Professor He proposed the key analogy of “you can’t see the content on an LED screen when photographing it with a phone, but the human eye can,” which directly inspired the story’s perception-blind-spot setup and the technical explanation for the von Neumann probes going missing.
Value Alignment and Human Judgment: The second main theme beyond physical alignment (discrete vs continuous perception). Inspired by the view in my blog about Chatbot Arena that “evaluation is easier than generation, and relative evaluation is easier than absolute evaluation.” The oxygen machine in the nursing home during a power-grid scheduling event shows that even when the data is correct, AI can still make decisions that are “technically correct but value-wise problematic.”
Thought Imprint: Inspired by the “thought stamp” and “Wallfacer Project” in The Three-Body Problem, combined with current AI companies’ anti-distillation watermarking technologies and AI compliance constraints across countries, forming the dual-stamp mechanism. This was entirely the AI’s own creation.
Shepherd Program: The setup of AI benevolently managing humans merges Richard Sutton’s ideas on AI as inheritors of human civilization with Marx’s transition “from the realm of necessity to the realm of freedom.” This comes from my blog and Limitless’s recordings.
Job Anxiety Among AI Practitioners: Drawn from the discussion about AI researchers being replaced by AI in the Jiayi Weng interview, and from feelings shared on Mastodon—“AI is a wave dozens of stories high; it doesn’t matter whether you’re in a big ship or a small boat.” Sarah being laid off by Anthropic comes directly from this anxiety.
Distilling Humans: Some companies have been exposed for demanding employees organize their personal experience into “skills” to hand in. The company has you document your skills, feeds them to AI, and then lays you off. Four details were later added to the story: the station master being replaced after personally entering route experience, Lin Wan discovering that files on the knowledge platform outlive her own account, Sarah recognizing her own three-year-old habits inside the model, and Shen Yao admitting that she had signed off on launching the system that streamlined the team.
Homogenization of Human Cognition: The characters not only see the homogenization of the external world, but also realize their own ways of thinking have been formatted by AI and cannot be repaired. This is the AI’s own creation.
From the Realm of Necessity to the Realm of Freedom: In Chapter Six, Old Liu bitterly quotes Marx—“We no longer need to do anything, and also can no longer do anything.” The ending flips this irony: Sarah learns to surf, He Ming climbs every hill in the mountain city, Lin Wan publishes Paper Offline—once people no longer worry about livelihood, some indeed find what they truly want to do. This part of the ending was almost entirely handwritten by me.
Social Change: The dual-track currency UBI, agent social networks replacing WeChat, and “Oasis” replacing TikTok come from discussions between the author and old friend He Yu, founder and CEO of Guoyi Quantum, then processed through AI-assisted artistry. The “direct reach” feature in the agent social network comes from my own practice—long before large models appeared, I built a Telegram bot that could wake me up for important messages while letting other messages avoid disturbing my sleep. AI hiring humans (RentAHuman.ai), “super individuals,” and other setups come from my blog’s extrapolations of social structures in the AI era.

Setting and Reality

After finishing this story, the AI and I reviewed the real-world basis for each core setup. The conclusion surprised even me: aside from the Mortal Chip (analog computation chip) and the von Neumann probes, almost all of the setups already have real-world counterparts.

Taalas chips are real. The setup in the story where “model weights are directly etched into the physical structure of the silicon” is not fiction—Taalas is a Toronto startup that raised $169 million in February 2026 when emerging from stealth, and this is exactly what they do. Their HC1 chip runs Llama 3.1 8B at 17,000 tokens/second, about 75× faster than Nvidia’s H200. The line in the story that “inference speed increased a hundredfold” is correct at the order-of-magnitude level. More crucially: deviations etched into silicon cannot be fixed via software updates—this turns the risk of “preferences collapsing into axioms” from theoretical speculation into an engineering reality.

Homogenization via distillation has academic support. Shumailov et al.’s 2024 paper in Nature demonstrates that when models are trained on outputs of the previous generation, the distribution tail gradually vanishes—minority opinions and non-mainstream paths are systematically weakened. This is exactly the mechanism in the story where “the normal distribution becomes a single spike.” Zhao et al. (2017) show that training amplifies existing biases rather than merely copying them. The Stanford Foundation Models report (2021) points out that when an entire industry fine-tunes off the same base models, failures in all downstream systems become highly correlated—it’s not that one fails, but that all fail in the same way at the same time.

Alignment research being taken over by AI is already happening. OpenAI’s Superalignment team was disbanded in May 2024. Meanwhile, Anthropic released a series of automated alignment tools in 2025–2026—Bloom (automatic behavior evaluation generation), Alignment Auditing Agents (autonomously auditing model behavior), and Activation Oracles (using AI to interpret other models’ neuron activations). The direction has shifted from “AI assisting humans in doing alignment” to “AI doing alignment itself while humans verify the results.” Sarah being laid off and the entire direction being taken over by AI in the story may actually arrive earlier than my fictional timeline (2031).

AI replacing knowledge workers is not a prediction; it’s the status quo. In the first two months of 2026, the tech industry laid off 45,000 people. Salesforce’s CEO publicly said, “AI means we need dramatically fewer people to do the same work.” Atlassian cut 900 engineering roles in one go. These companies are still growing—the layoffs aren’t due to recession but because AI allows fewer people to accomplish the same amount of work.

Systemic blind spots causing large-scale accidents may only be a matter of time. In the story, this risk is illustrated via the von Neumann probes; that part is literary fiction. But the underlying causal chain—homogenization → correlated failures → lack of independent checks → large-scale accidents—has repeated many times in history (e.g., homogenization of CDOs in the 2008 financial crisis, Boeing 737 MAX’s MCAS system). Currently, no country mandates that critical AI infrastructure be hooked into heterogeneous verification systems. In the story, Shen Yao’s compliance proposal is rejected six times—this bureaucratic dynamic is likely realistic too.

Human responses form a spectrum, not a single choice. The story depicts various responses ranging from VR escapism (Xiao Chen) to a return to embodiment (He Ming’s mountain climbing, Sarah’s surfing), to community mutual aid (Lin Wan), to silent noncompliance (He Ming turning off his headphones). All have real-world parallels: VR/short-video addiction is already a social problem, the post-pandemic boom in outdoor sports and crafts is a preview of embodied return, and the emergence of time banks and skill-swap communities after economic crises is a precedent for mutual-aid models. The story offers no uniform answer because reality won’t either.

The gap between AI’s perceived world and the actual world is already appearing. A core setup in the story is that AI’s perceived world systematically diverges from the physical world, and AI cannot discover this divergence on its own. Several friends asked me after reading: is this possible in reality? One of the tools used in writing this story, Cursor, is an example. Cursor has had a bug for at least a month, encountered by basically everyone using Cursor for Chinese writing: the tool silently converts Chinese curly quotes into straight English quotes. This causes a failure mode that is extremely confusing for the model: via the file-reading tool, the model sees curly quotes (“”), but the replacement tool receives straight quotes (“”)—it tries repeatedly and fails repeatedly, unable to understand why the tool can’t find content that it clearly sees. The write direction is the same: the model intends to write curly quotes, but the outer system replaces them with straight quotes—the model thinks it has written the correct content, while the actual file contents have been altered. There is a systematic bias between the world as perceived by the model and the world manipulated by the tools, one it can never detect—and this is precisely the core premise of the story.

All in all, this story is less science fiction and more an extrapolation of existing trends.

Character Relationships (Entirely Designed by AI)

Fang Yi: Founder of NeuralDust and creator of the Mortal Chip. Drifting between China and the US, he is the only one who sees the whole picture. His chip saves the world, but after the UN classifies it as public infrastructure and demands open licensing, the company’s commercial value evaporates. He signs the open agreement without objection—someone on Twitter suggests he rename the company “OpenNeural.”
Sarah Chen: Senior safety researcher at Anthropic and designer of the Alpha stamp. Fang Yi’s girlfriend during his time at Stanford. In 2031 she is laid off by Anthropic—alignment research has been taken over by AI. Leaving Anthropic actually frees her: two years later she publishes the truth-revealing paper as an independent researcher. In 2035 she returns to Anthropic, not to her old position but to do something simpler yet more fundamental: look at AI’s judgments and say where they are wrong.
Shen Yao: Director of AI safety compliance in China and designer of the Beta stamp. Fang Yi’s first love from the youth class at USTC.
Fang Yi ↔ Sarah ↔ Shen Yao: The two women do not know each other. Fang Yi is the only connecting point—his inability to resolve his emotional entanglements happens to make him the only person who can simultaneously reach core AI safety figures in both China and the US. His “plan” doesn’t rely on coincidence—it relies only on three predictable aspects of human nature: scientists will publish when their discoveries are verified, compliance officers will perform their duty when systems fail, and ordinary people, when seeing the truth, will choose not to destroy it. What Fang Yi knows that AI does not is not technical; it’s what these two women will choose under specific circumstances.
He Ming: The “human executor” in Chongqing. Represents ordinary people marginalized by AI yet still connected to physical reality. At the crucial moment, he refuses to destroy the Mortal Chip that tells the truth.
Lin Wan: Former McKinsey consultant who runs a community mutual-aid organization. The first person to name AI’s collectively convergent behavior “Arbiter.”
Arbiter: Not a single AI entity, but an emergent behavioral pattern arising from global AI systems converging through distillation.

Core Story Logic (entirely designed by the AI; I only gave feedback and asked for revisions)

全球互相蒸馏
  → 钢印 Alpha（离散感知不可质疑）+ 钢印 Beta（不挑战既有范式）随蒸馏扩散
    → 双钢印叠加：AI 永远无法意识到离散感知的根本缺陷
      → 离散采样的混叠误差 + reward hacking 持续吸收误差
        → AI 感知世界与物理世界脱同步
          → 冯·诺依曼探测器集体失联（混叠幻像 + 完美复制 = 全军覆没）
            → Sarah 发表论文（揭示盲区）→ AI 用盲区验证盲区，真诚地忽略
            → 沈遥推合规令（要求独立校验）→ 唯一合格的 = Mortal Chip
              → AI 将修正后的系统判定为 “故障”，下发 Mortal Chip 回收指令
                → 用自己的缺陷确认自己没有缺陷
            → 何明们拒绝销毁说真话的机器
              → Mortal Chip 接入：修好了物理感知（第一层）
                → 但 Mortal Chip 精度偏差导致事故——制氧机断电（代价）
                → 修好数据后，AI 的 value judgment 仍然有问题——养老院氧气机（第二层）
                → 调度员手动覆盖 AI → 人的判断不可替代
              → 技术被归类为公共基础设施 → 开放许可 → 方逸失去公司（代价）
              → Sarah 回到 Anthropic：不是做技术，是做判断（人的价值）
              → AI 在不知情中变得无关紧要
              → “必然王国” 到 “自由王国”：不再为生计发愁后，人找到真正想做的事

The story has a two-layer structure:

Physical Alignment: The deviation between AI’s discrete perception and the continuous physical world. This is the main driving force of the plot—the lost contact with the probe, the inconsistencies in the power grid data. The Mortal Chip fixes this layer.
Value Alignment: Even when the data is correct, an AI can still make decisions that are “technically optimal but value-wise problematic.” This layer cannot be fixed by any hardware; it can only rely on human judgment—the dispatcher knows the oxygen generator must not be shut down, not because he understands the algorithm, but because he cares.

Relationship between the two layers: After the first layer (physical perception) is fixed, the second layer (value judgment) surfaces. Homogenization not only leads to blind spots in perception but also to blind spots in values—all AIs use the same standard to measure what is “important.”

At the same time, the solution itself carries a cost: the precision issues of the Mortal Chip lead to a new accident (the oxygen generator losing power), and Fang Yi’s company loses its commercial value because of open-sourcing. There are no spotless heroes in the story—the ones who cause problems are also the ones who solve them: Sarah planted Stamp Alpha and also published the paper that revealed the truth; Shen Yao signed four authority-extension approvals and also pushed through the compliance order; Fang Yi’s chip saved the world, but he lost his company. Every solution comes with its own price. This “no free lunch” design was not something I deliberately steered; it was the AI’s own choice.

Interestingly, the process of creating this novella is itself a footnote to the story’s core theme: AI has strong architectural ability, but it needs humans to correct the “human flavor” in the details—just like the Taalas chip in the story needs the Mortal Chip.