Creation Notes for "Distillation"
This note records the background and sources of inspiration for the sci‑fi story “Distillation”.
Background
This is the first novel I’ve ever written. At the beginning I didn’t plan to make it a proper science fiction story; I just wanted to combine some ideas I heard from Professor He Jiyan with my recent thoughts, and reason out what the future might look like. But the AI‑generated plot and worldbuilding—such as the competition between three technical approaches, and the mutual distillation leading to cognitive inbreeding—went far beyond my expectations in many ways, so I felt it was worth pushing further and polishing it a bit more. Please forgive the parts that are not well written.
As a frontline researcher in AI, after OpenClaw went viral this Spring Festival, many people around me fell into anxiety—worried about being replaced by AI. Some people describe AI as a slowly rising flood that starts by submerging the lowest‑level jobs, then rises layer by layer, higher and higher, until one day it reaches them. Others describe AI as a giant wave dozens of stories high: no matter whether you’re in a small boat or a big one, you’ll be overturned in the end—so you might as well enjoy surfing.
These discussions have generated a lot of anxiety. This story weaves together what I’ve learned from talking with many experts in the field and my own long‑term reflections. It’s not a purely fictional story, but a theoretical extrapolation based on current technological progress—many of the things it describes are quite likely to really happen. Of course, because the impact of these things on society would be so great, most people may find them utterly implausible. So rather than write it as a technical report, I felt it would be better to present it in the form of a story, which might be easier for people to accept.
The story itself is also a collaboration between me and an AI—it was generated by an AI (Claude Opus 4.6), using as material several recent posts from my blog—including “Digital Worker”, “A Pale Blue Dot in the Age of AI”, “Permissions, Collaboration, and Employment for AI Agents: Lessons from Moltbook”, “Silicon Valley AI Insights 2025”, and also a couple of days’ worth of recorded conversations from Limitless AI.
The writing process of this story demonstrates one point: for both humans and AI, context is the most important thing. If you just ask an AI to write a sci‑fi story, or give it a few simple elements to combine, it’s unlikely to reach this level of depth. The key is that I gave it enough context. One of my advantages is that I really like to share, and I write a lot of my thoughts on my blog; at the same time I also have Limitless (now acquired by Meta and discontinued; similar products in China include Plaud, Anker Beans, etc.)—a wearable AI recording device that automatically records and transcribes conversations, essentially a 24‑hour AI input feeder. In everyday conversations with friends and at the dinner table, people actually say a huge amount that’s rich in information and insight, but most of it is never recorded and is lost as soon as it’s spoken. My habit is to wear it in almost all situations—this way I can keep everything I’ve thought about, and at the same time make my AI smarter. Whether I’m writing code, blog posts, or slides, I use information from these recordings. For me, my blog and Limitless are the two most important sources of context.
The thing about this story that struck me the most is that the AI was able to take those off‑the‑cuff ideas I had previously recorded, together with things I’d shared on my blog, and use them to write such a fully fleshed‑out piece with genuinely new insights. Many of the seeds of inspiration—for example, the discussion of Taalas and the idea of AI managing humans—were scattered across recordings of different conversations; it was the AI that strung them together into a complete story. Another thing that surprised me: in the first draft, the AI wrote a lot of life scenes—USTC, Wudaokou, San Francisco—throughout the whole process I never explicitly gave it this information, yet most of what it wrote was remarkably fitting.
Writing Process
The entire process—from generating the first draft to later revisions—used the same Writer‑Reviewer paradigm. Specifically, two Agents worked in alternation:
- Writer Agent: a Coding Agent (Claude Code), responsible for searching the web for information, then drafting and revising the story.
- Reviewer Agent: another Coding Agent (also Claude Code, but with a different prompt), which is automatically triggered when the Writer finishes a round of writing. It reviews from several angles and gives feedback: narrative consistency, factual correctness, scientific plausibility of the setting, and an overall assessment of whether the piece is suitable for publication as a sci‑fi story.
After the Reviewer gives its comments, the Writer revises based on the feedback, then the draft is reviewed again—this loop repeats. In each round, the Writer Agent works for about half an hour—first searching for information, then thinking for over ten minutes, then drafting or revising; the Reviewer Agent spends about 15 minutes reviewing and outputs a series of questions and feedback. Each iteration takes about 40–50 minutes.
Why use this external loop instead of asking the AI to get it right in a single pass? Because current Agents really like to “slack off”—even if you tell them to think carefully and self‑revise, they often just go over it once and stop looking. Only through an external loop that forces them to keep reflecting will they spend the time to think deeply and revise further. Simply putting “please check carefully” in the prompt is not enough; you need to guarantee that reflective loop at the architectural level.
The writing was divided into three stages:
Stage 1: Outline (about 2 hours). I first iterated the outline three times using the Writer‑Reviewer paradigm. After each round of AI review, I stepped in—pointing out which of the Reviewer’s comments were most important to address, and adding my own new ideas. After three rounds, the outline was basically fixed.
The biggest problem at the outline stage was that the AI’s first version had a strong “power fantasy web‑novel vibe”—it turned the protagonist into a domineering‑CEO‑type figure, the company saved the world, and it all ended in flowers and applause. But the real world is not that simple. In addition, the initial AI setting had the three people meet at an academic conference and be only loosely acquainted. I felt that if the later life‑and‑death collaborations were between people without a foundation of trust, it wouldn’t be very believable. So I made an adjustment: I set Fang Yi as male and the other two as female, and gave them a shared emotional history in the past—only then does the building of trust make sense. But after the AI revised this, it drifted into romance‑novel tropes, writing a lot about how two people flirted with each other. That didn’t fit the theme of the piece, so I finally asked it to remove the romantic plotlines and keep only the relationship backstory. This kind of back‑and‑forth tug‑of‑war happened many times at the outline stage.
Stage 2: First Draft (about 2 hours, with no human involvement at all). Once the outline was set, the AI began drafting the main text according to it, continuing with the same Writer‑Reviewer paradigm for three automatic rounds. This process ran overnight while I was asleep.
Stage 3: Iterative Revision. Once the first draft was finished, the problems appeared. Although the overall structure and core ideas of the AI‑generated text were good, there was always a certain “machine feel” in the details. First, the AI lacks real‑life experience, so some details don’t match reality—current Agents search the web and verify details when writing, which is already much better than raw LLM output, but at the end of the day, AI is not human: the more detail it gives, the more fake it sounds. Second, some plot points were not well thought through and didn’t align with scientific reality. Third, some parts were bloated. Overall, AI lacks the ability to painstakingly polish. This is a bit like the so‑called “Moravec’s paradox”—AI can solve complex math problems, but has a hard time writing a piece of text that feels truly human.
I pointed out these problems and provided several more blog posts of mine as new sources of perspective (including ones on value alignment and unemployment anxiety among AI practitioners), then had it continue revising with the same paradigm. It ran five more rounds overnight, basically four hours, again while I was asleep.
At noon the next day, I read through the AI’s revised draft and gave it more specific feedback on details—for example, certain things related to the real world and some parts of the setting that were not very appropriate, about a dozen issues in total. Then I had it run another round, I read again, gave feedback, and the AI revised again.
Another recurring tug‑of‑war in the revision stage was that the AI always gravitated towards a Hollywood‑blockbuster ending. Regarding “how humans defeat AI,” its initial plan was that humans persuade the AI. I felt this didn’t make sense: if an AI is already powerful enough to manage humans, you can’t simply defeat or persuade it. After a long discussion, I proposed the current solution: let AI defeat itself—it can’t see its own flaws, attempts to recall the Mortal Chip, but ordinary people don’t follow its orders, while the AI is still using its own blind spots to verify that it has no blind spots. For the ending, the AI also tended to think too much in absolute terms, such as assuming the Mortal Chip was perfect and had no flaws. But compared to digital chips, analog chips definitely have disadvantages in things like precision; the two should be complementary, and practical deployment would certainly expose problems. These are all things I later asked it to add.
There’s another chronic issue with AI writing: it’s extremely used to describing things in the style of “expository writing” or “technical reports”—it keeps explaining what the technology is like, but doesn’t understand “Show, Don’t Tell.” Although I’ve never written fiction before, my Chinese teachers did teach us: you have to show things through what characters say and do; you can’t lecture the reader. I emphasized this repeatedly in many of my prompts, but in the final manuscript there’s still too much “Tell”—in the end I was too lazy to keep fixing it.
One more observation: in some sense, writing fiction is more mentally taxing than writing academic papers. A paper usually revolves around one main thread, with linear logic; a story advances along multiple threads at once—timeline, causal chain of events, character behavior—and all of them have to be completely self‑consistent. Even harder is controlling point of view: each character can only know information within their own perspective; you can’t let them slip into an omniscient “god’s‑eye view.” When you arrange the revelation of each piece of information, you must consider “Is it possible for this character to know this at this particular moment?”
At the end there were still some small issues. I manually fixed a dozen or so; not many. But the last chapter was mostly written by me. Lying in bed at night I kept thinking: given the current setup, the endings for these characters are all too tragic, and the root of the tragedy is that they “have nothing left to do.” But earlier, Lin Wan’s mutual-aid group had actually already planted a clue: she went from being forced to do something to gradually pursuing her own interests voluntarily. So I arranged different endings for some of the characters, letting them find what they truly wanted to do.
Sources of inspiration (Human + AI)
- Global distillation ring and cognitive inbreeding: Comes from the reality of today’s AI industry, where everyone is distilling from everyone else—Anthropic models calling themselves DeepSeek when asked about identity is one example. The core narrative of mutual distillation and the possible consequences of cognitive inbreeding were independently discovered by the AI. My blog only has some theoretical introductions to model distillation.
- Three hardware paths: GPU (Nvidia), Taalas (hard-wired inference chips, see taalas.com), Mortal Chip (Geoffrey Hinton’s Mortal Computation theory). These were also independently discovered by the AI. Limitless logs show that my recent discussions mentioned Taalas a few times, but did not mention Mortal Computation or the comparison of these three paths.
- Discrete perception vs. continuous perception: The core technical premise in the story—the aliasing blind spot caused by the discrete sampling of React Loop, versus the absence of this issue in continuous processing on analog chips—was inspired by a discussion between the author and Professor He Jiyan at Zhongguancun Institute. Professor He proposed the key analogy of “you can’t see the content of an LED screen when filming it with a phone, but the human eye can,” which directly inspired both the perception-blind-spot setup and the technical explanation for why the von Neumann probes went missing.
- Value Alignment and human judgment: The second layer of theme beyond physical alignment (discrete vs. continuous perception). Inspired by a view from my blog post about Chatbot Arena: “evaluation is easier than generation, and relative evaluation is easier than absolute evaluation.” The nursing-home oxygen machine incident in the power-grid dispatch story arc illustrates how an AI can still make judgments that are “technically correct but value-wise wrong,” even when its data is correct.
- Thought imprint: Inspired by the “thought imprint” and “Wallfacer Project” in The Three-Body Problem, combined with current AI companies’ anti-distillation watermarking techniques and various countries’ AI compliance constraints, I built the dual-imprint mechanism. This part was entirely the AI’s own creation.
- Shepherd Program: The idea of AI benevolently managing humanity merges Richard Sutton’s views on AI as the inheritor of human civilization with Marx’s transition from the “realm of necessity” to the “realm of freedom.” This comes from my blog and recordings in Limitless.
- Job anxiety among AI practitioners: Comes from my interview with Jiayi Weng about AI researchers being replaced by AI, as well as feelings expressed on Mastodon—“AI is a wave tens of stories high; it doesn’t matter whether you’re in a big ship or a small boat.” The plot of Sarah being laid off by Anthropic is directly sourced from this anxiety.
- Homogenization of human cognition: The characters not only witness the homogenization of the external world, but also realize that their own ways of thinking have already been formatted by AI and cannot be repaired. This is the AI’s own invention.
- From the realm of necessity to the realm of freedom: In Chapter 6, Old Liu bitterly cites Marx—“Now there’s nothing left that needs doing, and nothing left you can do.” The ending flips this irony: Sarah learns to surf, He Ming climbs every hill in the mountain city, and Lin Wan publishes Paper Doesn’t Go Online—once people no longer have to worry about making a living, some of them indeed find what they really want to do. This part of the ending is almost entirely hand-written by me.
- Social change: The dual-track UBI currency, agent-based social networks replacing WeChat, and “Oasis” replacing TikTok are based on discussions between the author and an old friend, He Yu, founder and CEO of Origin Quantum, then artistically processed by the AI. The “Direct Reach” feature in the agent social network comes from my own practice—long before large language models appeared, I built a Telegram bot that could wake me up for important messages while letting everything else stay silent. The setting of AI hiring humans (RentAHuman.ai), “super individuals,” etc. comes from my blog’s extrapolations about social structures in the AI era.
Settings and reality
After finishing this novella, the AI and I went back over the factual basis for each core premise. The conclusion surprised even me: except for the Mortal Chip (analog computation chips) and von Neumann probes, almost every element has a real-world counterpart.
Taalas chips are real. The premise in the story that “model weights are directly baked into the physical structure of the silicon” is not fictional—Taalas is a Toronto startup that exited stealth in February 2026 with a $169 million funding round and is doing exactly this. Its HC1 chip runs Llama 3.1 8B at 17,000 tokens/second, roughly 75× faster than Nvidia’s H200. The line in the story about “inference speeds increasing by two orders of magnitude” has the right order of magnitude. More importantly, deviations etched into silicon cannot be fixed by software updates—this turns the risk of “preferences collapsing into axioms” from a theoretical concern into an engineering reality.
There is academic support for homogenization via distillation. Shumailov et al. (2024), in a paper published in Nature, showed that when models are trained on the outputs of previous-generation models, the tails of the distribution slowly disappear—minority viewpoints and non-mainstream paths are systematically weakened. This is exactly the mechanism by which the “bell curve turns into a needle” in the story. Zhao et al. (2017) showed that training amplifies existing biases rather than merely copying them. The Stanford Foundation Models report (2021) pointed out that when an entire industry fine-tunes from the same base models, failures of all downstream systems become highly correlated—it’s not that one model fails, but that all of them fail in the same way at the same time.
Alignment research being taken over by AI is already happening. OpenAI’s Superalignment team was disbanded in May 2024. Meanwhile, between 2025 and 2026 Anthropic released a series of automated alignment tools—Bloom (automatic generation of behavioral evaluations), Alignment Auditing Agents (autonomous auditing of model behavior), and Activation Oracles (using AI to interpret other models’ neuron activations). The focus has shifted from “AI assists humans in doing alignment” to “AI does alignment itself, and humans verify the results.” The plot in which Sarah is laid off and the whole alignment direction is taken over by AI, set in 2031 in the story, may in reality arrive sooner.
AI replacing knowledge workers is not a prediction; it is current reality. In the first two months of 2026, 45,000 people were laid off in the tech industry. The CEO of Salesforce publicly stated that “AI means we need dramatically fewer people to do the same work.” Atlassian cut 900 R&D positions in a single round. These companies are still growing—layoffs are not due to economic recession, but because AI allows fewer people to do the same work.
Systemic blind spots leading to large-scale accidents may be only a matter of time. In the story, von Neumann probes are used to dramatize this risk; that part is literary invention. But the underlying causal chain—homogenization → correlated failures → no independent verification → large-scale accidents—has recurred throughout history (e.g., the homogenization of CDOs in the 2008 financial crisis, the MCAS system in the Boeing 737 MAX). Currently, no country mandates that critical AI infrastructure must be connected to heterogeneous verification systems. In the story, Shen Yao’s compliance proposal gets sent back six times—this bureaucratic dynamic is probably realistic.
Human responses form a spectrum, not a multiple-choice answer. The story portrays a variety of responses: VR escapism (Xiao Chen), return to embodiment (He Ming’s climbing, Sarah’s surfing), community mutual aid (Lin Wan), and silent non-compliance (He Ming turning off his headset). All of these have real-world counterparts: VR/short-video addiction is already a social problem; the post-pandemic boom in outdoor sports and handicrafts is a preview of the return to embodiment; the emergence of time banks and skill-swap communities after economic crises is a precursor of mutual-aid models. The story offers no single unified answer, because reality won’t either.
The gap between AI’s perceived world and the actual world is already appearing. One core premise of the story is that the world as perceived by AI systematically differs from the physical world, and the AI cannot detect this discrepancy on its own. Several friends asked after reading whether this could really happen. One of the tools used to write this novella, Cursor, is an example. There was a bug in Cursor that had existed for at least a month and had been encountered by basically everyone using Cursor for Chinese writing: the tool silently converts Chinese curly quotes into straight English quotes. This caused a failure mode that was extremely confusing for the model: via the file-reading tool it sees curly quotes (“”), but the replace tool receives straight quotes ("")—it tries repeatedly, fails repeatedly, and cannot understand why the content it can clearly see cannot be found by the tool. The same applies in the write direction: the model intends to write curly quotes, but the outer system replaces them with straight quotes—the model believes it has written the correct content, yet the actual content in the file has been altered. There is a systematic discrepancy between the world the model perceives and the world operated on by the tools—one it can never detect. This is precisely the core proposition of the story.
All in all, this novella is less science fiction than a projection of trends already underway.
Character relationships (entirely designed by the AI)
- Fang Yi: Founder of NeuralDust, creator of the Mortal Chip. He drifts between China and the U.S. and is the only person who sees the whole picture. His chip saves the world, but after the UN classifies it as public infrastructure and requires open licensing, the company’s commercial value evaporates. He signs the open license agreement without objection—someone on Twitter suggests he rename the company OpenNeural.
- Sarah Chen: Senior safety researcher at Anthropic, designer of Imprint Alpha. Fang Yi’s girlfriend from his Stanford days. She is laid off by Anthropic in 2031—alignment research has been taken over by AI. Leaving Anthropic actually sets her free: two years later she publishes a paper revealing the truth as an independent researcher. In 2035 she returns to Anthropic, not to her previous role but to do something simpler and more fundamental: look at AI’s judgments and say where they are wrong.
- Shen Yao: China’s Director of AI Safety and Compliance, designer of Imprint Beta. Fang Yi’s first love from the Special Class for the Gifted Young at USTC.
- Fang Yi ↔ Sarah ↔ Shen Yao: The two women do not know each other. Fang Yi is the only point of connection—his inability to resolve his emotional entanglements happens to make him the only person who can simultaneously access the core AI safety figures in both China and the U.S. His “plan” does not rely on coincidence—only on three predictable aspects of human nature: scientists publish when their discoveries are validated, compliance officers do their duty when systems fail, and ordinary people choose not to destroy the truth when they see it. The one small thing Fang Yi knows that the AI does not is not technical; it is what each of the two women will choose under specific circumstances.
- He Ming: The “human executor” in Chongqing. He represents ordinary people marginalized by AI but still connected to physical reality. At the crucial moment, he refuses to destroy the truth-telling Mortal Chip.
- Lin Wan: Former McKinsey consultant who runs a community mutual-aid organization. The first person to name AI’s collective convergent behavior “Arbiter.”
- Arbiter: Not an AI entity, but an emergent collective behavioral pattern that appears after global AI systems converge through mutual distillation.
Core Story Logic (entirely AI‑designed; I only gave feedback for revisions)
1 | 全球互相蒸馏 |
The story has a two-layer structure:
- Physical Alignment: the gap between AI’s discrete perception and the continuous physical world. This is the main driver of the plot—probe loss, inconsistencies in the power grid data. The Mortal Chip fixes this layer.
- Value Alignment: even if the data is correct, an AI can still make decisions that are “technically optimal but value‑wise problematic.” This layer cannot be fixed by any hardware; it can only rely on human judgment—the dispatcher knows the oxygen machine must not be cut off, not because he understands algorithms, but because he cares.
Relationship between the two layers: once the first layer (physical perception) is fixed, the second layer (value judgment) surfaces. Homogenization leads not only to perceptual blind spots but also to value blind spots—all AIs use the same standard to measure what is “important.”
At the same time, the solution itself has a price: the Mortal Chip’s precision issues cause a new accident (the oxygen generator power cut), and Fang Yi’s company loses its commercial value because of open‑sourcing. There are no spotless heroes in the story—the people who cause the problems are also the ones who solve them: Sarah planted the Alpha watermark and also published the paper revealing the truth; Shen Yao signed four authority‑extension approvals and also pushed through the compliance order; Fang Yi’s chip saved the world, but he lost his company. Every solution comes with its own cost. This “no free lunch” design was not something I deliberately steered; it was the AI’s own choice.
Interestingly, the creation process of this story is itself a footnote to the story’s theme: AI has strong architectural ability, but needs humans to correct the “human touch” in the details—just like the Taalas chip in the story needs the Mortal Chip.