Zhihu Academic Bar Talk: What Moment Made You Feel Like the World Had a Bug?
On the evening of November 15, 2024, at the Zhihu Academic Bar, I, along with prominent figures like Kai-Fu Lee, Zhiyuan Liu, and Guohao Dai, participated in an open mic sharing session.
Question:
“Vulnerabilities & Bugs—What moment made you feel like the world had a bug?”
On Zhihu, there are several highly upvoted questions about bugs, such as “What moment made you feel like the world had a bug?” and “What are some bugs that left you dumbfounded?”
However, it’s not scary when the world has a bug; what’s scary is when AI discovers a bug.
Recently, did AI discover a major security vulnerability in the real world for the first time? A vulnerability in SQLite was fortunately discovered by Google’s AI Agent, and after being fixed, it caused no damage. Could it be that with further evolution, AI could permanently prevent global blue screen incidents like those from Microsoft? This possibility is exciting.
Answer:
I’ve also seen the highly upvoted question on Zhihu, “What moment made you feel like the world had a bug?” Many answers suggest that the world is a simulator, and many phenomena in quantum mechanics seem to indicate that the world is a real-time rendered game. I’ve always felt like I’m living in “The Truman Show” because I’ve encountered too many coincidences, with strange things happening that seem to have a very low probability, making me wonder if there’s a director constantly adding drama to my life.
Speaking of AI discovering software vulnerabilities, it’s not a new thing. The U.S. military’s intelligent network vulnerability mining system, Mayhem, can automatically perform binary analysis, password cracking, reverse engineering, vulnerability mining, and automatic programming to exploit vulnerabilities. In the 2016 DARPA Cyber Grand Challenge, Mayhem discovered a vulnerability in SQL Slammer in just 6 minutes, with no human intervention; all work was done automatically by AI.
Today’s large models are more powerful than systems like Mayhem because AI’s ability to read code is incredibly strong. Recently, I wanted to create a voice assistant to control my smart home, but the smart home used a binary private protocol, which seemed too cumbersome to reverse engineer. I decompiled the Android app controlling the smart home, with over ten thousand lines of code, and fed it into Claude 3.5 Sonnet. Surprisingly, it directly gave me a Python implementation. Of course, there were bugs in this implementation, but I combined packet capture results to let Claude modify it, and after a few iterations, it worked. It took me less than an afternoon, whereas without AI, it might have taken me several days.
This incident shows that AI’s strongest point is its ability to read a large amount of content at once, and this long-text reading ability is far superior to humans. There’s a quantitative company I can’t name that relies on reading a large amount of public information, analyzing clues, and making many macro event predictions.
For example, in the recent U.S. election, as Kai-Fu Lee just mentioned, Musk calculated long in advance that Trump would win, which is why he dared to make such bold moves. We know that domestic media reported at 3 PM, which is 11 PM U.S. time, that Trump would win, even before U.S. media reported it. When did everyone know Trump won? Raise your hand if you knew in the morning? There are indeed a few who knew in the morning; I’m very impressed. Before 3 PM, who knew at noon? Oh, not bad, quite a few. I personally knew at 2 PM because I bought some Polymarket, and its winning rate exceeded 99% at 2 PM. This quantitative company knew several hours in advance, in the morning China time.
Another example is last year’s OpenAI internal conflict, where Sam Altman was ousted. They predicted it ten days in advance. One important clue was that two months before the incident, Ilya hadn’t retweeted any company tweets, not even the OpenAI dev day, while other executives mostly did. They told me something that left a deep impression: if ten million dollars were used to create a Character AI chat app, it might not break even, but ten million dollars in OpenAI API fees would be enough to make such predictions, which would definitely be profitable.
These examples of automatic vulnerability mining and macro event prediction show that AI can simulate the world based on a large amount of input information. Therefore, I think OpenAI’s proposal that “generative models are world simulators” is a profound concept. This phrase became popular with Sora this year, but OpenAI’s chief scientist Ilya proposed it back in 2016, and the idea’s roots come from Hinton, who won the Nobel Prize in Physics this year and previously the Turing Award. His research on Boltzmann machines was to create a generative model to simulate the behavior of systems composed of many particles, essentially simulating the world.
Zhiyuan Liu just proposed a great concept, the Moore’s Law of model knowledge density, where the knowledge density of models doubles every 8 months, which is knowledge compression. Kai-Fu Lee also mentioned that last year Yi-Large spent 30 million dollars on training, while this year’s Yi-Lightning spent less than 10 million dollars, with inference costs reduced by 30 times, and the capability is even stronger. The key here is system optimization from algorithms to infrastructure. Guohao Dai also talked about software and hardware co-optimization.
With these software and hardware co-optimizations, in the next one to two years, our flagship phones might support models as large as 7B. Some leading phone manufacturers are already working on it. As model knowledge density increases, a 7B model on the device side might reach the intelligence level of GPT-4o mini, sufficient for many daily tasks. This will significantly address the issue of AI inference costs.
AI as a world simulator essentially means achieving the same level of intelligence with several orders of magnitude less energy. That’s why I really like Moonshot’s slogan, “Exploring the optimal solution for converting energy into intelligence.”
A world simulator can not only passively predict the future but also actively influence it. For example, if I have two different choices, I can put them into the world simulator to see which outcome I want, and our choices might act like a lever, achieving a significant effect with minimal effort.
Moreover, many of us can’t figure out what we want or what kind of person we want to be. If we can simulate multiple possible futures, we can know what we care about the most. For example, Liu Cixin, the author of “The Three-Body Problem,” said in a speech, “Among these countless possible futures, no matter how prosperous Earth becomes, those without space travel are dim.” Writing science fiction helped Liu discover that space travel is what he cares about the most, and AI can help each of us discover our true selves.
The premise for AI to act as a world simulator must be a large amount of input from the real world; otherwise, its output won’t have enough diversity. Many people ask, if AI can answer any question now, what’s the value of Zhihu? I say the value of Zhihu is the diversity of answers. Most AI products provide answers that are balanced and emotionless. AI finds it hard to have sharp opinions or share personal experiences. But what we readers prefer to see are diverse opinions and experiences.
Therefore, I think for AI-generated content to be interesting, it must have diverse information sources as input. For example, Zhihu’s direct answer product, its information source is actually all user answers, so the content generated by Zhihu direct answer is relatively diverse, allowing us to view the same question from many different perspectives and see many personalized and interesting experiences.
In the long run, I believe diverse information sources like Zhihu can also help AI always adhere to human values and do no evil, which is AI alignment. Why can a content-sharing website relate to AI values?
There’s a famous “paperclip maximizer” thought experiment. If a superintelligent machine’s goal is to produce paperclips, it will eventually turn all the matter in the universe into paperclips, which is obviously a disaster for humanity. This shows that a super-smart AI, if its optimization goal is singular, can be very dangerous.
But on Zhihu, we see all kinds of people with different values and life goals, with different interests and hobbies, and everyone’s life can be exciting. If we let AI learn these diverse values, it won’t fall into the extreme pursuit of a single goal, but rather multiple intelligent agents form an ecosystem, which is the emergence of collective intelligence that Zhiyuan Liu just talked about. I think the emergence of collective intelligence is a great concept, where diverse multiple intelligent agents can explore and discover a larger world together, even creating a larger world.
Thank you, everyone!
Zhihu Academic Bar Poster