Bojie Li

Highly flexible software network functions (NFs) are crucial components to enable multi-tenancy in the clouds. However, software packet processing on a commodity server has limited capacity and induces high latency. While software NFs could scale out using more servers, doing so adds significant cost. This paper focuses on accelerating NFs with programmable hardware, i.e., FPGA, which is now a mature technology and inexpensive for datacenters. However, FPGA is predominately programmed using low-level hardware description languages (HDLs), which are hard to code and difficult to debug. More importantly, HDLs are almost inaccessible for most software programmers.

This paper presents ClickNP, an FPGA-accelerated platform for highly flexible and high-performance NFs with commodity servers. ClickNP is highly flexible as it is completely programmable using high-level C-like languages, and exposes a modular programming abstraction that resembles Click Modular Router. ClickNP is also high performance. Our prototype NFs show that they can process traffic at up to 200 million packets per second with ultra-low latency (< 2µs). Compared to existing software counterparts, with FPGA, ClickNP improves throughput by 10x, while reducing latency by 10x. To the best of our knowledge, ClickNP is the first FPGA-accelerated platform for NFs, written completely in high-level language and achieving 40 Gbps line rate at any packet size.

Joint project with Tianyi Cui for Microsoft Hackathon 2016.

To achieve low TCP flow completion time (FCT) in data center networks (DCNs), it is critical and challenging to rapidly recover loss without adding extra congestion. Therefore, in this paper we propose a novel loss recovery approach FUSO that exploits multi-path diversity in DCN for transport loss recovery. In FUSO, when a multi-path transport sender suspects loss on one subflow, recovery packets are immediately sent over another sub-flow that is not or less lossy and has spare congestion window slots. FUSO is fast in that it does not need to wait for timeout on the lossy sub-flow, and it is cautious in that it does not violate congestion control algorithm. Testbed experiments and simulations show that FUSO decreases the latency-sensitive flows’ 99th percentile FCT by up to ~82.3% in a 1Gbps testbed, and up to ~87.9% in a 10Gpbs large-scale simulated network.

Last night (September 25th), we had a fight that led to a breakup and deleting each other from QQ. It was the most serious argument we’ve ever had. It all started because my self-esteem was hurt when some of my flaws were pointed out, and I got angry, lost my logic, and said many hurtful things. We broke up, deleted and re-added each other several times, but after both of us calmed down, we reconciled, and I promised to sincerely apologize. Jingning even went to my dorm to take the quilt cover back to her dorm to wash and dry it. I was moved to tears! Here, I publicly post an apology, also as a reminder to myself.

Lessons

1. When Jingning is feeling down, I need to comfort her sincerely and not criticize her actions at that time.
2. When Jingning points out my shortcomings or gives me feedback, I shouldn’t rush to defend myself but should remember it, think it over slowly, and not respond hastily. Only the closest people are willing to point out these character flaws, and these are very valuable suggestions. I shouldn’t be too proud and should be brave enough to admit my problems.
3. In any situation, I should avoid being emotionally unstable, acting impulsively, and speaking hurtfully without considering the consequences. Arguing not only wastes time but also damages the relationship.
4. During an argument, we are both irrational, completely forgetting the good times and happiness we’ve shared, as well as the long-term impact of breaking up. Especially me, I often say illogical things or go off on tangents.
5. Life, family, and technology are equally important. I should abandon the prejudice of putting technology above all else and not neglect feelings while focusing on technology.

Incident Review

Beginning

1. Last night (September 25th), Zhang Jingning was not happy and confided in me a lot on QQ.
2. I had just finished rushing a NSDI paper in the morning, came home, chatted a bit, and then fell asleep.
3. I woke up at 00:37, just when she wanted to talk to me. She said she was so bored that she started reading online novels and sent me a few passages. I said those online novels were boring and suggested she read “Those Things of the Ming Dynasty.”
4. She felt we couldn’t communicate and said I didn’t know how to comfort girls.
5. Then I lost my temper, saying I should focus on technology and let her find a guy who would play with her. “When you say you’re in a bad mood, I don’t want to make you feel better but want you to be alone.” She said I was irresponsible.
6. She tried to ease the situation by talking about the experience of picking up girls in novels. I said not only would I not do that, but I also found it very annoying. Jingning said I could read it to improve my emotional intelligence, calling me too much of a “straight man cancer.”

Climax

1. I said, “Most guys I know lie to girls, never say anything bad about them, and always agree with them. That’s human nature, and you just don’t want to admit it.”
2. Jingning was very angry, saying, “You lie yourself and then criticize others. My friends are all honest,” pointing out that I was just making excuses and dragging others down to make myself feel better.
3. Then I thought Jingning didn’t trust my character and started to drive her away…
4. I brought up another unrelated issue: “I find it interesting to discuss problems with other classmates every day, but with you, it’s often boring topics, and sometimes it even makes me unhappy. Why should I be with someone with whom I have no common topics and have to respond to your interruptions while working?”
5. At this point, we started talking about breaking up. Jingning said if she didn’t come to Hefei tomorrow, we should break up. I agreed without thinking, thinking breaking up was easy and that not contacting each other would be less annoying, completely forgetting the joy and deep love we’ve shared over the past year.
6. Jingning asked me to apologize. I said I needed some initiative, or I would be a doormat in the future.
7. I exported the chat logs, and Jingning deleted me as a friend and blacklisted me (she loves doing this every time we argue). This was the 383rd day since we added each other as friends.
8. It was 2:25 AM. We both posted statuses on Renren.

Development

1. We couldn’t sleep. Forty minutes later, Jingning sent a friend request.
2. I started to wake up from my emotions, realizing the nonsense I had said earlier and tried to make amends.
3. Jingning reviewed the hurtful things I said in the chat logs. She said, “You are very selfish. When you argue, you feel you have no needs and think I’m wasting your time. You don’t know how to cherish and care for girls. You don’t deserve to have one.”
4. I admitted my mistakes. Jingning asked me to write an article and post it online and to come to Hefei tomorrow to apologize formally. Then she deleted me as a friend again.

Reconciliation

1. Half an hour later (4:30 AM), Jingning posted a question on Zhihu, “My boyfriend has low emotional intelligence, argues illogically, and says hurtful things. Should I break up?” After posting, she felt that since I was confused and illogical during arguments, she shouldn’t have mentioned breaking up to escalate the conflict. She should have tolerated me, even if I said hurtful things.
2. Jingning felt it was just a sudden argument and asked me to apologize sincerely and promise not to do it again. The atmosphere immediately eased.
3. Jingning asked me to change the title of the Zhihu question from “low emotional intelligence” to “easily impulsive.” We started to rationally think about the problems exposed during the argument.
4. I recalled arguing with my mom when I was a child and realized that my problem was being used to being an excellent student, having too much pride, and not being able to accept others’ disapproval. Jingning said that when I argue with others or make mistakes, I always quickly agree or casually come up with a conclusion, seemingly to give myself an out.
5. Jingning pointed out another good point, saying I had a prejudice of putting technology above all else and didn’t care about life, family, and relationships.
6. I deeply regretted driving Jingning away and easily agreeing to break up.

Conclusion

1. I bought a ticket to Hefei for Sunday morning (Saturday tickets were only available for first-class and business class, which were too expensive, and transferring from Nanjing or Tianjin didn’t work either).
2. Actually, I could have bought a ticket to the nearest station on Saturday morning, stood for the latter part of the journey, and bought a supplementary ticket after getting off.
3. At noon, I found out that the LUG discussion group was talking about what happened to us… Exposing internal conflicts and worrying everyone really disturbed them.
4. This afternoon, I woke up, went to the company to organize experimental code and data, hugged Guoshen’s thigh, and went to Tsinghua to taste the food festival. In the evening, I came back to read the chat logs and summarize the lessons.

Avoid Hurting Those Who Love You

The last time we had a serious argument that almost led to a breakup was about three months ago. The discussion group knew about it too. We thought about separating for a few days to learn a lesson, but we reconciled the next day. Jingning said she should test how I would handle it, but she could never stick to ignoring me. It seems like a paradox: reconciling early means the lesson isn’t deep enough, but reconciling late causes more harm. Now that we’re in a long-distance relationship, if we don’t contact each other for a long time, the feelings will fade, and we’ll eventually break up. I think arguments shouldn’t be handled coldly; the longer it drags on, the greater the harm. Apologies should be timely.

Today (September 26th), Jingning went to my dorm to take the quilt cover back to her dorm to wash. I said it wasn’t necessary, but she did it anyway. I was moved to tears! Last night, I treated her so badly and said so many hurtful things, yet she took the initiative to reconcile in the middle of the night and was so good to me today. I was really touched.

Finding true love is not easy, and it must be cherished. Before impulsively arguing next time, look at our argument summaries on Tower and this article, and think about the good times we’ve shared. Maybe then, I won’t so easily hurt the one who loves me.

In several arguments, the trigger has always been my inability to tolerate Jingning saying anything bad about me. Even though I know I have these flaws, I can’t tolerate Jingning pointing them out, and I always want to find excuses. This is not just a relationship issue but also a personal psychological issue. Being worshipped all the time is not good; having too much pride makes me avoid my shortcomings. Having someone patiently point out my flaws is the best thing.

Preface: During the summer vacation, I messed up the open-source software mirror server of LUG. One of the punishments is to submit reading notes for the books borrowed from the LUG Library. It has been more than two years since I first read this book. It can be said that “The Top of the Wave” opened the door to the IT industry for me. Here, I share my humble opinions with you all.

“The Top of the Wave” has two main lines: one is the rise and fall of technology companies; the other is the regularity of the high-tech industry. The interesting part of this book lies in its first main line, which writes the history of technology companies as fascinating as a novel. The value of this book mainly lies in its second main line, that is, to see the rules through the phenomenon. This article will summarize the rise, glory, and decline of various business empires in “The Top of the Wave”, as well as the development rules of companies and the computer industry.

On July 28, 2015, bind9, the most widely used DNS server in the world, exposed a serious denial of service vulnerability (CVE-2015-5477).

A bit of background knowledge: DNS is a service that maps domain names to IP addresses. When you visit google.com, your computer will ask the DNS server in your area, what is the IP address of google.com? If your neighbor happens to be visiting google.com, the DNS server will directly return its IP; otherwise, this DNS server will ask Google’s official DNS server, get the IP address of google.com, and return it to you. The DNS server in this area is called recursive DNS; if the recursive DNS is down, it will cause the area it serves to be unable to access the internet. Google’s official DNS server is called authoritative DNS; if the authoritative DNS is down, it will cause the websites it serves to disappear from the earth.

DNS Recursive Query (Image Source)

How serious is this vulnerability? Just send a UDP data packet, and you can take down a DNS server. Whether it’s recursive DNS or authoritative DNS, no matter what configuration bind9 has made, as long as this data packet is received by the bind9 process, it will immediately throw an exception and terminate the service.

In mid-May, LUG held a white hat competition, conducting white hat vulnerability testing on campus websites, and awarded prizes based on the vulnerabilities found. The event went relatively smoothly, and hundreds of vulnerabilities in the campus network system were found within a few days. However, for some unknown reason, it attracted hackers.

In the early morning of May 31, the alarms from the monitoring system broke the tranquility of the late night. Dozens of LUG servers unexpectedly failed one after another, and the shocking event of LUG servers being hacked began. From the email sent by server administrator gyf to the Network Information Center for the second time on June 1, one can vaguely feel the tense atmosphere of that year.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

按照事件第一次发生时间排序：

【blog.ustc.edu.cn、freeshell.ustc.edu.cn】202.141.{160,176}.99（网络信息中心虚拟机）
31日 01:43，具体详情见上一封邮件（服务器失联），至今未修复。

【mirrors.ustc.edu.cn】202.38.95.110/25，202.141.{160,176}.110（网络信息中心实体机）
31日 01:51，黑客登陆后执行了 sudo /bin/sh；（P.S. 黑客登陆IP来自128.199.203.28（DigitalOcean），但该IP已被注销）
31日 01:58，系统崩溃， 由于我们设置了内核崩溃后60秒重启，因此该机器重启
31日 02:01，黑客再次登录
1日 19:42，管理员在排查问题时卸载了tun内核模块，随后，所有vlan配置消失。服务器失联。
1日 20:03，我和崔灏进入网络信息中心机房，物理接触服务器
1日 20:30，该服务器所有网络服务恢复正常

【lug.ustc.edu.cn】202.141.162.123（西区图书馆实体机）
31日11:54，服务器失联，无法ping通
31日13:47，我进入西区图书馆机房，发现服务器花屏，故障原因未知
31日14:07，强制重启服务器后，服务恢复
1日 19:58，服务器再次失联，至今未修复

【gitlab.lug.ustc.edu.cn】202.141.{162,176},93、202.38.93.93（东区图书馆机房虚拟机）
1日中午，接到用户反馈部分git仓库故障。
登录服务器发现btrfs文件系统损坏。
尝试通过vSphere Data Protection恢复失败，正在查找原因。。。

【vpn.lug.ustc.edu.cn】202.141.{160,176}.95（网络信息中心虚拟机）
1日20:35，该服务器失联
随后，我们通过vCenter查看，发现该机器正在循环重启。现象与blog服务器极为类似。
且该服务器故障前半小时的auth.log被删除，我们从硬盘中恢复出了部分入侵前后的日志。
服务至今未恢复。

我们正在全力抢修服务器，但由于事情发生得非常集中，服务全部恢复可能需要较长时间。

It was later found out that the server was remotely logged in using my account. In mid-May, the hacker invaded my laptop with a virus spread through a USB drive, implanted a keyboard logger, and remotely controlled my browser to visit some web pages related to network intrusion through unknown means. Since my personal account has no access record and the Bitcoin wallet was not stolen, it is very likely that this was an Advanced Persistent Threat (APT) targeting LUG rather than me personally. In the following half month, the hacker did not make any rash moves, presumably collecting information about the LUG server through various channels. On the night of May 30, the hacker logged into a large number of servers with the server password stolen by the keyboard logger and inserted malicious kernel modules. The hacker also invaded the LDAP database, tampered with the password of an old member who had already left the school, and logged into the most strictly defended mirrors server. The hacker also stole the private key of a VPN user, accessed the server’s internal network, and further invaded servers that were not allowed to be accessed from outside the school.

What this malicious kernel module does seems simple, it randomly modifies a byte on the hard disk every time a file operation is performed. This seemingly prank-like kernel module makes the server run normally when it is just invaded, but when key data is destroyed, system anomalies are discovered, and by this time a large amount of user data and system files have been destroyed. When the administrator tries to scan and repair these damaged files, a large number of file operations are generated, causing more files to be damaged, and they can never be repaired. Even when we NFS mount the backup server to copy backup data, the copied backup is also wrong, which is puzzling to us (fortunately, the backup server is NFS read-only mounted, otherwise the backup itself may also be damaged).

The interruption of the open-source software mirror (mirrors) service with over ten million daily visits, the interruption of the VPN service relied on by thousands of users in the school, the damage to the files in the freeshell virtual machine, the inability to access the blog, and even the LUG homepage cannot be opened, inquiry emails flew in like snowflakes, but the email server also hung up. This hacking incident even alarmed several Linux distributions and upstreams of open-source software. They all expressed that it was unheard of for open-source software mirrors to be hacked. The Freeshell service was terminated due to lack of backup, and the VPN service was no longer publicly run because it acted as an accomplice. This incident exposed many problems with the basic infrastructure of the LUG server, such as the public VPN service and the server using the same internal network, no two-step verification for password login, no alarm and defense for the dangerous operation of inserting kernel modules, and the account of the departed administrator was not disabled. Of course, the fundamental reason was that my laptop was hacked. Afterwards, LUG and teacher James treated me leniently and did not pursue my responsibility. I still feel very guilty to this day.

Background: Virtualbox virtual machine on Windows. Ubuntu 14.04.1 LTS, 3.13 kernel. ext4 file system.

Mistake: A few days ago, I was developing a website on this virtual machine, made N commits, thought I had git pushed, but in fact, the push failed.

Mystery: Today, my colleague git pulled and found no updates, then said I hadn’t been working these days.

Tragedy: Logged into the virtual machine and saw that a few newly written files in the project directory had become 0-byte empty files. (ext4 is so stable, it must be the evil NTFS and Virtualbox in the host machine that caused the trouble)
Many files in the .git directory also became 0-byte empty files. Git prompts that the repository is damaged.

1
2
3
4

$ git status
error: object file .git/objects/71/cbcbbc9d06a74f2fd8ea9109b81b88086f1430 is empty
error: object file .git/objects/71/cbcbbc9d06a74f2fd8ea9109b81b88086f1430 is empty
fatal: loose object 71cbcbbc9d06a74f2fd8ea9109b81b88086f1430 (stored in .git/objects/71/cbcbbc9d06a74f2fd8ea9109b81b88086f1430) is corrupt

1
2
3
4

$ git fsck
error: object file .git/objects/00/837a7e1f8afb8da8609369f7acf95fe9b7fc5b is empty
error: object file .git/objects/00/837a7e1f8afb8da8609369f7acf95fe9b7fc5b is empty
fatal: loose object 00837a7e1f8afb8da8609369f7acf95fe9b7fc5b (stored in .git/objects/00/837a7e1f8afb8da8609369f7acf95fe9b7fc5b) is corrupt

Has the code written in the past few days just vanished like this? We know that when you delete something, you just delete the reference to this thing in the current three-dimensional space, and the entity of this thing still exists in the four-dimensional space-time. Let’s time travel!

January 30, 2015, was the deadline for SIGCOMM 2015 paper submissions, but I went to Huangshan for a trip. IMG_20150130_083809

Preparations Before Departure {/examples/}

It was our first time going to Huangshan. As early as December 15, when our final exam schedule was confirmed, we started preparing.

Time {/examples/}

We set off the day after my final exams ended. After checking some travel guides, we found that Huangshan is suitable for a two-day trip, with one night spent on the mountain. Including the round trip to Hefei, it would take a total of four days.

Transportation {/examples/}

There are two routes from Hefei to Huangshan:

1. Take a train to Huangshan Railway Station (located in Tunxi, Huangshan City), then take a one-hour bus to Tangkou Town at the foot of Huangshan.
2. Take a direct bus from Hefei Long-Distance Bus Station to Tangkou Town at the foot of Huangshan.
   After arriving at Tangkou Town at the foot of Huangshan, to enter the Huangshan Scenic Area, you can only take the Xin Guoxian bus, which costs 19 yuan per person. The bus ride is a 20-minute winding mountain road, about 10 kilometers. The bus goes in two directions, one to Ciguang Pavilion at the front mountain and one to Yungu Temple at the back mountain. Both Ciguang Pavilion and Yungu Temple have cable cars to go up the mountain, or you can hike up (about 5 kilometers, taking 3 hours). Currently, the cable car at the front mountain is closed for maintenance, and hiking up is too tiring, so tourists usually go up from the back mountain, stay overnight on the mountain, and descend from the front mountain.

Since Jingning gets carsick and can’t sit on a bus for long periods, we chose the train-then-bus option. Currently, there are only K-series trains from Hefei to Huangshan, taking 6-7 hours (high-speed trains will be available this year). Although train tickets from Hefei to Huangshan are not in high demand during winter, we still booked our tickets a month in advance.

Buses within Huangshan City and the Huangshan Scenic Area run every half hour, and reservations are neither needed nor possible.

Accommodation {/examples/}

For the two nights at the foot of the mountain, you can choose to stay in Huangshan City or at the foot of the mountain. Accommodation on the mountain is relatively tight and needs to be booked in advance. You can do this on Ctrip.

Gear {/examples/}

• Documents: ID card, student ID
• Electronics: Mobile phone, iPad, power bank, flashlight (bought from Taobao, but didn’t use it)
• Warm clothing: Scarf, gloves, hat, seat cushion (Huangshan is very cold in winter)
• Climbing aids:
• Clothing: Each person brings 2 pairs of socks, underwear, towel for bathing
• Toiletries: Toothbrush, toothpaste, cup, lip balm, hand cream, skincare products…
  

January 28 {/examples/}

&nbsp;

On May 16, 2013, my blog got a top-level domain bojieli.com. On January 6, 2015, I registered and enabled a new domain ring0.me (it’s the number 0, the font looks like the letter O).

Ring0 is the highest privilege level in the CPU architecture, and the code running at the Ring0 level interacts directly with the physical hardware. The concept of privilege level can be traced back to MULTICS in the 1960s. In the x86 architecture, ring0 represents the operating system kernel and kernel drivers, as opposed to user-mode applications that usually run in ring 3. The first time I heard about ring0 was in an article about rootkits when I was in junior high school and was very curious about “hacker” technology. Shamefully, I still can’t write a rootkit.

My blog uses the domain ring0.me to show that my main interest is in building the foundation of computer systems and networks through research and technology.

Registering ring0.me was because I felt that the full-spelled domain name bojieli.com didn’t look geeky enough. I considered multiple unregistered domain names such as rdma, ssh22, http80, tcp80, printk, reisub, etc., but in the end, I still thought ring0 was better.

The original bojieli.com has already HTTP 301 redirected to the corresponding page of ring0.me. Due to the policy restrictions of StartSSL, SSL certificates can only be applied for three days after domain registration.Deployed the SSL certificate on January 17.bojieli.com will continue to serve until May 2016, and will not be renewed afterwards.

The blog title has been changed from “null != undefined” to “Ring0”, and the subtitle has been changed from “Seeking possibility for next-generation network” to “Fundamental research in networked systems”. Hope readers like it ^_^