The Whole Story of LUG Server Being Hacked
In mid-May, LUG held a white hat competition, conducting white hat vulnerability testing on campus websites, and awarded prizes based on the vulnerabilities found. The event went relatively smoothly, and hundreds of vulnerabilities in the campus network system were found within a few days. However, for some unknown reason, it attracted hackers.
In the early morning of May 31, the alarms from the monitoring system broke the tranquility of the late night. Dozens of LUG servers unexpectedly failed one after another, and the shocking event of LUG servers being hacked began. From the email sent by server administrator gyf to the Network Information Center for the second time on June 1, one can vaguely feel the tense atmosphere of that year.
1 | 按照事件第一次发生时间排序： |
It was later found out that the server was remotely logged in using my account. In mid-May, the hacker invaded my laptop with a virus spread through a USB drive, implanted a keyboard logger, and remotely controlled my browser to visit some web pages related to network intrusion through unknown means. Since my personal account has no access record and the Bitcoin wallet was not stolen, it is very likely that this was an Advanced Persistent Threat (APT) targeting LUG rather than me personally. In the following half month, the hacker did not make any rash moves, presumably collecting information about the LUG server through various channels. On the night of May 30, the hacker logged into a large number of servers with the server password stolen by the keyboard logger and inserted malicious kernel modules. The hacker also invaded the LDAP database, tampered with the password of an old member who had already left the school, and logged into the most strictly defended mirrors server. The hacker also stole the private key of a VPN user, accessed the server’s internal network, and further invaded servers that were not allowed to be accessed from outside the school.
What this malicious kernel module does seems simple, it randomly modifies a byte on the hard disk every time a file operation is performed. This seemingly prank-like kernel module makes the server run normally when it is just invaded, but when key data is destroyed, system anomalies are discovered, and by this time a large amount of user data and system files have been destroyed. When the administrator tries to scan and repair these damaged files, a large number of file operations are generated, causing more files to be damaged, and they can never be repaired. Even when we NFS mount the backup server to copy backup data, the copied backup is also wrong, which is puzzling to us (fortunately, the backup server is NFS read-only mounted, otherwise the backup itself may also be damaged).
The interruption of the open-source software mirror (mirrors) service with over ten million daily visits, the interruption of the VPN service relied on by thousands of users in the school, the damage to the files in the freeshell virtual machine, the inability to access the blog, and even the LUG homepage cannot be opened, inquiry emails flew in like snowflakes, but the email server also hung up. This hacking incident even alarmed several Linux distributions and upstreams of open-source software. They all expressed that it was unheard of for open-source software mirrors to be hacked. The Freeshell service was terminated due to lack of backup, and the VPN service was no longer publicly run because it acted as an accomplice. This incident exposed many problems with the basic infrastructure of the LUG server, such as the public VPN service and the server using the same internal network, no two-step verification for password login, no alarm and defense for the dangerous operation of inserting kernel modules, and the account of the departed administrator was not disabled. Of course, the fundamental reason was that my laptop was hacked. Afterwards, LUG and teacher James treated me leniently and did not pursue my responsibility. I still feel very guilty to this day.