How is the USTC Open Source Software Mirror Made?

Update (2014-09-29): Due to some inappropriate content in the mirrors configuration file, the configuration file is no longer public, and some links in this article have become dead links, I’m very sorry.

Due to the disk failure of the USTC open source software mirror (mirrors.ustc.edu.cn) disk failure, stephen, tux and I (boj) are not at school, and the mirrors have not fully recovered since the failure in July, it’s time to start over. This time the mirrors rebuild will be completed entirely by students on campus, which is also an opportunity to practice technology. Here, I will briefly explain what parts the open source software mirror includes and how to build it. Since sourceforge is still waiting for us to synchronize, we hope to restore basic services within three days and rebuild the entire system including synchronization within a week.

WTF?

The so-called open source software mirror is to synchronize some GNU/Linux distributions and well-known open source software repositories from the official site. Users can use the software repository mirror nearby by modifying the configuration file to speed up the download and reduce the load on the official site.

What does it take to make an open source software mirror?

  • Stable bandwidth, generally more than 100Mbps
  • Large hard disk, generally several TB
  • Stable server, online time above 99%
    The open source software mirror mainly consists of two modules: providing download services and regularly synchronizing from the official source. In fact, the USTC open source software mirror currently provides HTTP, rsync, FTP services, and can synchronize from upstream sources via rsync, ftpsync, ssh-trigger, ftp-push, git, etc. A running open source software mirror also needs service status monitoring.

The following will introduce how the USTC open source software mirror (hereinafter referred to as mirrors) is built from several aspects:

  1. Network configuration
  2. Storage configuration
  3. HTTP, rsync, FTP services
  4. Virtual machine
  5. Regular synchronization from upstream sources
  6. Server monitoring

Network Configuration

mirrors currently has 4 ISPs. They are:

  • 202.141.160.110   China Telecom
  • 202.141.176.110   China Mobile
  • 202.38.95.110  CERNET (Education Network)
  • 2001:da8:d800:95::110  CERNET2 (Education Network IPv6)
    mirrors has two gigabit network cards, eth0 is connected to the external network, and eth1 is directly connected to the disk array with a network cable. How can one network cable have 4 ISPs? The secret lies in the VLAN on the network center switch. The switch runs the 802.11Q protocol. The host needs to specify the VLAN ID on the network card and tag the data packets with the corresponding tag to access the corresponding line. The VLAN IDs of China Telecom, China Mobile, CERNET/CERNET2 are 10, 400, and 95 respectively, which means you need to specify vlan10, vlan400, and vlan95 in /etc/network/interfaces and bind the above IPs respectively.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    iface vlan95 inet static
        vlan-raw-device eth0
        address 202.38.95.110
        netmask 255.255.255.128

    iface vlan10 inet static
        vlan-raw-device eth0
        address 202.141.160.110
        netmask 255.255.255.128

    iface vlan400 inet static
        vlan-raw-device eth0
        address 202.141.176.110
        netmask 255.255.255.128
    Next, you need to configure the routing rules (written in /etc/rc.local). The global default is the education network exit. Requests from the education network should be replied from the education network, requests from Telecom should be replied from Telecom, and requests from Mobile should be replied from Mobile. For the principle of these configurations, see The Weird NAT of Multihomed Host.
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    ip route add default via 202.38.95.126

    ip route add default via 202.38.95.126 table 1000
    ip route add default via 202.141.160.126 table 1001
    ip route add default via 202.141.176.126 table 1002
    ip -6 route add default via 2001:da8:d800:95::1 table 1000

    ip rule add from 202.38.95.110 table 1000
    ip rule add from 202.141.160.110 table 1001
    ip rule add from 202.141.176.110 table 1002
    ip -6 rule add from 2001:da8:d800:95::110 table 1000
    We found in actual operation that the speed of accessing foreign networks from the mobile exit is relatively fast. There are two solutions:
  1. Bind a specific IP rather than 0.0.0.0 in the synchronization script, rsync, lftp, wget, curl, etc. all have parameters to specify bind-address.
  2. Use the ISP location IP table of http://gitlab.lug.ustc.edu.cn/boj/smartproxy (registration required), let the education network go through the education network, Telecom go through Telecom, and the default go through the mobile exit.

Storage Configuration

The storage resources of mirrors include:

  • 10 SATA hard drives, using hardware RAID1, forming sda, sdb, sdc, sdd, sde, with spaces of 1T or 2T respectively. The one that is currently broken is sda. LVM volume groups have been established on these disks. Two of the 1T disks are currently used for the root file system and logs, and three 2T disks are used for data.
  • 26TB disk array, directly connected via iscsi gigabit network. Currently, a 20TB XFS partition (the rest is free) has been divided, 13T has been used, of which 5T of data can be deleted. It needs to be expanded to at least 22TB to accommodate the estimated 10T sourceforge mirror.
  • 256GB SSD, planned to be used for cache optimization, not yet used.
    When reconfiguring the hard disk, pay attention to:

  1. When plugging and unplugging hard drives, do not change the hard drive slot, otherwise the hard RAID will be destroyed, which may lead to a large amount of data loss.

  2. Do not disassemble LVM volume groups at will. Familiarize yourself with commands such as vgdisplay, pvdisplay, etc., and operate after checking the LVM information clearly.

  3. After the configuration is complete, be sure to write a document stating the name, UUID, LVM volume group of each block device; the name, mount point, and purpose of each partition or LVM volume. (A lesson learned from blood and tears)
    iSCSI configuration method (from jameszhang):

  4. Set the eth1 interface to 192.168.10.2/24

  5. sudo apt-get install open-iscsi

  6. Edit /etc/iscsi/iscsid.conf, change to node.startup = automatic

  7. sudo iscsiadm -m discovery -t st -p 192.168.10.1

  8. sudo iscsiadm -m node –login You can see the disk
    After the configuration is complete, use udev rules to fix the block device name (sda…). Otherwise, the hard disk name changes after rebooting, which looks very troublesome. fstab and various scripts should use UUID as much as possible, just in case.

HTTP, rsync, FTP services

Please refer to the configuration of the old mirrors: (registration and login required)

http://gitlab.lug.ustc.edu.cn/mirrors/mirrors-system-conf/tree/master

HTTP - Nginx

Mirrors use the high-performance Nginx web server to provide HTTP services. The Nginx configuration file should be under version control and establish a separate git repository (currently mixed with other configurations).

nginx.conf is the entry of the nginx configuration file, it includes other configuration files, including the custom log format in conf.d, and the “virtual host” in sites-enabled.

Each file in sites-enabled is a relative path link to sites-available. Among them, mirrors.ustc.edu.cn-{localhost,vlan10,vlan400,vlan95} are the configuration files of the 4 lines of the mirrors main station, they include the common main station configuration file mirrors.ustc.edu.cn-common. Separate configuration files for the 4 lines can conveniently customize different access control strategies for different lines.

The various sub-stations of mirrors, including pypi, archlinuxarm, mirrors lab, are separate configuration files. newindex and testindex are our future homepages for testing, among which newindex has an example of judging whether it is a browser based on User Agent.

Note that the listen directive in the server block should list all listening IPs (three IPv4 and one IPv6) as much as possible, do not listen to 0.0.0.0, because mirrors may add IP for a specific purpose in the future, our “strategic IP reserve” has 202.38.95.106 and 202.141.176.111.

One problem with Nginx is that when disk I/O becomes a bottleneck, Nginx’s response speed will significantly decrease. Therefore, the I/O load of mirrors should be closely monitored.

Pay attention to configure LogRotate, regularly compress log files, and discard old log files. http://gitlab.lug.ustc.edu.cn/mirrors/mirrors-system-conf/blob/master/logrotate.d/nginx

According to the current nginx configuration, when adding a source, you just need to create a symbolic link to /var/www.

rsync - rsyncd

The rsync protocol is a good tool for synchronizing a large number of files, and rsync is preferred for synchronization from upstream sources. For the convenience of other open source software mirrors in China, in principle, all mirrors provide rsync services.

The rsyncd configuration file of the old mirrors:

http://gitlab.lug.ustc.edu.cn/mirrors/mirrors-system-conf/blob/master/rsyncd.conf

You need to write /etc/rsyncd.motd as the “Message Of ToDay”, which is a large amount of text displayed by rsync.

rsyncd will fork a process for each connection, and will read the latest configuration file when the connection is established, so there is no need to restart the service when modifying the configuration file.

According to the current rsync configuration, when adding a source, you need to modify rsyncd.conf to add the corresponding configuration.

FTP - vsftpd

mirrors uses vsftpd as the FTP server: http://gitlab.lug.ustc.edu.cn/mirrors/mirrors-system-conf/blob/master/vsftpd.conf

Since each FTP connection requires vsftpd to fork a new process, which consumes a lot of server resources, it is recommended to use HTTP access. Based on performance considerations, we previously only enabled FTP access for some necessary sources. However, many users need to download files in bulk and do not know how to use tools such as lftp (http mirror), rsync, etc., so it is necessary to provide FTP access. You can open it first, and then observe its access volume and impact on performance.

According to the current FTP configuration, when adding a source, you need to modify fstab to bind mount the data directory to /srv/ftp/project-name, and execute mount -a to make it effective. The need for bind mount instead of symbolic links is because the vsftpd worker will chroot, and the accessed directory cannot exceed the root directory (/srv/ftp).

Virtual Machine

We use LXC Linux Container to isolate services. For example, regular synchronization from upstream sources, generating status pages, etc., have nothing to do with core services such as HTTP, rsync, FTP, etc. If you put them into a virtual machine, you can avoid scripts with bugs consuming too many resources and affecting core services, or operational errors causing service interruptions or data damage.

The mirrors should include the following virtual machines (it is recommended to use Debian wheezy stable):

  • ftp-push, writable FTP synchronization method for upstream sources.
  • rsync-push, writable rsync synchronization method for upstream sources.
  • lab, a relatively open experimental virtual machine, newcomers to mirrors can try their hands here.
  • web, generate status pages, maintain web pages (in the future, web pages may have more regularly generated content besides status pages)
  • sync, rsync, ftpsync, git synchronization scripts run here.
  • npm, Nodejs source (requires couchdb database) and synchronization script.
  • rubygems, rubygems synchronization script.
  • pypi, pypi synchronization script.
  • sf, a virtual machine that will soon provide for SourceForge Mirror.
    For LXC operation methods and configuration files, you can refer to previous scripts:

http://gitlab.lug.ustc.edu.cn/mirrors/mirrors-main/tree/master

Network Isolation

It should be noted that the virtual machines in the above scripts do not isolate the network namespace. We hope that LXC uses an isolated network namespace, uses SNAT to initiate outbound connections, uses DNAT for inbound port mapping, and does not allow network access in LXC to possibly conflict with the core services of the main station.

Recommended network configuration for each virtual machine:

  • ftp-push: DNAT inbound only
  • rsync-push: DNAT inbound only
  • lab: DNAT inbound (if Xen virtualization is no longer used, 202.38.95.106 can also be bound), SNAT outbound
  • web: no network access required
  • sync: SNAT outbound only
  • npm: nginx reverse proxy, SNAT outbound
  • rubygems: SNAT outbound only
  • pypi: SNAT outbound only
  • sf: bind fixed IP 202.141.176.111

Regular Synchronization from Upstream Sources

Due to historical reasons, mirrors currently have two sets of synchronization scripts:

To read this set of scripts, crontab is the entry point. From here you can see the regular calls to ftpsync and ustcsync. (Small assignment: What is the difference between ftpsync and ustcsync?) As you can see, there are synchronization cycles of 2 hours, 4 hours, 6 hours, and 24 hours. Pay attention to the minutes in the crontab, these numbers are random, to spread the pressure of mirror synchronization on the disk and network as much as possible.

It is recommended to create a mirror user in the sync virtual machine and use this crontab for synchronization.

Some things in the crontab and bin directory should be moved to the web virtual machine:

  • The genindex.py that generates the homepage
  • The script that generates the awstats static page
  • The script that generates the synchronization status (status page)
    There are also some small tools for checking the server status in the bin directory.

Server Monitoring

A stable service requires the maintainer to know its status. This status is divided into two aspects, one is the synchronization status of various mirrors, and the other is the running status of the server itself.

Status Page

The mirror synchronization status is the well-known status page. The mirrors’ status page is queried every 5 minutes to obtain synchronization status, mirror size, last synchronization time, etc. The current script is written by Stephen:

http://gitlab.lug.ustc.edu.cn/mirrors/mirrors-scripts/blob/master/bin/get-mirror-status.py

As everyone has noticed, we need more information about the mirror, such as

  • Upstream source type and URL
  • The amount of change, time used, and average speed of the last synchronization
  • Estimated next synchronization time
    This information is best output in a structured format (such as JSON), and the front-end page will parse it, which is convenient for other programs to handle.

We also need to count the quality of each mirror’s upstream source, so it is necessary to store statistical information in the database after synchronization, otherwise it is too troublesome to flip through the log, and the expired log will be deleted.

Collectd

Collectd is a server status monitoring software that can monitor multiple indicators such as CPU, network, disk, memory, etc., store them in the rrd database, and browse them on the web page through collectd-web. Please Google its structure yourself. Since mirrors are now down, I’m sorry I can’t share screenshots of collectd with you.

Below is the collectd configuration file on mirrors:

http://gitlab.lug.ustc.edu.cn/mirrors/mirrors-system-conf/tree/master/collectd

Be sure to restore the collectd data in the original log partition, it is best to splice it with the new one, this is valuable data for studying the operation of mirrors over the past year.

External Monitoring

Self-monitoring is not very reliable, so mirrors also use ganglia and self-written SMS alerts.

Ganglia runs on the lug.ustc.edu.cn server, monitoring the operating status of almost all USTC LUG servers. http://status.lug.ustc.edu.cn/

When a machine goes down, the first thing I look at is status.lug.ustc.edu.cn, to see if there are any abnormalities in CPU, memory, disk, and network. It’s easy to pinpoint the time of the incident, guess the general cause, and then check syslog, dmesg or nginx’s access log, error log, which is more targeted.

For the basic availability of HTTP services, we also have a simple monitoring script that checks every minute and sends an alert SMS when a problem occurs.

  • Server Status List
  • Fault Log
    Of course, the current alarm mechanism of mirrors is still lacking. For example, when the hard disk is full, OOM Kill, nf_conntrack table is full, our operation and maintenance personnel cannot know at the first time. We hope to try to use alarm software such as nagios to make the operation and maintenance of mirrors more automated.

Conclusion

This article hopes to let everyone understand that setting up an open source software mirror is not a very complicated thing in concept, mainly HTTP, rsync, FTP services and automatic synchronization from upstream sources. But to maintain a mirror with nearly a hundred sources, data volume exceeding 10TB, nearly tens of millions of HTTP accesses per day, and daily traffic exceeding 4TB, it still requires some more complex architecture and tools. At present, our monitoring and log analysis of mirrors is very insufficient, and the whole system seems to be running in a black box; the recent stability of mirrors is also not satisfactory.

Now the mirrors hard disk is broken, let’s go to the machine room to check. If the problem with the RAID card or wiring cannot be found quickly, just treat those two hard disks as non-existent, and carve out a part from the current 1T log partition for the root partition of the LXC virtual machine (logs can be transferred to the disk array). In any case, service must be restored as soon as possible.

I am no longer able to decide on the architecture of the rebuilt mirrors, and I don’t have time to fuss about it. I hope this article and the code on GitLab can provide some help to everyone.

Comments

▲