Bojie Li (李博杰)

Container is an emerging lightweight virtualization technology, and a way to encapsulate software dependencies and simplify deployment. When we started to build Freeshell in 2012, Docker was premature and the dominant container technology was OpenVZ. So we built Freeshell, an elastic and efficient container cloud with OpenVZ. Freeshell attracted 2,000+ users in USTC campus. Our carefully tuned system consolidated 1,000+ active containers into 8 servers.

Freeshell offers multiple pre-created OS images, including Debian, Ubuntu, CentOS, Archlinux and more Linux distributions. To consolidate disk storage of multiple containers, we store the file system of each container as the delta of the base image, and AUFS provides transparency to applications.

Although Internet is designed to be end-to-end, not all IP addresses are reachable from everywhere due to NAT, firewall, organizational policies and other network middleboxes. Additionally, direct routing is possibly not the best path due to ISP QoS policies and limited interconnect bandwidth between some ASes. Furthermore, TCP Cubic congestion control performs notably bad on long fat pipes with occasional packet loss.

Since 2013, we design, deploy and operate a global VPN network to enable USTC students to efficiently access every host across the Internet from any network location. LUG VPN has over one thousand users, serving about one terabyte of network traffic every day via tens of servers in global datacenters.

Different from traditional VPNs where the user connects to a gateway server and the server directly accesses the Internet, LUG VPN forwards traffic among gateway servers and finds an optimal server to access the destination host. Consequently, there are two or three hops from a VPN user to the destination host: from the user to an access gateway server, from the access gateway to an egress gateway (this hop is optional), and from the egress gateway to the destination host. By adding an additional hop, LUG VPN has the freedom to optimize the tunnel between access gateway and egress gateway to bypass firewalls and improve QoS.

To use a best egress gateway from an access gateway to a destination, we use GeoIP based strategy. First, we run a recursive DNS server on each access gateway to resolve users’ DNS requests, so that if a multi-homed website uses GeoIP based authoritative DNS server, it can return IP addresses close to the access gateway. Second, a GeoIP based routing table is configured on each access gateway to route IP packets (including DNS packets) directly to the destination or via a tunnel to an egress gateway. To make the routing decision for every pair of geographic regions, we use periodic probing to find the latency and bandwidth for each route, i.e. pass through an egress gateway or direct route. In addition, we have some rules to exclude certain routes in order to bypass content-based firewalls as well as comply with copyright restrictions. Among all candidate routes, we first find the best route with lowest latency and highest bandwidth, then find other candidate routes whose latency and bandwidth are close to the best route. To load balance traffic among egress gateways, we use weighted round robin to route connections via all candidate routes.

To ensure efficient forwarding between access and egress gateways, we create multiple tunnels between each pair of gateways using different tunneling technologies. Some are standard tunneling protocols (e.g. GRE). Others disguise the tunnel as another protocol (e.g. HTTPS) to bypass protocol filters and improve QoS priority in ISP network. If one tunneling technology is blocked, the system automatically switches to another. To reduce TCP handshake latency, we terminate TCP connections at both access and egress gateways, so that each TCP connection is broken into three relayed TCP connections. For the tunnel connection between access and egress gateways, we deploy multiple WAN optimization techniques, including loss-agnostic congestion control, FEC (forward error correction) on lossy links and compression.

The user can access the VPN by connecting to any of the public gateway servers via multiple protocols, including IPSec, OpenVPN, PPTP, L2TP, ShadowSocks, Socks5, IP over DNS and HTTP(S) proxy. In order to enable the user to access a nearby gateway server, the user specifies a domain name, and our authoritative DNS server resolves it to the IP address of a closest gateway server based on the GeoIP location of the user’s IP. Using DNS also adds another layer of indirection for the system to automatically remove access gateways that fail or overload.

To authenticate users, we deploy a LDAP system that integrates with the access gateways and develop a Web-based application system. To recover from failures automatically, the servers monitor each other via periodic tunnel ping probing and exchange results. If more than half of peer servers find a server fails, the peer servers issue a remote reboot command via management API (for cloud servers) or IPMI (for bare-metal servers). To enable users to only route a range of destination hosts via VPN, we offer multiple configuration options in the software.

LUG VPN offers good reachability and performance in multiple usage scenarios. First, users with limited Internet access can access all sites. Second, users outside the campus can access the campus Intranet. Third, users outside the campus can access scholarly articles that is only available for campus IP addresses. Fourth, with WAN optimization, users experience much better throughput and latency, especially for international traffic. Additionally, users use LUG VPN to interconnect devices behind NAT and firewall.