Abstract— Distributed hash table (DHT)-based overlay networks, represented by Pastry, CAN, and Chord, offer an administration-free and fault-tolerant application-level overlay network. While elegant from a theoretical perspective, these systems have some disadvantages. First, they rely on applicationlevel routing, which may be inefficient with respect to network delays and bandwidth consumption. Second, they typically construct a homogeneously structured overlay even though nodes in these networks usually have varying physical connectivity and packet-forwarding capacities. In this paper, we propose two approaches for constructing an auxiliary expressway network to take advantage of the different connectivity, forwarding capacities, and availabilities of the nodes. As a result, we are able to reconcile the conflict of presenting the applications with a homogeneous structured overlay to simplify management, while at the same time taking advantage of the inherent heterogeneity of the ...