Scalable peer-to-peer substrates: A new foundation for distributedapplications

Peer-to-peer (p2p), initially conceived for the purpose of sharing music in the Internet, promises to be a more general paradigm for organizing large-scale distributed applications. We define p2p systems broadly as self-organizing, decentralized, distributed systems where most or all communication is symmetric. The self-organization, decentralization, diversity and redundancy of resources inherent in the approach lend themselves to a large domain of applications beyond file sharing, anonymity and anti-censorship. At the same time, the decentralization and diversity also pose difficult problems, particularly in resource management and security. Recent work on p2p overlay networks like CAN, Chord, Pastry and Tapestry has made significant strides towards providing a general substrate that simplifies the construction of a wide range of p2p applications. These overlay networks effectively shield applications from the complexities of organizing and maintaining a secure overlay network, tolerating node failure, and from distributing and locating resources. In this talk, I'll present an overview of Pastry, a p2p overlay network that provides self-organization, fault-tolerance, efficient resource location and distribution, and incorporates interesting heuristics that exploit proximity in the underlying Internet. I will also sketch two applications built upon Pastry to date: PAST, an archival, cooperative file storage and distribution facility, and SCRIBE, a highly scalable event notification system. I'll conclude with an outlook on key research problems and future directions.