This project is named based on the desire to shepherd the resources on the Internet in an efficient manner. A central requirement is for applications to have network awareness; some form of knowledge about the state of relevant portions of the network.
The simple task of downloading a file can serve as a good example. Assuming three identical copies of a file exist at different servers, no matter which server is chosen the downloaded file will be identical. The transfer time will however depend on the chosen server and several factors related to it, such as e.g., the connectivity to it from the client downloading the file. A fast download will not only please the user, but likely also make more efficient use of the network capacity on the Internet, because it will probably have to travel a shorter distance to arrive at the users machine; a user in Norway is likely to retrieve files from a server in Oslo at a faster rate than if the same file was downloaded from a server in e.g., Tokyo.
Today, the task of choosing an appropriate server is often left to the user. Many sites present a list of mirror sites for large files, from which the user has to choose one which is hopefully reasonably close, fast and available. It should however be possible for this entire process to be done automatically, without the user having to perform a task computers can do both faster and more efficiently.
Achieving this requires the choice of a metric suited for the task (for example transfer rate), and some way of obtaining it for the relevant machines, with a sufficient degree of accuracy and timeliness. It also requires knowledge of the locations at which the desired content can be found.
The problem essentially boils down to how to construct efficient content distribution networks? Building and maintaining a responsive system can quickly become expensive.
The Shepherd Project has its outspring in research on scalable Video-on-Demand streaming for legal content distribution done at the University of Oslo. This research resulted in the SPP Architecture, a scalable system for interactive video streaming.
The Shepherd project continues this research at Inferno Nettverk A/S, which has many years of experience with the development, maintenance, and support of software. Inferno Nettverk is building a set of tools for creating scalable content distribution networks, and the p2pd implementation of the SPP architecture is the first application released by the Shepherd project, and is available as free software.