The WOD's network design

Each system on the network is a node and each node is connected to zero or more peers.

Nodes

When a node is connected to less peers than its threshold, it broadcasts a packet onto a multicast address with the port on which it would like to receive data. It also listens on the multicast address and picks up advertisements for other nodes until it has reached its threshold. Once the threshold is reached the node stops advertising itself and stops listening on the multicast IP. If the number of connected peers ever drops below the threshold, the node will begin to advertise itself and listen again for other peers.

Peers

Once a packet is received from a prospective peer, the node examines the requested port and then connects to the peer on that port. Packets sent by any peer are gathered together and sent into a central pipe.

The central pipe

The central pipe of packets coming in from the network is watched by a series of functions that have registered to be notified of incoming packets. Each watcher may operate on a packet before it is passed on to the next watcher.

By default, a watcher is set to pick up request packets and handle them. If the cache contains the fingerprint requested, the data for the fingerprint is packaged up and the node connects straight back to the requestor to deliver the data. If the cache does not contain the fingerprint, the request is rebroadcast to a single peer. In this way requests move across the graph of nodes and are replied back to their sources once they can be fulfilled.

This means that the requesting of a fingerprint on the network acts a little like a future (we should probably have explicit support for that).

Another default watcher receives data packets and processes them.

Data Packet Processing

Each data packet is picked off the network and examined; if the TTL (a stupid name but I'll explain it in a second) is 1 then the data is stored on the node and not re-transmitted. If the TTL is greater than 1 then the node stores the data and re-transmits the packet after decrementing its TTL. In this way data blocks get stored on several nodes in the network to provide redundancy. Each node will be stored by a number of nodes equal to the TTL set by the original emitter. So if a packet is emitted with a TTL of 5, it is guaranteed to be stored on 5 nodes other than the original before the packet is removed from the network and no longer broadcast.