The replication factor in HDFS is the number of copies kept of every data block. The default is 3 — every block exists on three different DataNodes. Replication is what makes HDFS reliable in the face of node failures: if one machine dies, the other copies are still available, and HDFS notices and re-creates a fresh copy elsewhere to restore the replication factor.
With replication factor 3 on a cluster:
- A file is split into blocks (default 128 MB each).
- Each block is copied onto 3 different DataNodes.
- The NameNode tracks which nodes hold which replicas.
- Reads can come from any replica; writes propagate to all replicas before completing.
- If a DataNode is lost, the NameNode picks another DataNode and copies one of the remaining replicas there.
Higher replication factor means more reliability and more storage overhead. With factor 3, the cluster’s effective storage capacity is one-third of its raw disk total. Factor 2 saves space but tolerates fewer simultaneous failures. Factor 1 (no replication) is fast for ephemeral data but loses anything stored on a failing node.
Replication is set globally for the cluster, but can be overridden per file:
hdfs dfs -setrep -w 5 /important/fileThis sets the replication factor for /important/file to 5 — five copies across five different nodes. The -w flag waits for replication to complete before returning, useful for confirming that the safety margin is actually in place.
Replication is HDFS’s answer to a fundamental fact about clusters at scale: some node is always failing. With a thousand nodes, the probability that all of them are working at any given moment is essentially zero. Replication absorbs the failure rate, and automatic re-replication restores safety margins without human intervention. This is what fault tolerance and automatic recovery mean in practice.