Design, Analysis, Implementation and Performance of An Archipelagic File System

Abstract:

We have built a distributed file system called Archipelagic File System
(APFS). APFS is distinct from other file systems in its approach to failure
isolation and low-cost consistency maintenance. The building blocks of APFS
are smaller self-contained file servers called islands. The main idea
underlying APFS design is the one-island principle: almost every atomic
operation should require the participation of exactly one island or server.
The one-island principle improves partial reliability and availability
because each island is self-contained and therefore can function
independently of others' failures. The one-island principle allows an
archipelagic file system to scale well with the number of islands and
workloads because cross-island communication and synchronization are
minimized. Reconfiguration (addition or removal of islands, or dynamic load
balancing) is efficient because it only requires a minimal amount of data to
be migrated between islands. Existing file system structures cannot satisfy
the one-island principle. We built a new system structure in which data is
partitioned to servers at directory granularity by hashing the pathnames of
directories.

Experiments on the contents of five existing file systems in use show that
the hashing algorithm in APFS can evenly distribute workload to islands.
Statistics and scalability measurements show that on average 99.8% of
operations satisfy the one-island principle and that performance of these
one-island operations scales efficiently with the number of islands and
workloads. A trace-driven study of online reconfiguration for a web server
running on APFS shows that data migration during reconfiguration is made
transparent to the web server and imposes a performance penalty of only
4.5%.