Holistic Configuration Management at Facebook

Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chander,

Zhe Wen, Aravind Narayanan, Patrick Dowell, and Robert Karl

Facebook Inc.

{tang, thawan, pradvenkat, akshay, wenzhe, aravindn, pdowell, robertkarl}@fb.com

Abstract

Facebook’s web site and mobile apps are very dynamic.

Every day, they undergo thousands of online configuration

changes, and execute trillions of configuration checks to

personalize the product features experienced by hundreds

gives a comprehensive description of the use cases, design,

implementation, and usage statistics of a suite of tools that

manage Facebook’s configuration end-to-end, including the

frontend products, backend systems, and mobile apps.

1. Introduction

The software development and deployment cycle has accel-

erated dramatically [13]. A main driving force comes from

the Internet services, where frequent software upgrades are

not only possible with their service models, but also a neces-

sity for survival in a rapidly changing and competitive envi-

to Facebook’s frontend product code.

Frequent configuration changes executed by a large popu-

lation of engineers, along with unavoidable human mistakes,

lead to configuration errors, which is a major source of site

outages [24]. Preventing configuration errors is only one of

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for profit or commercial advantage and that copies bear this notice and the full citation

on the first page. Copyrights for components of this work owned by others than ACM

must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,

the many challenges. This paper presents Facebook’s holistic

configuration management solution. Facebook uses Chef [7]

to manage OS settings and software deployment [11], which

is not the focus of this paper. Instead, we focus on the home-

grown tools for managing applications’ dynamic runtime

configurations that may be updated live multiple times a

day, without application redeployment or restart. Examples

include gating product rollouts, managing application-level

traffic, and running A/B testing experiments.

bution mechanism, which makes the site as a whole hard to

manage. To curb the configuration sprawl, we use a suite

of tools built on top of a uniform foundation to support the

diverse use cases. Currently, the tools manage hundreds of

thousands of configs (i.e., configuration files) from a cen-

tral location, and distribute them to hundreds of thousands

of servers and more than one billion mobile devices.

Configuration authoring and version control. A large-

scale distributed system often has many flexible knobs that

can be tuned live. The median size of a config at Face-

Defending against configuration errors. We safeguard

configs in multiple ways. First, the configuration compiler

automatically runs validators to verify invariants defined for

configs. Second, a config change is treated the same as a

code change and goes though the same rigorous code re-

view process. Third, a config change that affects the frontend

products automatically goes through continuous integration

tests in a sandbox. Lastly, the automated canary testing tool

rolls out a config change to production in a staged fashion,

Each system has its own config but there are dependencies

among the configs. For example, after the monitoring tool’s

config is updated to enable a new monitoring feature, the

monitoring configs of all other systems might need be up-

dated accordingly. Our framework expresses configuration

dependency as source code dependency, similar to the in-

clude statement in a C++ program. The tool automatically

extracts dependencies from source code without the need to

manually edit a makefile.

Scalable and reliable configuration distribution. Our

to distribute configs to all servers and mobile devices in a

timely and reliable manner, and not to make the availability

of the configuration management tools become a bottleneck

of the applications’ availability.

In this paper, we describe our solutions to these chal-

lenges. We make the following contributions:

2. Use Cases and Solution Overview

We focus on the problem of managing an Internet service’s

dynamic runtime configurations that may be updated live

early and frequently. It forces us to get early feedback and

iterate rapidly. While a new product feature is still under de-

velopment, we commonly release the new code into produc-

tion early but in a disabled mode, and then use a tool called

Gatekeeper to incrementally enable it online. Gatekeeper can

quickly disable the new code if problems arise. It controls

which users will experience the new feature, e.g., Facebook

employees only or 1% of the users of a mobile device model.

The target can be changed live through a config update.

Conducting experiments. Good designs often require A/B

configs are stored in a version control tool, e.g., git [14]. In

the upper-left side of Figure 3, the engineer works in a “de-

velopment server” with a local clone of the git repository.

She edits the source code and invokes the Configerator com-

piler to generate JSON configs. At the top of Figure 3, config

changes can also be initiated by an engineer via a Web UI, or

programmatically by an automation tool invoking the APIs

provided by the “Mutator” component.

and the security team. The scheduler team implements the

scheduler software and provides the shared config code,

including the config schema job.thrift, the reusable mod-

ule create

which ensures that configs provided by other teams do not

accidentally break the scheduler. The cache team gener-

ates the config for a cache job by simply invoking cre-

pendencies through import

example is shown below.

“app.cconf ” instructs an application to listen on a specific

port. “firewall.cconf ” instructs the OS to allow traffic on

PORT in “app port.cinc”

is changed, the Configerator compiler automatically recom-

piles both “app.cconf ” and “firewall.cconf ”, and updates

Figure 2: The Configerator compiler generates a JSON configuration file from the Python and Thrift source code.

their JSON configs in one git commit, which ensures consis-

tency. Dependency can be expressed using any Python lan-

guage construct, not limited to shared constants.

Configerator is designed as a uniform platform to support all

erates the artifacts needed by Configerator.

The Sitevars tool is a shim layer on top of Configerator

to support simple configs used by frontend PHP products. It

provides configurable name-value pairs. The value is a PHP

expression. An engineer uses the Sitevars UI to easily update

a sitevar’s PHP content without writing any Python/Thrift

code. A sitevar can have a checker implemented in PHP to

verify the invariants, similar to the validator in Figure 2.

Because PHP is weakly typed, sitevars are more prone to

configuration errors, e.g., typos. Engineers are encouraged

data type, the UI displays a warning message to the engineer.

Configuration errors are a major source of site outages [24].

We take a holistic approach to prevent configuration errors,

including 1) config validators to ensure that invariants are

not violated, 2) code review for both config programs and

generated JSON configs, 3) manual config testing, 4) auto-

mated integration tests, and 5) automated canary tests. They

complement each other to catch different configuration er-

rors. We follow the flow in Figure 3 to explain them.

mand to temporarily deploy the new config to some pro-

duction servers or testing servers, and verifies that every-

tegration tests of the site under the new config. Sandcastle

posts the testing results to Phabricator for reviewers to ac-

cess. Once the reviewers approve, the engineer pushes the

config change to the remote “Canary Service”.

subset of production machines that serve live traffic. It com-

plements manual testing and automated integration tests.

Manual testing can execute tests that are hard to automate,

but may miss config errors due to oversight or shortcut under

time pressure. Continuous integration tests in a sandbox can

thousands of servers. For each phase, it specifies the testing

target servers, the healthcheck metrics, and the predicates

that decide whether the test passes or fails. For example, the

click-through rate (CTR) collected from the servers using

the new config should not be more than x% lower than the

CTR collected from the servers still using the old config.

servers under test to temporarily deploy the new config (see

ment tools); and 2) data consistency (i.e., an application’s

instances running on different servers should eventually re-

ceive all config updates delivered in the same order, although

there is no guarantee that they all receive a config update

exactly at the same time). In this section, we describe how

Configerator achieves these goals through the push model.

In Figure 3, the git repository serves as the ground truth

for committed configs. The “Git Tailer” continuously ex-

tracts config changes from the git repository, and writes

them to Zeus for distribution. Zeus is a forked version of

leader→observer→proxy, to distribute configs through the

push model. The leader has hundreds of observers as chil-

dren in the tree. A high-fanout tree is feasible because the

data-center network has high bandwidth and only small data

is sent through the tree. Large data is distributed through a

peer-to-peer protocol separately (see Section 3.5). The three-

level tree is simple to manage and sufficient for the current

scale. More levels can be added in the future as needed.

Each Facebook data center consists of multiple clusters.

Each cluster consists of thousands of servers, and has mul-

opt for the push model in our environment.

News Feed ranking. As these large configs may change fre-

quently, it is not scalable to deliver them through Zeus’ dis-

tribution tree, because it would overload the tree’s internal

nodes that have a high fan out. Moreover, it is hard to guar-

antee the quality of service if the distribution paths overlap

between large configs and small (but critical) configs.

Our PackageVessel tool solves the problem by separating

a large config’s metadata from its bulk content. When a large

config changes, its bulk content is uploaded to a storage

system. It then updates the config’s small metadata stored in

Multiple engineers making concurrent config commits into a

shared git repository causes contention and slows down the

commit process. We explain it through an example. When

an engineer tries to push a config diff X to the shared

git repository, git checks whether the local clone of the

repository is up to date. If not, she has to first bring her local

clone up to date, which may take 10s of seconds to finish.

After the update finishes, she tries to push diff X to the shard

according to the first-come-first-served order, and 3) pushing

them to the shared git repository on behalf of the committers,

without requiring the committers to bring their local repos-

itory clones up to date. If there is a true conflict between a

diff being pushed and some previously committed diffs, the

shared git repository rejects the diff, and the error is relayed

back to the committer. Only then, the committer has to up-

date her local repository clone and resolve the conflict.

lem, but does not fundamentally solve the commit-

titioned global name space. Files under different paths (e.g.,

“/feed” and “/tao”) can be served by different git reposito-

ries that can accept commits concurrently. Cross-repository

dependency is supported.

import_python(‘‘feed/A.cinc’’, ‘‘

*

’’)

import_python(‘‘tao/B.cinc’’, ‘‘

*

’’)

...

In the example above, the config imports two other configs.

git repository has its own mutator, landing strip, and tailer.

itory grows large, some of its files can be migrated into a

new repository. It only requires updating the metadata that

Every component in Figure 3 has built-in redundancy across

multiple regions. One region serves as the master. Each

backup region has its own copy of the git repository, and re-

ceives updates from the master region. The git repository in

Configerator addresses the key challenges in configuration

authoring, configuration error prevention, and configuration

distribution. It takes a configuration-as-code approach to

compile and generate configs from high-level source code.

It expresses configuration dependency as source code de-

pendency, and encourages config code modularization and

reuse. It takes a holistic approach to prevent configuration

errors, including config validators, code review, manual con-

fig testing, automated integration tests, and automated ca-

nary tests. It uses a distribution tree to deliver small con-

Facebook releases software early and frequently. It forces

us to get early feedback and iterate rapidly. It makes trou-

bleshooting easier, because the delta between two releases is

small. It minimizes the use of code branches that complicate

maintenance. On the other hand, frequent software releases

increase the risk of software bugs breaking the site. This

section describes how Gatekeeper helps mitigate the risk by

managing code rollouts through online config changes.

While a new product feature is still under development,

Facebook engineers commonly release the new code into

feature to the engineers developing the feature. Then Gate-

keeper can enable the feature for an increasing percentage

of Facebook employees, e.g., 1%→10%→100%. After suc-

cessful internal testing, it can target 5% of the users from a

specific region. Finally, the feature can be launched globally

with an increasing coverage, e.g., 1%→ 10%→100%.

if(gk_check(‘‘ProjectX’’, $user_id)) {

// Show the new feature to the user.

...

} else {

if ($project == ‘‘ProjectX’’) {

// The gating logic for ‘‘ProjectX’’.

if($restraint_1($user_id) AND

$restraint_2($user_id)) {

} else if ($restraint_4($user_id)) {

} else {

disable a product feature. Figure 5 shows the project’s in-

ternal logic, which consists of a series of if-then-else state-

ments. The condition in an if -statement is a conjunction

of predicates called restraints. Examples of restraints in-

clude checking whether the user is a Facebook employee and

checking the type of a mobile device. Once an if -statement

is satisfied, it probabilistically determines whether to pass

or fail the gate, depending on a configurable probability that

controls user sampling, e.g., 1% or 10%.

the restraints in an if -statement can be added or removed;

the probability threshold can be modified; and the parame-

ters specific to a restraint can be updated. For example, the

user IDs in the “ID()” restraint can be added or removed so

that only specific engineers will experience the new product

feature during the early development phase.

straints through configuration. Internally, a restraint is stat-

ically implemented in PHP or C++. Currently, hundreds of

form the gating logic. It strikes a balance among flexibil-

ECHO a different parameter value to ex-

periment with. After the experiment finishes and the best pa-

rameter is found, VOIP

ECHO can be remapped to a con-

stant stored in Configerator. In the long run, all the backend

systems (e.g., Gatekeeper and Configerator) may be replaced

by new systems. It only requires changing the mapping in the

translation layer to smoothly finish the migration. To scale to

more than one billion mobile devices, the translation layer

runs on many servers. The translation mapping is stored in

more valuable than experimental results in a sandbox en-

vironment. For the latter, we report Configerator’s commit-

throughput scalability test in a sandbox, because the data

cannot be obtained from production, and currently commit

throughput is Configerator’s biggest scalability challenge.

We attempt to answer the following questions:

Configerator stores different types of files: 1) the Python and

Thrift source code as shown in Figure 2; 2) the “compiled

configs”, i.e., the JSON files generated by the Configerator

compiler from the source code; 3) the “raw configs”, which

are not shown in Figure 2 for brevity. Configerator allows

engineers to check in raw configs of any format. They are not

produced by the Configerator compiler, but are distributed in

the same way as compiled configs. They are either manually

Figure 7: Number of configs in the repository. The scale on

edited. This validates one important hypothesis of Configer-

manually editing configs. The custom automation tools are

used to generate raw configs because they suit the specific

(often simpler) jobs better, or they predate Configerator.

have significant complexity and are not trivial name-value

pairs. The complexity is one reason why engineers prefer

not editing configs manually. Compiled configs are more

complex than raw configs. The P50’s of raw config size and

44/16/10 times during its lifetime, respectively. Raw configs

get updated 175% more frequently than compiled configs,

because most (89%) of the raw config changes are done by

automation tools. Compiled configs are generated from con-

fig source code, but the former changes 60% more frequently

than the latter does, because the change of one source code

file may generate multiple new compiled configs, similar to

a header file change in C++ causing recompilation of mul-

tiple .cpp files. This indicates that writing code to gener-