Facebook’s mobile apps are built for specific operating systems. No longer solely dependent on the Web, Facebook has finer control over mobile app development specific to your iPhone or Android. But by switching to native development two years ago, the company could no longer run A/B tests. So developers built Airlock.
Facebook just released new details about Airlock, the testing framework that allows its developers to compare data from various app implementations and decide which version of the application works best.
The narrative describes challenges many developers face when it comes to fixing the inner workings of applications to best perform for customers. But by going under the hood with its own A/B testing platform, Facebook shows its users how, exactly, it builds a UI people Like.
“We’re one of the biggest app developers and encountering new challenges,” a spokesperson for Facebook told us. “So we’re sharing our experiences so that when other companies hit these challenges they can learn from our process and understand how we did it.”
Even Facebook Makes Mistakes
Things are always changing at Facebook HQ. Every four weeks, iOS and Android teams at Facebook coordinate and produce a new executable file with updated features and bug fixes. And with each update, the teams track how new features perform, fix performance and reliability of the app, and make adjustments the user interface that change how and where people spend their time on Facebook.
The company exposes users to two different versions of the app—an A version and B version—and uses Airlock to compare data collected from both to decide which version to ship to all users.
In its post, Facebook noted that, for instance, one of the first experiments the company ran was changing a chat icon to the word “chat.” Using the A/B system, some people saw the word chat while others still saw the icon, and Facebook determined which version of chat users were most engaged with.
Implementing the word “chat” seemed like a massive success as more users were engaged with that version of the app. Unfortunately what developers found was a pile of bugs and a component incorrectly caching data.
As the experiment wore on, developers discovered an incongruence between the information the device was told and a delay during which the UI was incorrect. The company eventually added a “two-way handshake,” meaning the device requests data, and the server logs the response that sends it out.
Growing Pains
At first only able to support two experiments, the Airlock project evolved to support 10 or 15 variations of a single experiment that affects millions of people.
There are a variety of Facebook designs and interfaces lying on the developers’ cutting room floor that were never implemented to the masses, as A/B testing proved users didn’t like it. Though as the testing platform continues to scale, developers have to make sure different tests don’t pollute one another, and keep some Airlock experiments exclusive in order to deter bugs or incorrect logging in the control group.
Airlock started as a project with the goal of improving the navigation within its applications, and eventually, after constant iterations, we have the left-hand navigation drawer that enables us to use Facebook with one hand.
Although Facebook released details of how and why they use Airlock, it is not available to the open source community just yet. But a Facebook spokesperson told us the company “is always evaluating if it will provide value to the open source community.”
