Web giants are in a kind of Moore’s Law race to build ever larger distributed computer networks. They are well along in writing new chapters for the computer science history books, but it’s unclear where the trend leads.
Take, for example, Facebook. Over the past decade it has created a global network of 15 large data centers and hundreds of edge-networking sites. The network runs thousands of programs serving billions of users and gets code updates about every two hours.
“The distributed systems you are building are something billions of people will be impacted by daily — that is both cool and scary,” said Jay Parikh, the head of engineering and infrastructure at Facebook, at the company’s first conference on large distributed software systems at its headquarters here.
“Everything we deal with is a distributed-systems problem that has never been done before in a scaled environment…[including] putting our own cables under the ocean — things the industry has not had to deal with before,” Parikh told about 200 coders from Facebook and their invited guests.
Challenges in the computer and network hardware and the databases and other software they run “play off each other…[spawning issues in] efficiency, culture, budgeting…[because] everything is connected to everything else,” he added.
The event included talks from programmers at Amazon Web Services, Google, Lyft and Shopify, among others. They shared some of their latest techniques for issues ranging from managing globally distributed databases to debugging systems in ways that speed recovery from system outages.
For its part, Facebook discussed work on two distributed software systems it released as open source — a framework for quickly pushing configuration changes to millions of servers and code for handling out-of-order memory issues in an operating system’s user space.
They are part of a broad cloud-computing software platform Facebook has created to serve its four widely used applications -- News Feed, Instagram, Messenger, WhatsApp -- and a much smaller group of Oculus software users.