“Everything standing up today [at Uber] is in a container.” That’s according to Jane Arc, Senior Engineer, Security R&D at Uber. Arc stopped by the recent Interop Container Summit to discuss “Managing Microservices at Scale” with Joyent’s Bryan Cantrill, and she shared insight into how Uber is leveraging the microservices model.
“For background: we had half a million lines of python driving the backend of Uber and everytime it tipped over we had an outage,” Arc said. “We broke it down into microservices and doing that gave us all this velocity which is great.”
According to Arc, a single function of the Uber app (like pricing) might be broken into 30 or 50 individual services on the backend.
“Let’s talk about surcharges on roads or bridges,” Arc said. “If I want to figure out whether we’re doing that really well, instead of having something that calculates the whole fare for an entire route… I can figure out each individual part of that service. Instead of them being hundreds of thousands of lines of code, maybe some of them are 500. I can set up instrumentation for each of those and peek into them. I can also isolate the failure of those. It’s way better to me.”
The microservices model also allows the Uber team to stand up new services every day – and easily deprecate old ones.
“If you were to take this entire system and describe it today, tomorrow [your description] would be out of date,” Arc said. “The beauty of this system if you were to describe it as this huge net, is you could cut out an entire piece of it and the rest of it would keep working.”
But managing a complex net of microservices comes with challenges. According to Arc, “you have to have people that are watching not just their group of people but they have to socially know who to talk to – they have to know who’s working on what in which office and they have to know where there’s overlap so there isn’t contention.”
You also have to ensure everyone is trained on containers, with a thorough education program that teaches developers “how to develop a certain way.”
“We force [developers] to go through this and learn how to do it,” Arc said. “We bring people in who have never used Docker and have them sit through classes.”
According to Arc, the three crucial components to a container infrastructure are:
- Getting devs to “find the Kool-Aid and drink it.”
- Pushing systems people to “go deeper” by learning to “tune the systems settings on a box” instead of relying on a “web interface and a gui”
- Hiring Site Reliability Engineers (SREs)
“If you’ve got all these things together, you’ll have a pretty successful container infrastructure,” Arc said. “I don’t think you can expect to do containers without these three things.”
Follow the link below for the complete video on “Managing Microservices at Scale” with Jane Arc and Bryan Cantrill: