MercadoLibre: Tapping into a need and running like hell

If the need is strong enough, sometimes getting agile cloud infrastructure adopted can be almost as easy as ‘build it and they will come.’ In many ways, that is the experience that Alejandro Comisario, Maximiliano Venesio, and Leandro Reox had at their company, MercadoLibre, the biggest e-commerce company in Latin America and the 8th largest worldwide.

In 2011, with the company’s development organization starting down the path of decomposing their then-monolithic platform into one composed of loosely coupled services connected via APIs, the infrastructure team saw a dramatic uptick in the number of infrastructure requests their small team was required to fulfill.

“The shift happened so quickly,” said Comisario, technical leader for cloud services at MercadoLibre. “We literally realized overnight that we wouldn’t be able to keep up without help.”

Comisario, Venesio, and Reox, MercadoLibre’s entire infrastructure team at the time, began looking for technology that would allow them to eliminate the manual steps associated with delivering infrastructure to their developers.

The team set some lofty goals for their effort, which they used to articulate what they were doing to the broader company: They wanted to be able to reduce the time required to deliver a developer a production-ready virtual machine from 2 hours to 10 seconds, and they wanted to be able to do so without human intervention.

When they found OpenStack, they quickly realized that it could be exactly what they were looking for. MercadoLibre’s fast-moving culture allowed the trio to move quickly to put an OpenStack environment in place, in spite of the project’s relative immaturity.

“It turned out that the OpenStack way of doing things—researching, diving into the code, and testing how things work and scale—really fit with the way we do things at MercadoLibre,” said Leandro. “We were able to jump right in and define a set of tests for our OpenStack environment, and start testing.”

Their initial testing, on OpenStack’s second release, identified several issues that would prevent them from going to production, however the transition from the Bexar release to the Cactus release happened just at the right time. Further testing on Cactus gave them the confidence they needed to put their cloud into production.

Once in production, and as their developers realized they could provision infrastructure as fast as they could consume it, adoption took off.

“The whole company was craving something like this, and what it makes possible for us,” said Venesio, senior infrastructure engineer at MercadoLibre.

Still, the team was careful to manage developers’ expectations. They needed to make sure that their developers knew that existing apps wouldn’t be able to run directly on the new private cloud without modification.

“We had to make sure our developers were prepared to write stateless applications for the cloud,” said Comisario. “It was a huge cultural mind shift for them. In some cases we had to hurt developers’ feelings by insisting that not even one instance would persist their data. So they needed to adjust their thinking.”

The team was vigilant in educating their development team and challenging them to adopt the latest best practices for building cloud-ready applications. They sent emails, held brownbag lunches and formal training sessions, and ensured that the cloud environment was properly documented. As a result of their efforts, MercadoLibre’s developers are now as comfortable developing for the private cloud as they were for the company’s traditional virtualized environment.

The automation they were able to achieve with their private cloud has paid off in allowing MercadoLibre to dramatically scale its infrastructure footprint. What started as a highly committed infrastructure team of three people supporting 250 developers, 100 servers, and 1000 VMs has grown into a team of 10 people supporting over 500 developers, 2000 servers, and 12,000 VMs.
http://superuser.openstack.org/artic...r-organization