In this blog post we will go back 10 years to understand how OpenStack started at CERN. We will explore the motivation, the first installation, the prototypes, the deployment challenges and the initial production architecture.
10 years ago CERN IT was working to bring virtualization into the CERN data centre. At that time most applications were still deployed directly in the physical nodes. With the start of the LHC (Large Hadron Collider) it was expected that the resource requests from the users and experiments would increase significantly. Definitely, we needed to change how the resources were allocated, managed and provisioned.
Server virtualization was a hot topic during that time. The advantages of a virtualized data center were clear and very attractive for our use cases: server consolidation, improved efficiency, increased uptime for applications (I still remember the “wow” moment when I live migrated my first instance!), disaster recovery, image based backups, … just to mention a few.
CERN IT was exploring virtualization through two main projects, CVI and LxCloud.
CVI (CERN Virtualization Infrastructure)
CVI (CERN Virtualization Infrastructure) was a project based in Microsoft technologies, initially with Microsoft Virtual Server and later with Hyper-V. The idea was to enable server consolidation and to offer self service resources to all the CERN community.
CERN IT developed a SOAP interface and a web application to interact with those tools, allowing users to provision virtual machines. This solution was running on top of a wide range of servers, ranging from simple white-box compute nodes to expensive hardware solutions from Dell with Equallogic storage arrays. It was a very popular service in the Organization, hosting more than 3500 virtual machines when it started to be decommissioned in July 2013.
If you are interested about the CVI infrastructure, the following presentations from HEPiX conferences are full of details:
- Status update of the CERN Virtual Infrastructure (Tim Bell, Jan van Eldik) - November, 2010 - https://indico.cern.ch/event/92498/contributions/1278958/
- Evolution of virtual infrastructure with Hyper-V (Juraj Sucik) - October, 2009 - https://indico.cern.ch/event/61917/contributions/2053742/
LxCloud - Batch Service Virtualization
At the same time another group in the CERN IT was trying to virtualize the batch service. The batch service is the biggest IT service in terms of compute resources and virtualizing it would bring a lot of operational advantages.
The project was called LxCloud and it explored two cloud orchestrators. One was an open source solution - “OpenNebula”, and the other was a commercial product - “ISF” from Platform computing.
The hypervisor that we used initially was XEN, because the better performance and support on SLC5 (Scientific Linux CERN 5) when compared with KVM, at that time.
We tested the limits of those products because scalability was a big requirement in order to virtualize the batch service. We ran scalability tests with more than 16000 virtual machines running concurrently in both of these solutions, which provided very interesting results. Also, we developed an image distribution tool based on BitTorrent allowing fast image updates to thousands of servers.
If you are interested about the LxCloud infrastructure, the following presentations from HEPiX conferences are full of details:
- lxCloud Infrastructure - Status and Lessons learned (Belmiro Moreira, Ulrich Schwickerath) - October, 2011 - https://indico.cern.ch/event/138424/contributions/1368440/
- Batch Virtualization project at CERN (Ulrich Schwickerath, Sebastien Goasguen, Ewan Roche, Tony Cass) - October, 2009 - https://indico.cern.ch/event/61917/contributions/2053746
10 years ago, Cloud Computing was already very well established, mainly because of the AWS public Cloud offering. Other large companies would follow with similar offerings like Azure and later GCP. However, for everyone that was exploring how to deploy a private Cloud Infrastructure there weren’t a lot of open source solutions available. So, with no surprise, that the open source release of a new project from NASA and Rackspace that enabled the deployment of Private and Public Clouds was worth all the attention.
The “OpenStack” was born!
From the beginning, OpenStack got the support from several big companies creating a very vibrant community around it.
At that time, for every one working in the Cloud Computing space this was big news! So… we started to look into it… These were very exciting times!…
We installed and made it work :) the first OpenStack release “Austin”. As you may expect from an initial open source release there was no documentation or tutorials, just an engaging community, full of energy, developing the project at a very fast pace (the initial release cycle was 3 months).
In the following link, you can explore what I believe was the first OpenStack presentation at CERN, in January 2011, describing the project (Nova and Swift) and how to interact with it (at that time, with nova-manage and euca2ools):
It was very rough but with a solid foundation. Release, after release we saw more and more features and projects being added to offer a complete Cloud experience.
Agile Infrastructure project
In 2011 CERN IT was still managing the data centre using our custom made tools, the majority of applications were still running on physical nodes and in 2 years CERN would get a new data centre in Budapest (more than 1600 KMs away) to meet the compute and storage requirements of the LHC experiments and for services continuity.
This was the opportunity for reflection and understanding how we can use all the new open source projects now available and participate in the communities around them to improve how the resources are deployed, monitored and managed in our data centre, instead continue to develop and maintain our own tools.
In 2012, the creation of the OpenStack Foundation for the project governance and the consequent OpenStack adoption by the different Linux distributions, gave us the confidence to deploy our private Cloud Infrastructure using OpenStack.
Initial Cloud Infrastructure prototypes
Deploying OpenStack in a running data centre was a massive effort but a very challenge and fun experience.
To make it possible, our strategy was to build several prototypes to try different deployment architectures and configurations, give them to a selected number of users and iterate fast based on their feedback and our discoveries.
Functionality was added incrementally into each prototype. Also, we spent a lot of time with users to help them to test their applications in the Infrastructure and understand their requirements.
Each prototype and workloads were destroyed when a new prototype was configured.
“Guppy” Cloud prototype
“Guppy” (the smallest and weakest of the animals chosen) was deployed on top of Fedora 16 with packages provided by the Fedora Cloud SIG (there were no packages available for CentOS 6 or SLC6, yet). The OpenStack release was “Essex”. We used only a few servers to deploy this infrastructure and the main goal was to validate functionality. Also, we started testing the community Puppet modules.
“Hamster” Cloud prototype
“Hamster” (a much faster and active animal when compared with “Guppy”) used the OpenStack “Folsom” release. It was deployed on SLC6 and Windows Server. We started to test how we could support two different hypervisors in the same Infrastructure and how to schedule Linux instances to the KVM hypervisor and Windows instances to the Hyper-V hypervisor. We did the initial OpenStack integration with the CERN network infrastructure and added the LDAP support. We invited several users to test this Infrastructure for feedback.
“Ibex” Cloud prototype
“Ibex” (the strongest of our animals) also used the OpenStack “Folsom” release. In this prototype we started testing how to configure the different services in a fault tolerant setup. Also, this Infrastructure was running on top of more than 600 compute nodes. We used it to perform scalability tests and we opened it to a wider user community.
The Production CERN Cloud Infrastructure in 2013
In July 2013 we moved from prototypes to a production Infrastructure. Meaning that we wouldn’t destroy the infrastructure anymore (at least voluntarily!) and we will support all running workloads considering our SLA.
How the initial deployment look like?
In 2013 we were running the most recent OpenStack release - “Grizzly”. The Infrastructure was designed to be scalable because, from the beginning, we wanted to convert all the compute servers available in the data centre into OpenStack compute nodes.
Originally we deployed, Keystone, Nova, Glance and Horizon. Few months later we added Ceilometer and Cinder.
We had two Nova cells (cells version 1), one cell in each data centre (one in Geneva and the other in Budapest). We didn’t offer regions at the beginning because we didn’t want to expose the different data centres to our users. Also, we didn’t want to expose unnecessary complexity to be easier for the user when migrating their applications. Nova cells are transparent to the users.
The control plane was deployed to be fault tolerant, with replicated services running on physical nodes.
Each cell database was hosted considering the data centre. We found early that latency was an issue for Nova cells (the latency between the 2 data centres was 24 ms). However, all the other services databases were hosted in Geneva.
Inside the same Nova cell in Geneva we supported KVM and Hyper-V hypervisors. All the linux instances were hosted by the KVM hypervisor and all the Windows instances were hosted by Hyper-V hypervisor. The goal was to offer enough confidence in the Infrastructure to our users and if a problem was discovered we could trigger the proper Operating System support. Also, remember that we had more than 3500 virtual machines running on the CVI project and we needed to migrate them into the OpenStack Infrastructure. Migrating all those virtual machines to run on top of Hyper-V was the easiest migration path.
Nova was integrated with the CERN network infrastructure. The CERN network infrastructure was not designed for a Cloud model, however the data centre was full and running. Touching the network infrastructure was not an option!
Keystone was integrated with CERN Active Directory. This was a key achievement because it simplified user management operations in the OpenStack Infrastructure. The Organization has more than 50k users and ~200 arrivals/departures per month.
We had 3 RabbitMQ clusters with mirrored queues (the main control plane and then 1 per cell). Later we added another RabbitMQ cluster for Cinder.
Glance was initially backed by AFS and only later by Ceph (all images were then migrated from AFS to Ceph). Initially, we configured “glance-api” and “glance-registry”, both in Geneva and Budapest. The compute nodes of each Nova cell were pointed to the correspondent set of Glance servers. Also, Glance was configured with image cache enabled.
Understanding resource utilization is important for our use cases, so we deployed Ceilometer a few months later. It was a challenging project to manage and scale as the Infrastructure grew.
Cinder was also added a few months later when our Ceph infrastructure was deployed. We needed to use a qemu-kvm patched version by Inktank to support RBD.
To be continued…
We had no idea how much the infrastructure would change over time…
In part 2 we will explore the architecture evolution and service offering of the CERN Cloud Infrastructure over the last 7 years and how we scaled from 2 cells and few hundred compute nodes to 80 cells with more than 8k compute nodes.