Category: OpenStack

This is a more detailed write-up of a lightning talk I gave at the Open Infrastructure Summit in 2019. Slides are available here, and the video here. (See also the related talk, “Don’t Repeat Our Mistakes: Lessons Learned from Running Go Daddy’s Private Cloud“) Currently I work on the TechOps team at Twilio SendGrid, which […]
Slides from our talk at the OpenStack Queens Summit in Sydney, Australia in November 2017.  The video is also posted here. Don’t Repeat Our Mistakes! Lessons Learned from Running Go Daddy’s Private Cloud (OpenStack Sydney Summit) from Mike Dorman
Like many OpenStack operators, we use Open vSwitch on our hypervisors.  Unfortunately, when running Open vSwitch, you lose some ability to do some functions on the lower level physical interfaces on the box (packet flows do not pass through the regular netfilter path in the kernel, for example.)  This has a side effect of LLDP […]
(Note: Credit goes to Sam Morrison at NeCTAR for passing this tip on to us!) We use the Ceilometer notification agent to collect notification events from the various OpenStack services, and then republish those events to a Kafka cluster.  Those events are consumed by other teams within the company that want to be notified about […]
We’ve published our Ansible playbook and associated scripts we used to handle upgrading our CentOS 6 OpenStack hypervisor hosts to CentOS 7: https://github.com/godaddy/centos7-upgrade-scripts The README describes how it all works, but you will almost certainly have to make some changes for it to work in your environment.  Use at your own risk!  But hopefully this […]
In a previous post, I described how we implemented Nova Cells v1 support for server group objects using MySQL triggers.  We’d been running that for a while, but needed a real solution (mostly because we created a new cell whose database was in on a different MySQL server.) The patch for Nova is here, and […]
Here’s a fun talk I did with Matt Fischer, Eric Peterson, and Kris Lindgren at the Tokyo summit. This lightning round talk will present various operators horror stories on weird things needed to keep OpenStack running.  Audience participation is almost assured as we share feats of strength and airing of grievances. Slides (PPT) Slides (Google Slides)
We have given a few “show and tell” talks at the last summits and operators’ meetups about the OpenStack architecture we’re running at Go Daddy. We thought we’d some more in-depth posts to dive into it further.
As mentioned previously we do some customizations in Neutron in order to support our layer 3 networking model.  There are no tenant networks, tunneling or overlays.
After our first two posts detailing OpenStack architecture at Go Daddy, this third one in the series focuses on Nova.
We recently had a requirement to enforce selection of an availability zone* when booting instances in nova.  There’s no way to do this in the stock API, we didn’t want to modify code, and a waffle seems like overkill.  So I spent some time playing with policy.json config to see if we could enforce this there: […]
The Liberty Summit going on this week in Vancouver is my third consecutive OpenStack Summit.  After working with OpenStack for almost two years, I no longer feel like a noob when I’m in the company of everyone else at the Summit. As things are winding down here (most of the operators sessions wrapped up yesterday), […]
My tally of favorite talks during the OpenStack Liberty Summit. Thursday By Thursday, you can start to feel the Summit winding down.  For the most part all the operators stuff is done, and many of the remaining time slots are project work sessions.  However, there are still some highlights from the general conference tracks. A […]
Since moving to nova-cells a few months back, we really wanted to still utilize host aggregates and availability zones. Fortunately, we were able to piggyback off work NeCTAR had done to implement this functionality.  Here’s a list of patches we pulled in, and they’ve been working well. nova-api server read cell state from DB:  https://github.com/NeCTAR-RC/nova/commit/6fe7057fb4957485d3bac06579ddc38c93458064 […]
Slides from my talk at the Melbourne Connect Expo OpenStack Conference.  Lots of other good talks by other OpenStack superusers and contributors. OpenStack Cloud Platform Enables Innovation and Efficiencies from Mike Dorman  
Like many other Nova objects, server groups (which can be used to provide affinity/anti-affinity scheduling policy) are not supported by Nova cells. The core problem is the server groups created by nova-api only get populated in the API cell database.  They aren’t put into the compute cell database(s), and thus aren’t available to nova-scheduler.  Instance flavors […]
One of the things that’s come out of the proposed “Refresh quota usages” spec are the little-known nova max_age and until_refresh quota settings.  These options allow you to trigger quota refreshes within Nova at different intervals. This can help with the various quota bugs, where occasionally the quota usages can get out of sync with […]
Key Takeaway – We Need Operators’ Feedback! A common thread throughout the entire meetup is operators providing feedback and comments on specs and reviews.  It’s clear this feedback is invaluable to the developer community.  It also provides benefit to the operators community as our feedback helps shape the priorities of the various projects. I think the […]
Here are some of my notes and main take-aways from the OpenStack Operators meet up going on today and tomorrow in Philadelphia.  (Obviously there is a lot more data and details in the individual etherpads.) The meeting was originally scheduled to take place at the Comcast Center, but was moved to The Hub at United […]
Note: I did a talk at the OpenStack Liberty Summit in Vancouver based on this post.  The slides are available on Slideshare, as well as a video recording of the session. (See the Exploring OpenStack Nova Cells post for an introduction and background information about Nova cells.) If you already have a Nova instance configured with […]