Author: Mike Dorman

This is a more detailed write-up of a lightning talk I gave at the Open Infrastructure Summit in 2019. Slides are available here, and the video here. (See also the related talk, “Don’t Repeat Our Mistakes: Lessons Learned from Running Go Daddy’s Private Cloud“) Currently I work on the TechOps team at Twilio SendGrid, which […]
The following is a rough transcript of my DevOpsDays Rockies 2019 Ignite talk entitled “Be a Boat Lifter! A Rising Tide that Lifts Everyone Around You.” I also gave a longer lightning talk version of this at Develop Denver 2019. Slides are available here, and the video from the event is here. Additional resources about this topic are […]
The following is a rough transcript of my DevOpsDays Rockies 2018 Ignite talk entitled “The New Hire: Navigating Your First Days.” Slides are available here, and the video from the event is here. Most of us are going to change jobs every four to five years during our careers.  The problem with that is when […]
Slides from our talk at the OpenStack Queens Summit in Sydney, Australia in November 2017.  The video is also posted here. Don’t Repeat Our Mistakes! Lessons Learned from Running Go Daddy’s Private Cloud (OpenStack Sydney Summit) from Mike Dorman
Like many OpenStack operators, we use Open vSwitch on our hypervisors.  Unfortunately, when running Open vSwitch, you lose some ability to do some functions on the lower level physical interfaces on the box (packet flows do not pass through the regular netfilter path in the kernel, for example.)  This has a side effect of LLDP […]
(Note: Credit goes to Sam Morrison at NeCTAR for passing this tip on to us!) We use the Ceilometer notification agent to collect notification events from the various OpenStack services, and then republish those events to a Kafka cluster.  Those events are consumed by other teams within the company that want to be notified about […]
These corrugated steel garden boxes are a nice addition to our back yard.  We have a pretty big garden area, so these are fairly large (4 ft W x 10 ft L x 3 ft H), but they could be scaled to about whatever size you want. The only limitation is the manufactured dimensions of the corrugated […]
We’ve published our Ansible playbook and associated scripts we used to handle upgrading our CentOS 6 OpenStack hypervisor hosts to CentOS 7: https://github.com/godaddy/centos7-upgrade-scripts The README describes how it all works, but you will almost certainly have to make some changes for it to work in your environment.  Use at your own risk!  But hopefully this […]
In a previous post, I described how we implemented Nova Cells v1 support for server group objects using MySQL triggers.  We’d been running that for a while, but needed a real solution (mostly because we created a new cell whose database was in on a different MySQL server.) The patch for Nova is here, and […]
Here’s a fun talk I did with Matt Fischer, Eric Peterson, and Kris Lindgren at the Tokyo summit. This lightning round talk will present various operators horror stories on weird things needed to keep OpenStack running.  Audience participation is almost assured as we share feats of strength and airing of grievances. Slides (PPT) Slides (Google Slides)
We have given a few “show and tell” talks at the last summits and operators’ meetups about the OpenStack architecture we’re running at Go Daddy. We thought we’d some more in-depth posts to dive into it further.
As mentioned previously we do some customizations in Neutron in order to support our layer 3 networking model.  There are no tenant networks, tunneling or overlays.
After our first two posts detailing OpenStack architecture at Go Daddy, this third one in the series focuses on Nova.
We recently had a requirement to enforce selection of an availability zone* when booting instances in nova.  There’s no way to do this in the stock API, we didn’t want to modify code, and a waffle seems like overkill.  So I spent some time playing with policy.json config to see if we could enforce this there: […]
The Liberty Summit going on this week in Vancouver is my third consecutive OpenStack Summit.  After working with OpenStack for almost two years, I no longer feel like a noob when I’m in the company of everyone else at the Summit. As things are winding down here (most of the operators sessions wrapped up yesterday), […]
My tally of favorite talks during the OpenStack Liberty Summit. Thursday By Thursday, you can start to feel the Summit winding down.  For the most part all the operators stuff is done, and many of the remaining time slots are project work sessions.  However, there are still some highlights from the general conference tracks. A […]
Servers that were upgraded from CentOS 6 to CentOS 7 (or RedHat or other derivative) end up with an inconsistent boot loader configuration.  While the grub2 package is actually installed during the CentOS 7 upgrade, it is not actually installed to the MBR/boot sector of the disk. This causes issues later on, because sometimes new […]
Since moving to nova-cells a few months back, we really wanted to still utilize host aggregates and availability zones. Fortunately, we were able to piggyback off work NeCTAR had done to implement this functionality.  Here’s a list of patches we pulled in, and they’ve been working well. nova-api server read cell state from DB:  https://github.com/NeCTAR-RC/nova/commit/6fe7057fb4957485d3bac06579ddc38c93458064 […]
Slides from my talk at the Melbourne Connect Expo OpenStack Conference.  Lots of other good talks by other OpenStack superusers and contributors. OpenStack Cloud Platform Enables Innovation and Efficiencies from Mike Dorman  
Like many other Nova objects, server groups (which can be used to provide affinity/anti-affinity scheduling policy) are not supported by Nova cells. The core problem is the server groups created by nova-api only get populated in the API cell database.  They aren’t put into the compute cell database(s), and thus aren’t available to nova-scheduler.  Instance flavors […]