Blogs on Stephen Finucane (Fin-oo-can)

Are We Typed Yet?

Mon, 05 Jan 2026 00:00:00 +0000

This is a living document that describes the status of typing in various Python packages in the OpenStack system. Currently this document focuses on libraries and clients. While many services have introduced some degree of typing, none are complete and they are not intended to be consumed by others, thus they are not documented here.

We determine whether a package is typed or not by looking for two things:

The py.typed file.

This is what Python itself uses to determine whether to consume type hints from the library. If this file is present, the package is considered at least partially typed.
Use of mypy’s strict mode.

mypy is currently the only type checker in use across OpenStack projects. If [tool.mypy] strict is set to true in a package’s pyproject.toml and the py.typed file is present, then the library is considered fully typed.

Oslo libraries

Package	Typed?
automaton	✅ (unreleased)
castellan	❌
debtcollector	✅ (unreleased)
etcd3gw	❌
futurist	❌
microversion-parse	✅ (since 2.1.0)
openstackdocstheme	✅ (unreleased)
os-api-ref	✅ (unreleased)
oslo.cache	✅ (since 4.1.0)
oslo.concurrency	✅ (since 7.3.0)
oslo.config	❌
oslo.context	✅ (since 6.3.0)
oslo.db	❌
oslo.i18n	✅ (since 6.7.0)
oslo.limit	✅ (since 2.9.0)
oslo.log	✅ (since 8.0.0)
oslo.messaging	❌
oslo.metrics	✅ (since 0.15.0)
oslo.middleware	✅ (since 7.0.0)
oslo.policy	✅ (since 5.0.0)
oslo.privsep	✅ (since 3.10.0)
oslo.reports	❌
oslo.rootwrap	✅ (since 7.9.0)
oslo.serialization	✅ (since 5.9.0)
oslo.service	❌
oslo.upgradecheck	✅ (since 2.7.0)
oslo.utils	✅ (since 9.2.0)
oslo.versionedobjects	❌
oslo.vmware	❌
oslotest	✅ (unreleased)
osprofiler	❌
sphinx-feature-classification	✅ (unreleased)
stevedore	✅ (since 5.6.0)
taskflow	❌
tooz	❌
whereto	✅ (unreleased)

SDKs and Clients

Package	Typed?
cliff	✅ (since 4.11.0)
keystoneauth	✅ (since 5.13.0)
openstacksdk	⚠️ (partial, since 0.37.0)
os-service-types	✅ (since 1.8.0)
osc-lib	✅ (since 4.2.0)
python-openstackclient	❌

Glance Tasks API

Fri, 15 Aug 2025 00:00:00 +0000

One the joys (or miseries) of working on OpenStack client tooling is that you get exposed to all the nuances of the OpenStack API tooling. The microservice-like architecture, with projects being managed by often wholly separate teams, means there’s precious little consistency across APIs, beyond that enforced by the long-dormant API SIG. I’ve touched on this before, covering things like the disparity in API versioning, and its been particularly notable during work on the OpenAPI effort. OpenStack is also pretty old (at least in software terms) and most service APIs come with at least a decade of baggage: there is an abundance of APIs that still exist in services but are no longer relevant or useful in modern deployments.

The Glance Tasks API is an example of one such API. It’s a positively ancient API, dating back to the Havana release (2013), and has long-since been made redundant by the image import API introduced in the Mitaka release (2016). Nonetheless, python-openstackclient (OSC) and openstacksdk (SDK) aim to support all OpenStack APIs, and the tasks API, while deprecated, is still present in Glance in the Flamingo release, meaning OSC and SDK should support them.

As always, when adding support for a new API to SDK and OSC, its important that you figure out how that API actually works, ideally by playing around with the API using a local DevStack: the OpenStack API docs are good, but they’re not flawless, and if I had a euro for every time the API docs had said one thing only for the API to behave a different way, I wouldn’t be stuck doing my own painting. Anywho, adding support for the tasks API is no different and I conducted a few tests to prove things out while reviewing some recent patches against both SDK and OSC. Below are my test notes, which I’m sharing in case anyone else ends up having to do something similar.

Notes

We start by creating a task. Because the tasks API is deprecated and replaced by the image import process, OSC has indicated that it won’t add support for task creation. Thus, for this one we need to use either glanceclient or curl. I used the former, creating a sample task for an existing image. I provided syntactically valid-but-nonsense input since I only wanted to test the behavior of the API; I didn’t actually need the task to complete:

❯ glance task-create --type "api_image_import" --input '{
  "image_id": "8bd3bebc-8e1b-40fe-a292-1a63a53dcfd9",
  "import_req": {
    "method": {
      "name": "copy-image"
    },
    "all_stores": true,
    "all_stores_must_succeed": false
  },
  "backend": [
    "fast",
    "cheap",
    "slow",
    "reliable",
    "common"
  ]
}'

Once created, I was able to view this task in a few ways. Firstly, by listing all tasks. This is supported by OSC, so I used that:

❯ openstack image task list
+--------------------------------------+------------------+------------+----------------------------------+
| ID                                   | Type             | Status     | Owner                            |
+--------------------------------------+------------------+------------+----------------------------------+
| 546641fe-6a35-4b0e-90d2-b45367fc94b2 | api_image_import | processing | d8dea803f1fe44e2bdf9d9d017750b4e |
+--------------------------------------+------------------+------------+----------------------------------+

or, via curl:

❯ token=$(openstack token issue -f value -c id)
❯ curl -s -X GET "http://10.0.111.148/image/v2/tasks" \
    -H "Accept: application/json"-H "Content-Type: application/json" -H "X-Auth-Token: ${token}" | jq
{
  "tasks": [
    {
      "id": "546641fe-6a35-4b0e-90d2-b45367fc94b2",
      "type": "api_image_import",
      "status": "processing",
      "owner": "d8dea803f1fe44e2bdf9d9d017750b4e",
      "created_at": "2025-08-15T11:32:10Z",
      "updated_at": "2025-08-15T11:32:10Z",
      "self": "/v2/tasks/546641fe-6a35-4b0e-90d2-b45367fc94b2",
      "schema": "/v2/schemas/task"
    }
  ],
  "first": "/v2/tasks",
  "schema": "/v2/schemas/tasks"
}

I could also inspect the individual task:

❯ openstack image task show 546641fe-6a35-4b0e-90d2-b45367fc94b2
+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                                                    |
+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2025-08-15T11:32:10Z                                                                                                                                                                                                     |
| expires_at | None                                                                                                                                                                                                                     |
| id         | 546641fe-6a35-4b0e-90d2-b45367fc94b2                                                                                                                                                                                     |
| input      | {'image_id': '8bd3bebc-8e1b-40fe-a292-1a63a53dcfd9', 'import_req': {'method': {'name': 'copy-image'}, 'all_stores': True, 'all_stores_must_succeed': False}, 'backend': ['fast', 'cheap', 'slow', 'reliable', 'common']} |
| message    |                                                                                                                                                                                                                          |
| owner_id   | d8dea803f1fe44e2bdf9d9d017750b4e                                                                                                                                                                                         |
| properties |                                                                                                                                                                                                                          |
| result     | None                                                                                                                                                                                                                     |
| status     | processing                                                                                                                                                                                                               |
| type       | api_image_import                                                                                                                                                                                                         |
| updated_at | 2025-08-15T11:32:10Z                                                                                                                                                                                                     |
+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

again, via curl:

❯ curl -s -X GET "http://10.0.111.148/image/v2/tasks/546641fe-6a35-4b0e-90d2-b45367fc94b2"
    -H "Accept: application/json"-H "Content-Type: application/json" -H "X-Auth-Token: ${token}" | jq
{
  "id": "546641fe-6a35-4b0e-90d2-b45367fc94b2",
  "input": {
    "image_id": "8bd3bebc-8e1b-40fe-a292-1a63a53dcfd9",
    "import_req": {
      "method": {
        "name": "copy-image"
      },
      "all_stores": true,
      "all_stores_must_succeed": false
    },
    "backend": [
      "fast",
      "cheap",
      "slow",
      "reliable",
      "common"
    ]
  },
  "type": "api_image_import",
  "status": "processing",
  "owner": "d8dea803f1fe44e2bdf9d9d017750b4e",
  "message": "",
  "result": null,
  "created_at": "2025-08-15T11:32:10Z",
  "updated_at": "2025-08-15T11:32:10Z",
  "self": "/v2/tasks/546641fe-6a35-4b0e-90d2-b45367fc94b2",
  "schema": "/v2/schemas/task",
  "image_id": "8bd3bebc-8e1b-40fe-a292-1a63a53dcfd9",
  "request_id": "req-1918a068-0ce3-4eff-81f5-625e2d0037ae",
  "user_id": "105a84e11f0a421ea814833a5d23f37b"
}

Finally, I could list tasks associated with the image. For reasons unbeknownst to me, glance opted to implement this via a separate, nested API (/images/{imageID}/tasks) rather than a new filter on the existing API (/tasks?image_id={imageID}). This is supposedly supported by glanceclient, but attempting to use this command with the latest glanceclient version yielded the following error:

❯ glance image-tasks 8bd3bebc-8e1b-40fe-a292-1a63a53dcfd9
Server does not support image tasks API (v2.12)

This seemed odd since the API should be supported. Sure enough, when I tried with curl, things worked as expected:

❯ curl -s -X GET "http://10.0.111.148/image/v2/tasks/546641fe-6a35-4b0e-90d2-b45367fc94b2" \
    -H "Accept: application/json"-H "Content-Type: application/json" -H "X-Auth-Token: ${token}" | jq
{
  "id": "546641fe-6a35-4b0e-90d2-b45367fc94b2",
  "input": {
    "image_id": "8bd3bebc-8e1b-40fe-a292-1a63a53dcfd9",
    "import_req": {
      "method": {
        "name": "copy-image"
      },
      "all_stores": true,
      "all_stores_must_succeed": false
    },
    "backend": [
      "fast",
      "cheap",
      "slow",
      "reliable",
      "common"
    ]
  },
  "type": "api_image_import",
  "status": "processing",
  "owner": "d8dea803f1fe44e2bdf9d9d017750b4e",
  "message": "",
  "result": null,
  "created_at": "2025-08-15T11:32:10Z",
  "updated_at": "2025-08-15T11:32:10Z",
  "self": "/v2/tasks/546641fe-6a35-4b0e-90d2-b45367fc94b2",
  "schema": "/v2/schemas/task",
  "image_id": "8bd3bebc-8e1b-40fe-a292-1a63a53dcfd9",
  "request_id": "req-1918a068-0ce3-4eff-81f5-625e2d0037ae",
  "user_id": "105a84e11f0a421ea814833a5d23f37b"
}

Clearly a bug report is needed here.

Finally, task deletion. There isn’t a task deletion API. Rather, tasks have an expiry date, after which glance will reap the task. You can likely force this with glance-manage but I didn’t care enough to do so.

A Closer Look at the Cinder CSI Driver and the Topology Feature

Thu, 03 Apr 2025 00:00:00 +0000

I’ve recently found myself once again working on the OpenStack Cinder CSI Driver and the Operator that OpenShift uses to deploy this. This work has inspired me to improve my knowledge of how the Cinder CSI Driver - and CSI drivers in general - work. Below is my current high-level understanding of both as well as a quick summary of changes we are making to the Cinder CSI Driver Operator in OpenShift 4.19.

Deployment of the Cinder CSI Driver

The Cinder CSI Driver Operator deploys the driver itself as two components: a controller component and a per-node component, which is the typical deployment model for CSI Drivers. The controller component is managed via a Deployment which you can see here. It consists of the controller plugin and a number of sidecar containers which interface between the controller and the Kubernetes controller manager (kube-controller-manager) via a Unix domain socket and handle different RPC calls. Breaking these down one-by-one:

The controller plugin container (csi-driver) implements the Controller Service and Identity Service set of RPCs described in the CSI spec. It is responsible for handling requests by calling the cloud provider’s APIs (Cinder and Nova, this case).

You can find the Cinder CSI implementation of the Controller Service here.
The attacher sidecar container (csi-attacher) watches for attach and detach calls and calls ControllerPublishVolume and ControllerUnpublishVolume, respectively. (source).
The provisioner sidecar container (csi-provisioner) watches for PVC creation and deletion and calls CreateVolume and DeleteVolume, respectively. (source).
The snapshotter sidecar container (csi-snapshotter) does the same as the provisioner but for snapshots, calling CreateSnapshot and DeleteSnapshot. (source).
The resizer sidecar container (csi-resizer) watches for changes to a PVC and calls ControllerExpandVolume as necessary. (source).

The per-node component, by comparison, is deployed to each node using a DaemonSet. You can see the definition for this here. It consists of the node plugin and a single sidecar container:

The node plugin container (csi-driver) implements the Node Service and Identity Service sets of RPCs described in the CSI spec. It is responsible for reporting information about the node and for bind mounting volumes once they are attached to the host. Specifically, it reports an ID of the node, the maximum number of volumes it supports, and topology information. In the case of Cinder, both the ID and topology information are sourced from the metadata service, while the volume limit is determined via a configuration option.

You can find the Cinder CSI implementation of the Node Service here.
The node-driver-registrar sidecar container (csi-node-driver-registrar) registers the CSI driver with kubelet, allowing kubelet to call NodeGetInfo, NodeStageVolume, NodePublishVolume etc. (source).

Changes to topology auto-configuration

Now that we understand the various components that make up the CSI Driver, let’s take a look at the changes we’ve been working on in this area. As I’ve previously discussed, the Cinder CSI Driver has support for Availability Zones (or, in CSI parlance, the CSI Topology Feature) and since OpenShift 4.16 or so the Cinder CSI Driver Operator has supported auto-configuration of this feature. Without getting too into the weeds, the way this is determined is via a simple set comparison: the set of Compute AZs is compared to the set of Block Storage AZs, and if the former isn’t a subset of the latter (e.g. if there was a Compute AZ called foo but no equivalent Block Storage AZ of the same name) then we determine that the feature should be disabled. Once we’ve determined this, we toggle the Topology feature gate of the CSI Provisioner sidecar container, thus ensuring that the AccessibilityRequirements field of the CreateVolumeRequest struct generated by the provisioner (and fed to the controller plugin) would not be populated.

However, things change and the Topology feature is now considered mature and is enabled by default. This means it is likely that the feature flag will be removed at some point in the not-too-distant future, which in turn means we need to find another way to enable and disable topology support from the operator. The solution we’ve arrived at is to copy what was done in Manila and add support for a new --with-topology option to both the controller plugin and node plugin services. This new option has different effects depending on where it is set:

For the controller plugin, the option determines whether (a) the calls to Cinder include a requested AZ and (b) whether the CreateVolumeResponse returned by the CreateVolume call includes topology accessibility information.
For the node plugin, the option determines whether the node reports (a) the capability (as part of the GetPluginCapabilities RPC) and (b) a topology information (as part of the NodeGetInfo call).

This work has been implemented in kubernetes/cloud-provider-openstack#2743 (with some follow-ups in kubernetes/cloud-provider-openstack#2862 and kubernetes/cloud-provider-openstack#2865). With the new option in place, we’ve been able to change how the Operator toggles the Topology feature. Now, instead of enabling and disabling the feature gate on the csi-provisioner container, it can enable and disable the feature on the csi-driver containers in the controller deployment and node daemonsets. That work has been implemented in openshift/csi-operator#345.

Next steps

I’m hoping this is last time I feel the need to write about the Cinder CSI Driver and its Operator. The work we’ve done here should future proof both and ensure that, barring major changes to the CSI Spec itself, few other changes will be needed for the foreseeable. I would however like to get a better understanding of how the equivalent feature in the Manila CSI Driver works, so watch our for a possible post on that topic down the line.

An Update on OpenAPI in Openstack

Thu, 03 Apr 2025 00:00:00 +0000

I previously presented on our work to bring OpenAPI to OpenStack as part of the 2024 OpenInfra Summit Asia, the slides for which you can find here. Since that talk, another release cycle has come and gone and our work has continued to progress. Below is a summary of the current “state of play” for OpenAPI support in OpenStack and a reminder of our long-term goals in the area.

Overview

The work to add OpenAPI schemas can be broken into two parts: adding schemas to services and writing tooling to consume these schemas and generate both documentation and clients.

Services

The tl;dr: of this section is that we are relying on the fact that OpenAPI 3.1 is a superset of JSON Schema Draft 2020-12 and are adding JSON Schema schemas to as many services as possible. When put this way, it sounds pretty simple but as is often the case, the devil is in the detail. OpenStack is made up of multiple independent services maintained by different groups of people, and while the Oslo project has helped ensure some level of consistency around things like configuration (oslo.conf), database connectivity (oslo.db) and messaging (oslo.messaging), there is huge variance in the API frameworks used:

Nova, Cinder, Glance, and Manila use a combination of Routes (for routing), WebOb (for request/response models), Paste (for application dispatch) and PasteDeploy (for middleware).
Neutron uses the same Routes + WebOb + Paste + PasteDeploy combo used by Nova, Cinder, Glance, and Manila, but with the addition of Pecan in place of the “homegrown” WSGI frameworks used in those projects.
Keystone uses Flask and Flask-Restful.
Swift uses PasteDeploy (but I admittedly know very little about Swift).
Ironic uses Pecan (which in turn uses WebOb, so you’ll see references to this too)
etc etc…

Fortunately, despite the wide differences in frameworks, they pretty much all have the same building blocks:

Controllers (or Applications) which are responsible for a given resource and handles calls for same. These are typically classes with methods for each HTTP verb that the API supports.
Middleware that inspects and/or modifies requests and responses.
Routers that map a HTTP request to a controller. This is typically implemented as a special type of middleware.

In this model, both parsing of requests and generation of responses happens in the controller methods, which makes this a natural place to add validation for same. And there are a few things we want to validate:

API version (for services that uses API microversions)
Request path and query string parameters
Request and response bodies

By wrapping the controller methods with decorators, we we are able to inspect both the request or response objects, comparing them against schemas for same.

Tooling

Having schemas in place for all services is of little help if we don’t do anything with them. To this end, we want tooling that can inspect services and extract their JSON Schema schemas, combining them to produce OpenAPI schemas.

Once we have these OpenAPI schemas, we can start generating (self-validating) documentation and clients/libraries. For the former, OpenAPI will replace the os-api-ref Sphinx extension currently used across OpenStack. os-api-ref allows us to describe our APIs in reStructuredText and it is a tool that has worked relatively well for us, but the lack of machine readability means it’s hard to validate against the code. For the latter, we hope to lessen the burden of maintaining libraries and clients in multiple languages, as this has proven very challenging to do given the very large number of OpenStack APIs in multiple. Once again, OpenAPI is better suited to this challenge than os-api-ref or anything else we have.

Current status and future plans

Nova

Nova started from the best position, since it was already using JSON Schema for request validation. However, there were a couple of issues to overcome:

The version of JSON Schema used for the schemas, Draft 04, is over 12 years old, meaning the schemas needed migrating to Draft 2020-12.
There were two different mechanisms for API versioning: if-else checks inside the controller methods, plus a decorator that relied on the descriptor protocol to allow us to define version-specific controller methods. The latter made inspection of the API router more difficult than necessary and had to be replaced.
While many APIs had request query string and request body schemas, not all of them did.
Most importantly, there were no schemas for response bodies.

Most of these have now been resolved. We have bumped our schemas to Draft 2020-12, we have added all the missing request schemas, and we have added response body schemas for a number of resources. The outstanding changes to address the API versioning issues and add the remaining response body schemas are all ready and just waiting for review, so with any luck we will be able to close this out in the 2025.2 (Flamingo) release. Once this work has merged, the final step will be to start generating api-ref documentation from the OpenAPI schemas instead of using the os-api-ref tool we currently use. The patches for Nova can be found here.

Cinder

Cinder was in a very similar position to Nova thanks to their shared lineage. This means it had the same advantage - pre-existing use of JSON Schema for request validation - and the same issues. However, it also had the added issue of having a mountain of tech debt mainly related to the support for multiple API versions that Cinder offered until relatively recently. This has necessitated a lot of additional cleanup patches to do things like remove the cinder.api.v2 module and consolidate everything under cinder.api.v3.

Unfortunately, while the bulk of the changes have been written, none of them have merged. I’m hoping we can double down on this in 2025.2 (Flamingo) release and start making some progress. The patches for Cinder can be found here.

Keystone

Once again, Keystone had the good fortune of having existing use of JSON Schema for request query string and request body validation, but this was done inconsistently and didn’t cover response body schemas. Most of the work here has focused on adding these missing schemas and making the router easier to inspect by moving validation from inside the controller methods to decorators and splitting methods that previously handling multiple endpoints (e.g. GET /foo and GET /foo/123) into multiple single-purpose methods.

Many resources have been fully specced by now but some remain. In addition, there are some decisions to be made regarding support for undocumented API options that have been (re)discovered during this work. None of it should be insurmountable, however. The patches for Keystone can be found here.

Manila

Yet another service with JSON Schema already in place. Once again, the work here consists of adding missing schemas and making the router easier to inspect. A small number of patches have merged here and more are in-flight. The patches for Manila can be found here.

Ironic

Ironic hasn’t historically used JSON Schema for validation, instead using its own homegrown validation framework and taking advantage of some Pecan functionality to maintain API versioning. As a result, there’s quite a bit of work needed to get schemas into Ironic:

Move API versioning to decorators
Rework request path, query string and body parameter validation to use JSON Schema schemas (via decorators) instead of the homegrown framework
Add request body parameter validation

This work is in very early stages, but there are people working on it and there appears to be broad buy-in from the Ironic team, so I hope to see significant progress over the course of the 2025.2 (Flamingo) cycle. The patches for Ironic can be found here.

Other services (Neutron, Swift, Glance, Octavia, …)

To the best of my knowledge, no work has been started in other projects. Of the “core” projects, I expect the Glance effort to be relatively small since they already have formal schemas (exposed via the API!) for most of their resources. Conversely, I expect the Neutron effort to be both large and complicated, given the number and highly dynamic nature of Neutron’s API, driven by it’s extension-based “versioning” system. I have somehow never worked on Swift and haven’t a clue about that.

Tooling

There are two tooling-related efforts ongoing:

Development of the codegenerator project.
Improvements to the openstacksdk project, python-openstackclient project, and related projects.

The codegenerator project provides a collection of utilities for generating OpenAPI schemas and a (Rust-based 🦀) client from these schemas. Development on this tool first started in earnest during the 2024.1 (Caracal) cycle, but the tool has continued to evolve over the course of the 2024.2 (Dalmatian) and 2025.1 (Epoxy) cycles. I expect to see more progress as schemas continue to get added to the Nova project. If you’re interested in learning more about this work, I’d encourage to review the following material from my upstream colleague, Artem Goncharov (gtema), who is driving much of it.

The openstacksdk and python-openstackclient projects, meanwhile, are far more established. As a quick reminder, the openstacksdk project provides the primary OpenStack client library, while the python-openstackclient project is the primary OpenStack CLI. In recent releases, we’ve been undertaking work to prepare them for a future when they can both be at least partially auto-generated. This has taken the form of the addition of typing and a more general “removal of weirdness” goal.

The addition of typing is probably easier to grok and easier to measure. To achieve this, we have been adding type hints to openstacksdk and python-openstackclient, as well as dependencies of same. As of the 2025.1 (Epoxy) release, we have full type coverage for keystoneauth (a dependency of openstacksdk) and for both cliff and osc-lib (dependencies of python-openstackclient). We also have initial coverage of both openstacksdk and python-openstackclient themselves, with more comprehensive coverage of some of the non-core aspects of the former. Completely typing openstacksdk in particular though is more challenging due to the aforementioned weirdness, but we will continue to make progress on this over the 2025.2 (Flamingo) cycle.

The “removal of weirdness” goal is, as we noted, a little more general and cover a few related issues we’re facing. The two biggest of these, however, is (a) the over-use of variadic arguments and keyword arguments (*args and **kwargs) in the proxy and cloud layers in particular, and (b) the munging of request and response body parameters in the Resource class used throughout openstacksdk. The former is hopefully self-explanatory so let’s focus on the latter. By way of a demonstration of the issue, consider the compute service’s /servers API. This API expects you to pass networks and block_device_mapping_v2 fields in the request for POST /servers, to configure networking and block devices respectively. However, it returns this information - in a different form, no less - via the addresses and os-extended-volumes:volumes_attached in response to GET /servers/{serverID} and GET /servers/detail. Because we currently munge the request and response body fields, our Server class therefore looks like this (in highly minimized form):

class Server(resource.Resource):
    addresses = resource.Body('addresses', type=dict)
    attached_volumes = resource.Body(
        'os-extended-volumes:volumes_attached',
        aka='volumes',
        type=list,
        list_type=volume_attachment.VolumeAttachment,
        default=[],
    )
    block_device_mapping = resource.Body('block_device_mapping_v2')
    networks = resource.Body('networks')

But if you attempt to set either addresses or attached_volumes before a Server.create call, you will get an error. On the other hand, if you do a Server.fetch or Server.list call, the block_device_mapping and networks attributes will always be set to None since they don’t form part of the response body. This has necessitated a whole load of special casing in python-openstackclient, and special casing is bad for things you wish to auto-generate. By separating these, we make it easier to type and easier to auto-generate. While a number of PoCs have been produced, none have been implemented yet. We expect work on this to continue during the 2025.2 (Flamingo) cycle.

Questions?

Come discuss these topics in the OpenStack Virtual PTG, running from 07-11 April 2025. More information of the event and sessions can be found at ptg.opendev.org.

Block Storage in OpenStack

Thu, 22 Aug 2024 00:00:00 +0000

It’s an often overlooked fact that OpenStack has two wholly different mechanisms for provisioning block devices for instances. While most people are aware of the Block Storage service, Cinder, not everyone is aware of or gives much thought to the various types of block devices or “ephemeral” storage that the Compute service, Nova, is able to provide. This article serves to provide a high-level overview and comparison of these two different mechanisms, including examples of how you can use them and some hopefully (interesting) asides.

Types of block storage

Broadly speaking, OpenStack has two types of block storage: storage provisioned by Nova, and storage provisioned by Cinder. Somewhat confusingly, these are often referred to as ephemeral storage and persistent storage, respectively. The origin for these names is likely based on the fact that Nova-provisioned storage is associated with an individual instance and has a lifecycle that is tied to the lifecycle instance itself, meaning if you delete the instance then any Nova-provisioned storage associated with that instance is also deleted. By comparison, Cinder-provisioned storage can “persist” after an instance is deleted and may be re-attached to other instances (or in some cases even attached to multiple instances at the same time).

When and where you can use either type of storage depends on the type of disk you want to create, something we’ll get to in a moment. However, to call these storage types ephemeral and persistent invites confusion on multiple fronts. Firstly, there is the fact that Nova already has its own separate concept of “ephemeral disks”, which we cover below. More pressingly, calling Nova-provisioned storage ephemeral suggests it is somehow unsafe or unreliable. In reality, both Nova and Cinder provide a level of configurability about where or how the underlying data for the block devices are stored. Cinder achieves this by being pluggable and supporting a wide variety of drivers, which can be found here. Cinder supports backends like LVM and Ceph, as well as a large variety of other proprietary and open source backends. For Nova, this is determined by the virt driver in use, the value of the [compute] use_cow_images configuration option, and optionally one or more virt driver-specific configuration options. If using the libvirt driver then the following options are all relevant:

[compute] use_cow_image,
[compute] force_raw_images,
[libvirt] images_types (and mechanism-specific options),
and [DEFAULT] instances_path

In a default DevStack deployment, block devices will be stored as qcow2 image files in the path indicated by [DEFAULT] instances_path, /opt/stack/data/nova/instances, on the compute host that instance is located on. When using this default configuration, a migrated instance will need its block storage files migrated too and if the host dies then you will lose any of these Nova-provisioned block devices with it. However, this is only one potential configuration: you can also choose to store the images in Ceph ([libvirt] images_types = rdb, with [libvirt] images_rbd_pool and [libvirt] images_rbd_ceph_conf set to relevant options). Alternatively, you can choose to use use any of the other local storage mechanisms with a suitable replication or backup strategy, whether that’s LVM ([libvirt] images_types = lvm and [libvirt] images_volume_group set to a relevant volume group (VG)) and backups, or one of the file-based mechanisms with the directory indicated by [DEFAULT] instances_path placed on a network-attached filesystem. Suffice to say that, with correct configuration, your data should never be at risk regardless of where it’s placed.

Types of disk

Now that we’re aware of how the block storage is actually stored, let’s look at how these block devices are exposed and used in OpenStack. OpenStack typically refers to block devices as either disks or volumes. I don’t have a good explanation for when you’d use one or the other, and my own mental shortcut is to call any block device attached to an instance a disk, unless that block device is a non-root device provisioned by Cinder in which case it’s a volume. With that clarified (🙃), let’s take a look at the first disk type, root disks.

Root disks

This is the one most people will be familiar with and it’s the only disk that’s absolutely required. As the name suggests, the root disk is where / is mounted. By default, the root disk will be provisioned by Nova and its size will be configured in GB using the disk property of the flavor used to create the instance. For example, by looking at the m1.small flavor on a local DevStack deployment I can see disk set to 20 or 20GB:

❯ openstack flavor show m1.small -f value -c disk
20

If we create a instance using this flavor (and no other block device-related configuration, obviously), we will get a root disk that is 20GB in size.

❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    test-server

We can confirm this by SSH’ing into the machine:

❯ openstack server add floating ip test-server ${FIP}
❯ openstack server ssh test-server -- -l cirros

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda     252:0    0  20G  0 disk
|-vda1  252:1    0  20G  0 part /
`-vda15 252:15   0   8M  0 part

We can also opt to use a Cinder volume for the root disk (which some people will now call a “root volume” - see previous “clarification” 😅). If you’re using OpenStackClient (OSC), this can be accomplished using the --boot-from-volume option:

❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --boot-from-volume 5 \
    test-server

When you do this, Nova will create the volume for you by proxying the request through to Cinder. You can confirm this using the openstack volume list command:

❯ openstack volume list
+--------------------------------------+------+--------+------+--------------------------------------+
| ID                                   | Name | Status | Size | Attached to                          |
+--------------------------------------+------+--------+------+--------------------------------------+
| 68584677-0d53-4c52-8d1c-5600c96768a1 |      | in-use |    5 | Attached to test-server on /dev/vda  |
+--------------------------------------+------+--------+------+--------------------------------------+

And, like above, you can confirm the disk size in the guest:

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda     252:0    0   5G  0 disk
|-vda1  252:1    0   5G  0 part /
`-vda15 252:15   0   8M  0 part

Finally, you can choose to use an existing volume as the root disk. To do this, use the --volume option:

❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --volume test-volume \
    test-server

This volume must be bootable, and you would likely use this to re-create a deleted instance using its root volume.

It’s probably useful to note that, by default, volumes attached to an instance are not automatically removed when the instance is deleted. This is either or a pro or a con, depending on how you look at it. On the positive side, this allows you to re-attach the disk to a new instance (more on this later). On the negative side, this means you can end up with a whole load of stale volumes sitting around, taking up space on your Ceph cluster (or whatever backend Cinder is using). To work around this, you can modify the delete_on_termination attribute of the volume attachment:

❯ openstack server volume list test-server
+----------+--------------------------------------+--------------------------------------+------+------------------------+--------------------------------------+--------------------------------------+
| Device   | Server ID                            | Volume ID                            | Tag  | Delete On Termination? | Attachment ID                        | BlockDeviceMapping UUID              |
+----------+--------------------------------------+--------------------------------------+------+------------------------+--------------------------------------+--------------------------------------+
| /dev/vda | 339acd3a-41e3-45b9-8dbf-1d7dcc1bd8bc | 1bd2c131-7961-4eeb-84ff-4990917bb9b8 | None | False                  | 819c7472-b14f-4867-b894-b68758c6d398 | c69ba987-e66a-4381-8c8a-ceff5f652f0e |
+----------+--------------------------------------+--------------------------------------+------+------------------------+--------------------------------------+--------------------------------------+

❯ openstack server volume set --delete-on-termination 339acd3a-41e3-45b9-8dbf-1d7dcc1bd8bc 1bd2c131-7961-4eeb-84ff-4990917bb9b8

Alternatively, you can set this property when creating the instance. Unfortunately doing this requires using the --block-device option to specify the BDM rather than the --boot-from-volume alias. The equivalent of the previous command using --boot-from-volume would be:

❯ image_id=$(openstack image show cirros-0.6.2-x86_64-disk -f value -c id)
❯ openstack server create \
    --flavor m1.small --network private --key-name test-key \
    --block-device uuid=${image_id},source_type=image,destination_type=volume,delete_on_termination=true,boot_index=0,volume_size=5 \
    test-server

Ephemeral disks

Ephemeral disks are so called because they are associated with a single instance and only exist for the lifetime of that instance. Therefore, as you may guess, these disks can only be provided by Nova. Like root disks, the size of the ephemeral disk is configured in GB via a flavor property, OS-FLV-EXT-DATA:ephemeral. None of the flavors provided in a default DevStack install provide for ephemeral storage, so to demonstrate this we need to create our own flavor which we’re going to call m1.ephemeral:

❯ openstack flavor create --id 99 --vcpus 1 --ram 512 --disk 20 --ephemeral 10 m1.ephemeral

Once created, we can look at the flavor and ensure that the OS-FLV-EXT-DATA:ephemeral property has been set to the relevant size (in this case, 10 or 10GB):

❯ openstack flavor show m1.ephemeral -f value -c 'OS-FLV-EXT-DATA:ephemeral'
10

If we create a instance using this flavor, we will get a root disk that is 20GB in size and a new, additional disk that is 10GB in size:

❯ openstack server create \
    --flavor m1.ephemeral --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    test-server

Once again, we can confirm this by SSH’ing into the machine:

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda     252:0    0   5G  0 disk
|-vda1  252:1    0   5G  0 part /
`-vda15 252:15   0   8M  0 part
vdb     252:16   0  10G  0 disk /mnt

You’ll note that the disk is already mounted for us. This isn’t actually an OpenStack doing this for us. Rather, it’s part of cloud-init, which is included in the Cirros images. If you were using an image that didn’t include cloud-init or were to attach multiple ephemeral disks, you’d need to handle this mounting yourself.

You may also note that there’s an --ephemeral option available for the openstack server create command. This option allows you to change both the layout of the ephemeral storage and the filesystem used on them. For example, say that instead of having a single 10GB disk, I wanted to have an 8GB disk formatted as ext4 and a 2GB disk formatted as XFS, I could invoke OSC like so:

❯ openstack server create \
    --flavor m1.ephemeral --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --ephemeral size=8,format=ext4 --ephemeral size=2,format=xfs \
    test-server

We can see that these are available in the machine using lsblk again and confirm their filesystem types using blkid:

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda     252:0    0  20G  0 disk
|-vda1  252:1    0  20G  0 part /
`-vda15 252:15   0   8M  0 part
vdb     252:16   0   8G  0 disk /mnt
vdc     252:32   0   2G  0 disk

$ blkid
/dev/vdb: LABEL="ephemeral0" UUID="a34015c1-be25-4757-8d17-cc82285b12b2" BLOCK_SIZE="4096" TYPE="ext4"
/dev/vdc: LABEL="ephemeral1" UUID="0cf5d85b-0357-44fc-9549-6f71720af8ba" BLOCK_SIZE="512" TYPE="xfs"
/dev/vda15: SEC_TYPE="msdos" UUID="AE31-5342" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="05ecb6a9-6cb3-4697-b819-25c6cee5bd9a"
/dev/vda1: LABEL="cirros-rootfs" UUID="f1511162-06fb-4482-9dab-9a0c76633fb2" BLOCK_SIZE="4096" TYPE="ext3" PARTUUID="df2f017d-edcc-4371-81c7-fcfa0c5c1b09"

When configuring ephemeral disks this way, the total size of all disks does not have to add up to the size indicted by the flavor’s OS-FLV-EXT-DATA:ephemeral property (meaning you could choose only to configure e.g. the 2GB volume above) but it cannot exceed the size indicated in the flavor. For example, if I repeated the above command but using single 20GB disk (i.e. 10GB larger than the 10GB value specified for the m1.ephemeral flavor we created previously), the request will fail:

❯ openstack server create \
    --flavor m1.ephemeral --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --ephemeral size=20 \
    test-server
BadRequestException: 400: Client Error for url: http://10.0.108.50/compute/v2.1/servers, Ephemeral disks requested are
larger than the instance type allows. If no size is given in one block device mapping, flavor ephemeral size will be
used.

Finally, as we noted at the top, ephemeral disks can only be provisioned by Nova. If you attempt to create an ephemeral disk using destination_type=remote, what you have is no longer an ephemeral disk (in the Nova sense of the term) but rather a volume. Which is a nice segue to…

Volumes

Volumes are not a disk type but rather the name given to the primary resource type provided by Cinder. As noted previously, a Cinder volume can be used as the backing device for the root disk, but it can also be used to provide additional disks to the instance (but not ephemeral disks - an ephemeral disk ceases to be an ephemeral disk once it’s no longer provisioned by Nova). For example, say we wanted to use a standard Nova-provisioned block device for our root disk but attach two additional volumes provisioned by Cinder, we could run:

❯ vol_a_id=$(openstack volume create --size 5 test-volume-a -f value -c id)
❯ vol_b_id=$(openstack volume create --size 5 test-volume-b -f value -c id)
❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --block-device uuid=${vol_a_id},source_type=volume \
    --block-device uuid=${vol_b_id},source_type=volume \
    test-server

We can examine these in the guest:

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda     252:0    0  20G  0 disk
|-vda1  252:1    0  20G  0 part /
`-vda15 252:15   0   8M  0 part
vdb     252:16   0   5G  0 disk
vdc     252:32   0   5G  0 disk

Instead of pre-creating the volume with Cinder, we could also let Nova do this for us. This avoids an extra call to Cinder (at least from us - Nova simply makes the call on our behalf), at the expense of some fine grained configuration for the volume. We could modify the above command like so:

❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --block-device source_type=blank,destination_type=volume,volume_size=5 \
    --block-device source_type=blank,destination_type=volume,volume_size=5 \
    test-server

Unlike the block devices provisioned by Nova, it’s also possible to attach additional block devices to an existing instance:

❯ openstack volume create --size 5 test-volume-c
❯ openstack server add volume test-server test-volume-c

Likewise, you can remove devices as long as they’re not used for the root disk:

❯ openstack server remove volume test-server test-volume-c

Finally, Cinder volumes can be attached to multiple instances at the same time. As discussed in the Cinder documentation, this requires the use of a clustered or multi-attach creation of a special volume type with the multiattach property set to <is> True. Fortunately, the reference LVM driver configured by default in a DevStack deployment supports this, so we can demo creating the volume type, volume, and instances:

❯ openstack volume type create --multiattach multiattach
❯ vol_id=$(openstack volume create --size 5 --type multiattach test-volume -f value -c id)
❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --block-device uuid=${vol_id},source_type=volume \
    test-server-a
❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --block-device uuid=${vol_id},source_type=volume \
    test-server-b

If we log in to these two instances and mount the volume (after partitioning and formatting it), we’ll be able to create a file in one and see it appear in the other:

❯ openstack server ssh test-server-a -- -l cirros

# create partitions using fdisk since parted is not available in cirros images
$ sudo fdisk /dev/vdb
$ sudo mkfs.ext4 /dev/vdb1

$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda     252:0    0  20G  0 disk
|-vda1  252:1    0  20G  0 part /
`-vda15 252:15   0   8M  0 part
vdb     252:16   0   5G  0 disk
`-vdb1  252:17   0   5G  0 part
$ mkdir test
$ sudo mount /dev/vdb1 test
$ sudo touch test/foo
$ exit

❯ openstack server ssh test-server-b -- -l cirros

$ mkdir test
$ sudo mount /dev/vdb1 test  # you may need to reboot to pick up the changes to this disk for this to work
$ ls test
$ ls disk/
foo

If parted was available in the Cirros image used, we could avoid the interactive use of fdisk:

$ sudo parted /dev/vdb mklabel gpt
$ sudo parted /dev/vdb mkpart primary ext4 0% 100%
$ sudo parted /dev/vdc print

Swap disks

The final type of disk is the swap disk. Swap disks are very like ephemeral disks, in that they’re exclusively managed by Nova and configured via a flavor property called swap. Unlike the disk and OS-FLV-EXT-DATA:ephemeral properties though, the swap property is a size in MB, not GB. Once again, the default DevStack configuration does not include flavors with swap enabled so we need to configure these ourselves. Lets do that, creating a flavor called m1.swap with 2048 MB of swap:

❯ openstack flavor create --id 98 --vcpus 1 --ram 512 --disk 20 --swap 2048 m1.swap

Once created, we can look at the flavor and ensure that the swap property has been set to the relevant size (in this case, 1024 or 1024MB):

❯ openstack flavor show m1.swap -f value -c 'swap'
1024

If we create a instance using this flavor, we will get a root disk that is 20GB in size and a additional disk that is 1024MB in size and formatted as a swap disk:

❯ openstack server create \
    --flavor m1.swap --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    test-server

Inspecting the guest itself, we see the disk present and that it has an fstype of swap, as expected:

$ sudo lsblk -f
NAME    FSTYPE FSVER LABEL         UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
vda
|-vda1  ext3         cirros-rootfs f1511162-06fb-4482-9dab-9a0c76633fb2   18.4G     0% /
`-vda15 vfat                       AE31-5342
vdb     swap                       6f7adada-a19f-47e1-915c-20570535a619
$ cat /proc/meminfo | grep -i swap
SwapCached:            0 kB
SwapTotal:             0 kB
SwapFree:              0 kB

However, unlike ephemeral disks, cloud-init does not mount this additional disk for us. If you want that, you’ll need to do so manually:

$ sudo swapon /dev/vdb
$ sudo lsblk -f
NAME    FSTYPE FSVER LABEL         UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
vda
|-vda1  ext3         cirros-rootfs f1511162-06fb-4482-9dab-9a0c76633fb2   18.4G     0% /
`-vda15 vfat                       AE31-5342
vdb     swap                       6f7adada-a19f-47e1-915c-20570535a619
$ cat /proc/meminfo | grep -i swap
SwapCached:            0 kB
SwapTotal:       1048572 kB
SwapFree:        1048572 kB

Like ephemeral disk, there’s also a --swap option for the openstack server create command that you can use to override the default swap size. Once again you cannot exceed the total size given in the flavor, but you also can’t specify the option multiple times to divide the swap into different disks. For example, if we wanted to configure a smaller swap disk size of, say, 512MB, we could do:

❯ openstack server create \
    --flavor m1.swap --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --swap 512 \
    test-server

Attempting to use something larger than the size indicated in the swap property of the flavor will result in an error:

❯ openstack server create \
    --flavor m1.swap --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --swap 2048 \
    test-server
BadRequestException: 400: Client Error for url: http://10.0.108.50/compute/v2.1/servers, Swap drive requested is larger
than instance type allows.

Block Device Mappings (BDMs)

Finally, we get to the final piece of the block storage puzzle: Block Device Mappings, or BDMs. BDMs are how Nova describes mapping of block devices to instances. BDMs are specified during instance creation using the block_device_mapping_v2 field and are objects with a number of well known fields.

Both the Nova user docs and the Nova API Reference docs provide a good overview of these fields, but there are a few in particular worth discussing here in light of the above.

destination_type

This field, which has been referenced earlier in this article, determines what service manages the block devices and therefore where the block device resides. This value can either be local (meaning managed by Nova) or remote (managed by Cinder).
source_type

This indicates the source of the block devices. This value can one of blank, image, snapshot, or volume, and the docs describe all of these in detail. We’ll explore some different use cases for these shortly.
boot_index

This is an integer value that indicates the order that the VM will use when attempting to boot from disks. When attaching a volume as a root disk (i.e. boot from volume), you will use a boot_index of 0 to indicate that the guest OS should boot from that volume from.

With that knowledge in hand, we can look at a few of the examples previously from the perspective of the BDMs used during instance creation. We’ll do this using the openstack server create command with the --debug flag, which allows us to see the raw requests and responses issued to the server.

First, let’s look a root disks. If we create a “standard” instance using a local root disk, then the block_device_mapping_v2 field can be (and, if using OSC, will be) empty:

❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --debug \
    test-server

{
    "server":
    {
        // ...
        "flavorRef": "2",
        "imageRef": "9acb21b3-0516-459a-9b0a-6357e66ff74a"
        // ...
    }
}

This is because Nova automatically creates the BDM for us, making it unnecessary to specify. If we want to use boot from volume instead, placing our root disk on a volume, we will need to specify a BDM:

❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --boot-from-volume 5 --debug \
    test-server

{
    "server":
    {
        // ...
        "flavorRef": "2",
        "block_device_mapping_v2": [
            {
                "uuid": "9acb21b3-0516-459a-9b0a-6357e66ff74a",
                "boot_index": 0,
                "source_type": "image",
                "destination_type": "volume",
                "volume_size": 5
            }
        ],
        "imageRef": ""
    }
}

(9acb21b3-0516-459a-9b0a-6357e66ff74a is the ID of the cirros-0.6.2-x86_64-disk image we created the server with, and it’s passed via the uuid field of the BDM rather than via the imageRef field)

Next, let’s look at ephemeral disks. As with root disks, you can simply use a relevant flavor and omit the block_device_mapping_v2 field and Nova will automatically do the right thing. If you wanted to divide the ephemeral disk into multiple devices though, you’d use a different variation of this field. For example, our prior example of an 8GB ext4 ephemeral disk and a 2GB XFS disk would look like so:

❯ openstack server create \
    --flavor m1.ephemeral --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --ephemeral size=8,format=ext4 --ephemeral size=2,format=xfs \
    test-server

{
    "server":
    {
        // ...
        "flavorRef": "99",
        "block_device_mapping_v2": [
            {
                "uuid": "9acb21b3-0516-459a-9b0a-6357e66ff74a",
                "boot_index": 0,
                "source_type": "image",
                "destination_type": "local",
                "delete_on_termination": true
            },
            {
                "boot_index": -1,
                "source_type": "blank",
                "destination_type": "local",
                "delete_on_termination": true,
                "volume_size": "8",
                "guest_format": "ext4"
            },
            {
                "boot_index": -1,
                "source_type": "blank",
                "destination_type": "local",
                "delete_on_termination": true,
                "volume_size": "2",
                "guest_format": "xfs"
            }
        ],
        "imageRef": "9acb21b3-0516-459a-9b0a-6357e66ff74a"
    }
}

Yes, things are starting to get lengthy…

We’ll skip over volumes for a moment and move onto swap volumes. Yet again, if you don’t want to do anything bar use the swap configured in the flavor then you can omit the block_device_mapping_v2 field, but you’ll need to specify it if you wish to use a different size:

❯ openstack server create \
    --flavor m1.swap --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --swap 512 \
    test-server

{
    "server": {
        // ...
        "flavorRef": "98",
        "block_device_mapping_v2": [
            {
                "uuid": "9acb21b3-0516-459a-9b0a-6357e66ff74a",
                "boot_index": 0,
                "source_type": "image",
                "destination_type": "local",
                "delete_on_termination": true
            },
            {
                "boot_index": -1,
                "source_type": "blank",
                "destination_type": "local",
                "guest_format": "swap",
                "volume_size": 512,
                "delete_on_termination": true
            }
        ],
        "imageRef": "9acb21b3-0516-459a-9b0a-6357e66ff74a"
    }
}

And finally there are volumes. As discussed above, there are huge variety of combinations available here and we’re only going to look at a few of those we previously discussed. Firstly, let’s look at our example of using two precreated volumes:

❯ vol_a_id=$(openstack volume create --size 5 test-volume-a -f value -c id)
❯ vol_b_id=$(openstack volume create --size 5 test-volume-b -f value -c id)
❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --block-device uuid=${vol_a_id},source_type=volume \
    --block-device uuid=${vol_b_id},source_type=volume \
    test-server

{
    "server":
    {
        // ...
        "flavorRef": "2",
        "block_device_mapping_v2":
        [
            {
                "uuid": "9acb21b3-0516-459a-9b0a-6357e66ff74a",
                "boot_index": 0,
                "source_type": "image",
                "destination_type": "local",
                "delete_on_termination": true
            },
            {
                "uuid": "668d9851-b878-4fe5-a244-67585c963a09",
                "source_type": "volume",
                "destination_type": "volume"
            },
            {
                "uuid": "ed7c9976-9fe6-4ada-93cb-4e60bc2a0a05",
                "source_type": "volume",
                "destination_type": "volume"
            }
        ],
        "imageRef": "9acb21b3-0516-459a-9b0a-6357e66ff74a"
    }
}

And now compare this to our example of letting Nova create the volumes for us:

❯ openstack server create \
    --flavor m1.small --image cirros-0.6.2-x86_64-disk --network private --key-name test-key \
    --block-device source_type=blank,destination_type=volume,volume_size=5 \
    --block-device source_type=blank,destination_type=volume,volume_size=5 \
    test-server

{
    "server":
    {
        // ...
        "flavorRef": "2",
        "block_device_mapping_v2":
        [
            {
                "uuid": "9acb21b3-0516-459a-9b0a-6357e66ff74a",
                "boot_index": 0,
                "source_type": "image",
                "destination_type": "local",
                "delete_on_termination": true
            },
            {
                "source_type": "blank",
                "destination_type": "volume",
                "volume_size": "5"
            },
            {
                "source_type": "blank",
                "destination_type": "volume",
                "volume_size": "5"
            }
        ],
        "imageRef": "9acb21b3-0516-459a-9b0a-6357e66ff74a"
    }
}

There are loads of other combinations available here, from creating volumes from snapshots to tagging created volumes, but this should be sufficient to demonstrate the general “feel” of BDMs.

Final thoughts

The main takeaway from this article should be that if you want to understand block devices in OpenStack, you’d be well placed to understand BDMs. While they are terse, they do expose the most important aspects of block devices in OpenStack, such as their differing sources and variety of ways that they can be configured. Does that mean that this article is written backwards and should open with a piece on BDMs? Probably, but what fun would that be.

Availability Zones in Openstack and Openshift (Part 2)

Fri, 03 May 2024 00:00:00 +0000

After seeing a few too many availability zone-related issues popping up in OpenShift clusters of late, I’ve decided it might make sense to document the situation with OpenStack AZs on OpenShift (and, by extension, Kubernetes). This is the second part of two. The first part provided some background on what AZs are and how you can configure them, while this part will examine how AZs affect OpenShift and Kubernetes components such as the OpenStack Machine API Provider, the OpenStack Cluster API Provider, and the Cinder and Manila CSI drivers.

The line up

There’s a couple of OpenStack-specific components we need to be aware of in a typical OpenShift-on-OpenStack (a.k.a. ShiftStack) deployment. My former colleague Michał Dulko provided a good overview of many of these on his blog but to (re-)summarise, you’ve got:

Cloud Provider OpenStack (CPO)
Machine API Provider OpenStack (MAPO)
Cluster API Provider OpenStack (CAPO)

In addition to these three components, there are two others to consider:

Cinder CSI Driver
Manila CSI Driver

Today we’re going to take a look at three of these five components - CPO, MAPO, and the Cinder CSI Driver - and explore how availability zones - both Compute and Block Storage - impact them.

Cloud Provider OpenStack

In contrast to a favoured programming language of mine 🐍, Kubernetes operates on very much batteries not included model. It does not provide out-of-the-box support for such important things as block storage, networking, or ingress. To resolve this, you normally run Kubernetes on top of another platform - be that AWS, vSphere, GCE, or in our case OpenStack - and add additional components that provide integration between your Kubernetes cluster and said platform and its APIs. The cloud-provider interface provides one part of the integration puzzle here, managing the lifecycle of Nodes (including their removal from the cluster if the underlying instance is deleted), Services of type LoadBalancer, and routes.

The OpenStack CCM uses Compute AZ information for the underlying instance to label the corresponding Node. It sets two labels, the topology.kubernetes.io/zone label and the legacy failure-domain.beta.kubernetes.io/zone label. You can see this if you retrieve the labels for a Node:

❯ oc get Node -o jsonpath='{.items[*].metadata.labels}' | jq
{
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/instance-type": "ci.m1.xlarge",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "regionOne",
  "failure-domain.beta.kubernetes.io/zone": "nova",  # <--- !!! here !!!
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "stephenfin-5ps6d-master-0",
  "kubernetes.io/os": "linux",
  "node-role.kubernetes.io/control-plane": "",
  "node-role.kubernetes.io/master": "",
  "node.kubernetes.io/instance-type": "ci.m1.xlarge",
  "node.openshift.io/os_id": "rhcos",
  "topology.cinder.csi.openstack.org/zone": "nova",
  "topology.kubernetes.io/region": "regionOne",
  "topology.kubernetes.io/zone": "nova"    # <--- !!! and here !!!
}

To fetch the AZ information, OpenStack CCM queries the Nova API. You can see that happening here, and publishes this information via the GetZoneByProviderID and GetZoneByName functions, which form part of the cloud-provider API.

Because the labels are defined on Nodes, they are useful for controlling the scheduling of pods, allowing users to spread pods across multiple AZs. This is discussed in more details in the Kubernetes docs, in Assigning Pods to Nodes and Pod Topology Spread Constraints for example. They’re also used for other topology-related features, such as Topology Aware Routing.

With regards to configuring these labels, there’s currently nothing to stop you modifying the labels in place. You shouldn’t do this, however, since it doesn’t change the AZ of the underlying instance and pretty much kills whatever advantage the topology feature has. If you want to change the AZ of the Node then you’ll need to either migrate it (an operation that comes with its own issues) or recreate it.

Machine API Provider OpenStack

The Machine API Provider OpenStack, or MAPO, is a Machine API provider for the OpenStack platform. The Machine API allows you to scale up or scale down your cluster based on workload policies or other preferences and functions quite similarly to the Cluster API, albeit with a different API. You can create Machines manually, but its more common to instead create or modify MachineSets (for workers) or ControlPlaneMachineSets (for masters). You can find more information about the Machine API in the OpenShift documentation.

Like OpenStack CCM, MAPO uses AZ information to label resources - this time setting the machine.openshift.io/zone label on Machine resources:

❯ oc get -A Machine -o jsonpath='{.items[*].metadata.labels}' | jq
{
  "machine.openshift.io/cluster-api-cluster": "stephenfin-5ps6d",
  "machine.openshift.io/cluster-api-machine-role": "master",
  "machine.openshift.io/cluster-api-machine-type": "master",
  "machine.openshift.io/instance-type": "ci.m1.xlarge",
  "machine.openshift.io/region": "regionOne",
  "machine.openshift.io/zone": "nova"  # <-- !!! here !!
}

Also like OpenStack CCM, MAPO sources this AZ information from the Nova API, as you can see here (instanceStatus is a thin wrapper around a ServerExt resource used by Gophercloud). They’re also editable but again, editing them won’t actually change the AZ on the underlying instance and you’ll need to make changes elsewhere to do this. However, unlike with Nodes, you can configure the AZ of a new or existing Machine or MachineSet / ControlPlaneMachineSet as part of the object definition and this change will get reflected in the labels, both of the Machine and of the Node. For example, to define a AZ when creating a new MachineSet:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: <infrastructure_id>-<role>
  namespace: openshift-machine-api
  # ...
spec:
  # ...
  template:
    spec:
      providerSpec:
        value:
          # ...
          availabilityZone: nova-az0
          rootVolume:
            # ...
            availabilityZone: cinder-az0
  # ...

Alternatively, to define one for a ControlPlaneMachineSet:

apiVersion: machine.openshift.io/v1
kind: ControlPlaneMachineSet
metadata:
  name: cluster
  namespace: openshift-machine-api
spec:
  # ...
  template:
    # ...
    machines_v1beta1_machine_openshift_io:
      failureDomains:
        platform: OpenStack
        openstack:
        - availabilityZone: nova-az0
          rootVolume:
            availabilityZone: cinder-az0
        - availabilityZone: nova-az1
          rootVolume:
            availabilityZone: cinder-az1
        - availabilityZone: nova-az2
          rootVolume:
            availabilityZone: cinder-az2
  # ...

Cinder CSI Driver

The last component we’re going to look at here is the Cinder CSI Driver. The Container Storage Interface (CSI) defines a standardised way to expose arbitrary block and file storage systems to Kubernetes workloads, allowing us to plug in storage backends for various cloud platforms or networked storage solutions like NFS or SMB. As you might suspect, the Cinder CSI Driver allows us to plug in storage from the OpenStack Block Storage service, Cinder, and to create PersistentVolumes or PersistentVolumeClaims that correspond to Cinder volumes.

Once again, the Cinder CSI driver uses AZ information to label resources and once again it’s the Nodes that get the resulting label. The Cinder CSI driver sets a single label on a node, the topology.cinder.csi.openstack.org/zone label:

❯ oc get Node -o jsonpath='{.items[*].metadata.labels}' | jq
{
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/instance-type": "ci.m1.xlarge",
  "beta.kubernetes.io/os": "linux",
  "failure-domain.beta.kubernetes.io/region": "regionOne",
  "failure-domain.beta.kubernetes.io/zone": "nova",
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "stephenfin-5ps6d-master-0",
  "kubernetes.io/os": "linux",
  "node-role.kubernetes.io/control-plane": "",
  "node-role.kubernetes.io/master": "",
  "node.kubernetes.io/instance-type": "ci.m1.xlarge",
  "node.openshift.io/os_id": "rhcos",
  "topology.cinder.csi.openstack.org/zone": "nova",    # <--- !!! here !!!
  "topology.kubernetes.io/region": "regionOne",
  "topology.kubernetes.io/zone": "nova"
}

Unlike the other two components we’ve talked about though, the Cinder CSI Driver doesn’t fetch AZ information from the API. Instead, it fetches it from the Metadata API, as you can see here. That NodeGetInfo function forms part of the CSI spec and is used by kubelet, as detailed in the Kubernetes documentation.

Now this use of the Metadata service is somewhat of an usual choice: the Metadata service is provided by Nova and the AZ information it exposes is the AZ of the instance (derived from the host the instance is scheduled to). It’s therefore a Compute AZ so why is being used for a storage-related component? The answer is that we’re using it because there’s nothing else to use: as described in part 1, OpenStack doesn’t provide any mechanism to associate compute hosts with storage AZs outside of using the same naming scheme across both compute and block storage AZs. In my experience, this loose coupling is a very frequent source of bugs. To use the topology feature (which we’ll go into more detail on shortly), you really need to have a common set of AZs for both the Compute and Block Storage services. Until OCP 4.15, the OpenStack Cinder CSI Driver Operator (i.e. the operator that deploys and manages the lifecycle of the Cinder CSI Driver in an OpenShift deployment) assumed this to be the case and always enabled the topology feature. This has since changed but if you’re running an older release then you’re likely to encounter this issue if e.g. using multiple Nova AZs and single Cinder AZ.

Making things even more complicated, migration of the Nova instance corresponding to a Node can result in an instance moving between AZs, assuming the Nova instance in question was not created in a specific AZ initially. The CSI driver will detect this change and will attempt to update the labels on the Node, resulting in the following rather nasty error that will require manual intervention to solve.

Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: error updating Node object with CSI driver node info: error updating node: timed out waiting for the condition; caused by: detected topology value collision: driver reported "topology.cinder.csi.openstack.org/zone":"nova" but existing label is "topology.cinder.csi.openstack.org/zone":"nova-az3",}
Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: error updating Node object with CSI driver node info: error updating node: timed out waiting for the condition; caused by: detected topology value collision: driver reported "topology.cinder.csi.openstack.org/zone":"nova" but existing label is "topology.cinder.csi.openstack.org/zone":"nova-az3", restarting registration container.

However, assuming we know about these issues and work to avoid them, how does one actually use the topology features provided by the Cinder CSI driver? There are actually two ways. The first is to configure a topology-aware StorageClass and use this for a PersistentVolumeClaim, as seen in the examples for the Cinder CSI Driver. For example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: topology-aware-standard
provisioner: cinder.csi.openstack.org
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: topology.cinder.csi.openstack.org/zone
    values:
    - az1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-pvc-with-az
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: topology-aware-standard

By using this storage class, we ensure that the Cinder volume created for the PVC will request an availability zone of az1. This is the standard mechanism promoted by Kubernetes and is also supported by other non-OpenStack CSI drivers that provide topology support.

The other mechanism is to specify a Cinder CSI driver-specific parameter, availability, when creating the storage class. This is effectively a legacy option that pre-dates topology support in CSI but it’s still available:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: legacy-az
provisioner: cinder.csi.openstack.org
volumeBindingMode: WaitForFirstConsumer
parameters:
  availability: az1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-pvc-with-az
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: legacy-az

With regards to configuring these labels there’s currently nothing to stop you modifying the labels in place, just like the labels OpenStack CCM sets on Nodes, and like that you don’t really want to do this since the Cinder CSI driver effectively owns them. The only time you may wish to do this is if you’ve migrated your Nova instance and hit the issue described above. In this case, you can opt to manually relabel the Node, taking care to drain any workload from it first to avoid topology mismatches.

Wrap up

As so concludes this two part series looking at Availability Zones in OpenStack and OpenShift. As you’ve hopefully ascertained, they can be a very useful feature, particularly in larger deployments, but there are more than a few potential banana skins to be aware of when you start using them for storage. By way of recommendations, I would suggest either using a single common AZ for you deployment (you can stick to the default of nova) or a common set of AZs across both the compute and block storage hosts (e.g. in a two-AZ deployment, you could use az0 and az1 for both compute and block storage AZs, rather than nova-az0 and nova-az1 for compute AZs and cinder-az0 and cinder-az1 for block storage AZs). If you insist on sticking with divergent sets of AZs, you should disable the topology feature and rely on the legacy availability parameter of the StorageClass instead.

I hope this has been useful to someone. If you spot any mistakes or identify things that I should have covered but didn’t, feel free to send me an email and I’ll try get things sorted.

API routing in Openstack

Wed, 01 May 2024 00:00:00 +0000

In the Dalamation (2024.2) cycle, we’re working on adding OpenAPI schemas for a number of the OpenStack services. As part of this effort, I’ve had to learn more than I would like to know about how various services’ API machinery works. The below is a quick summary of how the mapping of URLs (or rather, paths) to API controller methods works in Nova (and Cinder, Manila and other projects that have copied or inherited Nova’s patterns). This is very much inside baseball and probably not useful outside of OpenStack, since we’re using older libraries - namely Routes, Paste, and WebOb - that have been mostly superseded by new libraries like Flask, Falcon, or Starlette. Still, maybe it’s useful for someone.

The main entry point for routing in Nova is the APIRouterV21 class. APIRouterV21 provides mappings of URLs (or rather, URL paths) to methods of Resource objects using routes.middleware.RoutesMiddleware as the ultimate dispatcher. A Resource object is a wrapper around some Controller objects: a main controller and zero or more sub-controllers. If you look at nova/api/openstack/compute/routes.py you’ll see a whole load of functools.partial calls where we create Resource objects via calls to _create_controller:

flavor_controller = functools.partial(_create_controller,
    flavors.FlavorsController,
    [
        flavor_access.FlavorActionController
    ]
)

Here, flavors.FlavorsController is the “main” controller and flavor_access.FlavorActionController is a “sub” (or “action”/“child”) controller. The sub-controller extends the main controller by adding new APIs or actions and to the best of my knowledge it is legacy from the days where we had API extensions.

These wrapped controllers are then mapped to paths latter in the same file:

ROUTE_LIST = (
    # ...
    ('/flavors/{id}', {
        'GET': [flavor_controller, 'show'],
        'PUT': [flavor_controller, 'update'],
        'DELETE': [flavor_controller, 'delete']
    }),
    ('/flavors/{id}/action', {
        'POST': [flavor_controller, 'action']
    }),
    # ...
)

class APIRouterV21(base_wsgi.Router):
    # ...
    def __init__(self, custom_routes=None):
        super().__init__()

        # ...

        for path, methods in ROUTE_LIST + custom_routes:
            # register route to mapper
            # ...

Now with that knowledge, you can run this script locally to see the generated path-method mappings:

from nova.api.openstack import compute
from oslo_config import cfg
from nova.tests import fixtures

CONF = cfg.CONF
fixtures.ConfFixture(CONF).setUp()
fixtures.RPCFixture('nova.test').setUp()

router = compute.APIRouterV21()

count = 0
for route in router.map.matchlist:
    if 'controller' not in route.defaults:
        continue
    controller = route.defaults['controller']
    if controller.wsgi_actions == {}:
        continue
    for action, method in controller.wsgi_actions.items():
        print(f'  action: {action}, method: {method}')
        if 'version_select' in str(method):
            count += 1

print(f'useless versioned method count: {count}')

All that this is doing is configuring enough config-related fixtures to allow us to create an APIRouterV21 instance so that we can iterate through the path-controller mappings it has. If you run this, you’ll see a whole load of output finishing in this:

useless versioned method count: 288

This is a count of the number of actions that resolve to version_select methods. version_select methods are not the “real” method and are instead wrappers around the real methods (potentially plural, depending on amount of microversioned revisions) that allow us to handle API versioning. This wrapper methods are useless to us in the OpenAPI work because we need to get attributes of the real methods - namely, some private attributes we’re using to store the schema used for a given method. The way to find the “real” method is to look at the versioned_methods attribute of a Controller, which contains a mapping of method name to real methods. If you change the above for loop you can see this:

count = 0
for route in router.map.matchlist:
    if 'controller' not in route.defaults:
        continue
    controller = route.defaults['controller']
    if controller.wsgi_actions == {}:
        continue
    for action, method in controller.wsgi_actions.items():
        method_name = controller.controller.wsgi_actions.get(action)
        if method_name:
            versioned_methods = getattr(
                controller.controller, 'versioned_methods', {}
            ).get(method_name)
            if versioned_methods:
                method = versioned_methods[0].func
        print(f'  action: {action}, method: {method}')
        if 'version_select' in str(method):
            count += 1

print(f'useless versioned method count: {count}')

Now if you run this, you’ll get a reduced count:

useless versioned method count: 224

However, it’s still not 0. This is because we’re not able to resolve to all “real” methods using only the “main” controller alone. To do that, we need the versioned_methods attribute of the sub-controllers also, or to use the case of the controllers we gave at the start, the versioned_methods attribute of both the FlavorsController controller and the FlavorActionController controller. However, we currently have no reference to FlavorActionController (or any other sub controller) so we can’t do this.

A patch I’ve proposed fixes this so that we store the reference to FlavorActionController (and any other sub-controller) under Resource.sub_controllers, thus giving us a mechanism to retrieve versioned_methods attributes (and any other attribute we might need later) from these sub-controller. With this patch applied, we can change the for loop further:

count = 0
for route in router.map.matchlist:
    if 'controller' not in route.defaults:
        continue
    controller = route.defaults['controller']
    if controller.wsgi_actions == {}:
        continue
    for action, method in controller.wsgi_actions.items():
        main_controller = controller.controller
        method_name = main_controller.wsgi_actions.get(action)
        if method_name:
            versioned_methods = getattr(
                main_controller, 'versioned_methods', {}
            ).get(method_name)
            if versioned_methods:
                method = versioned_methods[0].func
        for sub_controller in controller.sub_controllers:
            method_name = sub_controller.wsgi_actions.get(action)
            if method_name:
                versioned_methods = getattr(
                    sub_controller, 'versioned_methods', {}
                ).get(method_name)
                if versioned_methods:
                    method = versioned_methods[0].func
        print(f'  action: {action}, method: {method}')
        if 'version_select' in str(method):
            count += 1

print(f'useless versioned method count: {count}')

With this done, we finally get a 0 count:

useless versioned method count: 0

This patch therefore means we’ll now be able to inspect elements of the various controller methods. We’re planning to use this as part of the OpenAPI effort to associate a schema with a method so we can ensure all API methods have schemas.

Debugging Failed OpenShift-on-OpenStack Deployments

Mon, 22 Apr 2024 00:00:00 +0000

I deploy OpenShift-on-OpenStack quite regularly these days. Some times these deployments fail and the most common failure I usually see is a timeout during bootstrapping.

$ openshift-install --log-level debug create cluster
DEBUG OpenShift Installer 4.15.10
DEBUG Built from commit 24a827900e76d8f9c79122307415b47a4921bbd7
DEBUG Fetching Metadata...
...
DEBUG Reusing previously-fetched Install Config
INFO Skipping VM console logs gather: no gather methods registered for "openstack"
INFO Pulling debug logs from the bootstrap machine
DEBUG Using SSH_AUTH_SOCK /run/user/1000/keyring/ssh to connect to an existing agent
ERROR Attempted to gather debug logs after installation failure: failed to connect to the bootstrap machine: dial tcp 10.0.212.9:22: connect: connection timed out
ERROR Attempted to gather ClusterOperator status after installation failure: listing ClusterOperator objects: Get "https://api.stephenfin.shiftstack-demo.com:6443/apis/config.openshift.io/v1/clusteroperators": dial tcp 10.0.214.50:6443: i/o timeout
ERROR Bootstrap failed to complete: timed out waiting for the condition
ERROR Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane.

You’ve a couple of tools that you can use to validate this. The first of these is to check the serial console. This will highlight the more egregious issues with your deployment. You can do this with:

$ openstack console url show stephenfin-5ps6d-bootstrap  # replace with your own bootstrap server's name

If this doesn’t show anything weird then the next step is to log in to the server and check the status of the bootkube service. As is custom with OpenStack, to SSH into a machine you need (a) a floating IP and (b) a security group (or more accurately a security group rule) that allows SSH access. The Installer automatically assigns a floating IP to the bootstrap machine so (a) is taken care of. That leaves (b). You like already have an “allow SSH” security group lying around and if so, you can use that now:

$ openstack server add security group stephenfin-5ps6d-bootstrap allow_ssh  # replace with your own server, SG names

If you don’t have such a group, creating one is easy. The following ought to do the trick:

$ openstack security group create allow_ssh
$ openstack security group rule create \
    --protocol tcp --dst-port 22 --remote-ip 0.0.0.0/0 \
    allow_ssh
$ openstack security group rule create \
    --protocol icmp --remote-ip 0.0.0.0/0 \
    allow_ssh

Once you’ve allowed SSH traffic you can SSH into the machine.

$ openstack server ssh stephenfin-5ps6d-bootstrap -- -l core

From here you can follow the directions given in the MOTD and check the bootkube service first:

$ journalctl -b -f -u release-image.service -u bootkube.service

In my case it appeared the issue was the lack of access to the master nodes:

s the base image from which all OpenShift Container Platform images inherit.)
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap bootkube.sh[2449]: Check if API and API-Int URLs are reachable during bootstrap
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap bootkube.sh[2449]: Checking if api.stephenfin.shiftstack-demo.com of type API_URL reachable
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap bootkube.sh[2449]: Unable to reach API_URL's https endpoint at https://api.stephenfin.shiftstack-demo.com:6443/version
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap bootkube.sh[2449]: Unable to validate. https://api.stephenfin.shiftstack-demo.com:6443/version is currently unreachable.
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap bootkube.sh[2449]: Checking if api-int.stephenfin.shiftstack-demo.com of type API_INT_URL reachable
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap bootkube.sh[2449]: Unable to reach API_INT_URL's https endpoint at https://api-int.stephenfin.shiftstack-demo.com:6443/version
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap bootkube.sh[2449]: Unable to validate. https://api-int.stephenfin.shiftstack-demo.com:6443/version is currently unreachable.
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap bootkube.sh[2449]: bootkube.service complete
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap systemd[1]: bootkube.service: Deactivated successfully.
Apr 22 14:01:09 stephenfin-5ps6d-bootstrap systemd[1]: bootkube.service: Consumed 1min 2.337s CPU time.

The same steps apply for debugging issues with master or worker nodes: add a floating IP, allow SSH access, then SSH into the machine.

$ openstack server add floating ip stephenfin-5ps6d-master-0 10.0.214.101
$ openstack server add security group stephenfin-5ps6d-master-0 allow_ssh
$ openstack server ssh stephenfin-5ps6d-master-0 -- -l core

Availability Zones in Openstack and Openshift (Part 1)

Tue, 22 Aug 2023 00:00:00 +0000

After seeing a few too many availability zone-related issues popping up in OpenShift clusters of late, I’ve decided it might make sense to document the situation with OpenStack AZs on OpenShift (and, by extension, Kubernetes). This is the first of two parts. This part provides some background on what AZs are and how you can configure them, while the second part examines how AZs affect OpenShift and Kubernetes components such as the OpenStack Machine API Provider, the OpenStack Cluster API Provider, and the Cinder and Manila CSI drivers.

Background

Both the Compute (Nova) and Block Storage (Cinder) services in OpenStack support the concept of Availability Zones (AZs) and the envisioned use cases is very similar for both. Quoting from the Nova documentation:

Availability Zones are an end-user visible logical abstraction for partitioning a cloud without knowing the physical infrastructure. They can be used to partition a cloud on arbitrary factors, such as location (country, datacenter, rack), network layout and/or power source.

The Nova documentation then goes on to specifically note that the AZ feature provides no HA benefit in and of itself - whatever benefits there are are entirely down to how the deployment is designed - thus it’s really just a way to signal something you’ve done in your physical deployment. All of this is equally true of both Nova and Cinder, and in my experience I’ve seen AZs used to demarcate both compute and block storage nodes existing on different racks or in different datacenters.

Configuring AZs for hosts

As you might expect, Cinder AZ’s are an attribute of the block storage hosts (i.e. hosts running the cinder-volume service). As discussed later, you can configure a host’s AZ by setting the [DEFAULT] storage_availability_zone configuration option in cinder.conf. By comparison, Nova’s AZs are not typically configured via nova.conf but are actually attributes of host aggregates and can be configured by setting the availability_zone metadata key of an aggregate. If a compute host (i.e. a host running the nova-compute service) belongs to a host aggregate with the AZ metadata key set then the host will inherit the AZ of that host aggregate. It’s only when a host doesn’t belong to a host aggregate - or none of the host aggregates it belongs to have AZ metadata set - that this information will be sourced from elsewhere, namely the [DEFAULT] default_availability_zone config option described below. Unlike Cinder’s config option, this is not intended to differ by host and should be set to the same value across all compute nodes. Nova will prevent you adding a host to more than one aggregate with AZ metadata set since a host can only belong to one AZ.

❯ openstack aggregate create --zone nova-az1 foo
❯ openstack aggregate create --zone nova-az2 bar
❯ openstack aggregate add host foo stephenfin-devstack
❯ openstack aggregate add host bar stephenfin-devstack
ConflictException: 409: Client Error for url: http://10.0.109.204/compute/v2.1/os-aggregates/13/action, Cannot add host to aggregate 13. Reason: One or more hosts already in availability zone(s) ['nova-az1'].

In addition, if a host has instances on it, the Nova will also prevent you from modifying the AZ metadata of an aggregate it already belongs to - since this would break the AZ constraint placed on any of the existing instances:

❯ openstack server show test-server -f value -c OS-EXT-AZ:availability_zone -c OS-EXT-SRV-ATTR:host
nova-az1
stephenfin-devstack
❯ openstack aggregate show foo -f value -c availability_zone -c hosts
nova-az1
['stephenfin-devstack']
❯ openstack aggregate set --zone nova-az2 foo
BadRequestException: 400: Client Error for url: http://10.0.109.204/compute/v2.1/os-aggregates/12, Cannot update aggregate 12. Reason: One or more hosts contain instances in this zone.

Requesting AZs for resources (servers, volumes, volume backups, …)

Nova allows you to specify an AZ when creating an instance (or “server”, in OpenStackClient parlance), while Cinder allows you to specify them when creating a volume, a volume backup, a volume group, or (volume groups’ deprecated predecessor) a consistency group. For example, to create an instance (or “server”) with an explicit compute AZ:

openstack server create --availability-zone compute-az1 ...

Likewise, to create a volume and volume backup with an explicit block storage AZ:

openstack volume create --availability-zone volume-az1 ...
openstack volume backup create --availability-zone volume-az2 ...

However, you’ll note that these resource types will always have AZ information associated with them, even when an AZ wasn’t specifically requested during creation. This is because, in the absence of specific AZ information, both services will default to setting the AZ of the resource to the AZ of the host that the resource was created on. Put another way, if I create instance my-server with no AZ information and it ends up on host my-host, then my-server will inherit the AZ of my-host. Block storage resources work in the same way, meaning volume my-volume will inherit the AZ of the host it is scheduled to. As a result, there has historically been no way for an end-user to tell if an AZ was explicitly requested when creating a server or not. In fact, the only way they will find out is if they try to move the server since Nova will insist of moving the instance to another host within the same AZ (this wouldn’t happen for a server that wasn’t explicitly created in an AZ). As we’ll touch on in part 2, this has been rather frustrating from an OpenShift or Kubernetes perspective since Kubernetes’ topology feature is a hard requirement and it does not like us changes the AZ-related labels of Node or Machine objects, which can happen when you migrate the underlying server and the server picks up the AZ of the new host. Fortunately, the 2024.1 (Caracal) release of OpenStack introduced a new field to the GET /servers/{serverID} response called pinned_availability_zone which will show the AZ requested during initial instance creation, if set and it’s just a matter of time before we’re able to start consuming this in the various OpenShift and Kubernetes components.

Combining Nova and Cinder’s AZ features

Finally, it’s worth exploring the interplay of the Nova and Cinder AZ features since this will be particularly relevant in part 2. In a Hyperconverged Infrastructure (HCI) deployment, where compute and block storage services run side-by-side on hyperconverged hosts, the compute hosts are the block storage hosts and there is no difference between the AZs. In a non-HCI deployment, this is unlikely to be the case but this hasn’t prevented people and applications from frequently munging the two types of AZ, as we will see later. Because this conflation of different AZ types can happen, the general expectation we would have is that one of the following is true:

There is only a single compute AZ, a single block storage and they have the same name. This is the default configuration if you use “stock” OpenStack: Nova’s default AZ is nova and Cinder helpfully defaults to the same value.
There are multiple compute and block storage AZs, but there is the same number of both and they share the same name. For example, both the compute and block storage services have the following AZs defined: AZ0, AZ1, and AZ2. In this case, users and applications which incorrectly use compute host AZ information to configure the AZ of volumes and related block storage resources will “just work”.
There are multiple compute and block storage AZs, and there is either a different number of each or they have different names. For example, the compute services have the compute-az0 and compute-az1 AZs defined while the block storage services have the volume-az0 and volume-az1 AZs defined. In this case, the users and applications must be very careful to explicitly specify a correct AZ when creating volumes and related block storage resources and must ensure Nova is configured to allow attaching volumes in other AZs (more of this later too).

The last case above isn’t helped by the fact that neither Nova nor Cinder provide an API to request the correct block storage AZ for a given compute host. To be fair, such an API would likely be a rather difficult thing to do, given multiple backends are a thing to be considered. It would be effectively impossible to do automatically, meaning there would still be initial manual configuration required. The closest analog we have for his today is the Volume Type AZ feature, which allows you to indicate the AZs that can be used when creating a volume with a given volume type (so that e.g. a particular block storage backend that is only available to one rack can’t be requested by volumes hosted by block storage services running on another rack). As the docs for that indicate, this configuration is entirely deployment specific and therefore totally manual.

Wrap up

That concludes part 1 of this OpenShift-centric examination of OpenStack Availability Zones. In this part we focused almost exclusively on OpenStack itself, looking at what AZs are, how they’re configured and used, and the various issues people are likely to encounter along the way, but in part 2 we’re going to turn our focus to how OpenStack AZs are consumed and represented by OpenShift components when an OpenShift cluster is deployed on an OpenStack cloud. Stay tuned!

Reference

Configuration

Since this feature exists across two services, there are two sets of configuration options to be concerned with.

As of the 2023.1 (Antelope) release, Nova has three relevant configuration options:

[DEFAULT] default_availability_zone defines the default AZ of each compute host, which can be changed by adding the host to a host aggregate and setting the special availability_zone metadata property as described in the nova docs. This option defaults to nova and as noted in the nova docs, the default AZ should never explicitly requesting this AZ when creating new instances since it will prevent migration of instance between different hosts in different AZs (which is allowed by default if the AZ was unset during initial creation) as well as identification of hosts that are missing AZ information. You have been warned.
[DEFAULT] default_schedule_zone defines the default AZ that should be assigned to an instance on creation. If this is unset, the instance will be assigned an implicit AZ of the host it lands on. You might want to use this if you wanted the majority of instances to go into a “generic” AZ while special instances can go into specific AZs.
[cinder] cross_az_attach determines whether volumes are allowed to be attached to an instance if the instance host’s compute AZ differs from that of the volume’s block storage AZ. It also determines whether volumes created when creating a boot-from-volume server have an explicit AZ associated with them or not. This defaults to true and with good reason, given the aforementioned caveats around munging of compute and block storage AZs and the need for them to be identical.

There is also the [DEFAULT] internal_service_availability_zone configuration option, but this has no real impact for end-users.

As of the 2023.1 (Antelope) release, Cinder has four relevant configuration options:

[DEFAULT] storage_availability_zone defines the default AZ of the block storage host. This defaults to nova and can be overridden on a per-backend basis using [foo] backend_availability_zone. Speaking of which…
[foo] backend_availability_zone define the default AZ for a specific backend of the block storage host. foo should be the name of the volume backend, as defined in [DEFAULT] enabled_backends.
[DEFAULT] default_availability_zone defines the default AZ that should be assigned to a volume on creation. If this is unset, the volume will be assigned the AZ of the host it lands on (which in turn defaults to [DEFAULT] storage_availability_zone, per above).
[DEFAULT] allow_availability_zone_fallback allows you to ignore an request for an invalid block storage AZ and instead fallback to the default AZ defined in [DEFAULT] default_availability_zone. This defaults to false, though to be honest true is probably a sensible value for configurations where e.g. there are multiple compute AZs and a single volume AZ.

Usage

Again, since this feature exists across two services, there are two sets of resource types to be concerned with.

To configure the AZ of a compute host, you configure AZ information for a host aggregate and then add the host to this aggregate.

❯ openstack aggregate create --zone nova-az1 foo
❯ openstack aggregate add host foo stephenfin-devstack

Once this is done, you can request the AZ when creating an instance (or “server”):

❯ openstack server create --availability-zone nova-az1 ...

On the other hand, the AZ of a storage host is configured via config and there’s no API method to configure it. You can use it when creating a volume just like creating a server though:

openstack volume create --availability-zone volume-az1 ...

Or when creating a volume backup:

openstack volume backup create --availability-zone volume-az2 ...

Other API libraries like gophercloud also expose these attributes and allow them to be configured, but we won’t go into that here.

Listening to notifications in Openstack

Fri, 07 Jul 2023 00:00:00 +0000

Another short one. As mostly anyone that has had to maintain an OpenStack cluster will know, many of the OpenStack services has the ability to report notifications via the message bus. These notifications will be enabled in most production deployments but DevStack doesn’t enable them by default. Fortunately enabling them is pretty painless. If we wanted to enable the notifications in, say, Keystone, we can do so by configuring the following in /etc/keystone/keystone.conf:

[oslo_messaging_notifications]
transport_url = rabbit://stackrabbit:***@***:5672/
driver = messagingv2

These options are provided by the oslo.messaging project, which virtually every OpenStack service harnesses.

Once you’ve enabled them though, you what can you do with them? Using the messagingv2 means they’re published over the AMQP message bus (and the message is enveloped, which is the difference to messaging), however, they’re not exposed via HTTP APIs or similar. We could whip out a telemetry tool like Ceilometer but that seems overkill for a quick bit of debugging. Well it turns out oslo.messaging provides utilities for both creating notifications and listening to them. Writing a listener is relatively simple:

import json
import logging

from oslo_config import cfg
import oslo_messaging

cfg.CONF()
logging.basicConfig(level=logging.INFO)
LOG = logging.getLogger()

URL = 'rabbit://stackrabbit:***@***:5672/'


class NotificationEndpoint:

    def _notify(self, ctxt, publisher_id, event_type, payload, metadata):
        LOG.info('notification received %s:%s' % (publisher_id, event_type))
        output = json.dumps(
            {
                "payload": payload,
                "publisher_id": publisher_id,
                "event_type": event_type,
            },
            indent=4,
        )
        print(output)

    audit = _notify
    debug = _notify
    info = _notify
    warn = _notify
    error = _notify
    critical = _notify
    sample = _notify


def main():
    transport = oslo_messaging.get_notification_transport(cfg.CONF, url=URL)
    targets = [oslo_messaging.Target(topic='notifications')]
    endpoints = [NotificationEndpoint()]

    server = oslo_messaging.get_notification_listener(
        transport,
        targets,
        endpoints,
        executor='threading',
    )

    LOG.info("messaging starting")
    server.start()
    LOG.info("messaging started")
    server.wait()
    LOG.info("exit")


if __name__ == '__main__':
    main()

If you run this and then, say, attempt to authenticate with invalid credentials, you’ll see notifications arriving:

{
    "payload": {
        "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
        "eventType": "activity",
        "id": "68c5b24b-89f0-5c93-abe4-5c3d6651fb00",
        "eventTime": "2023-07-07T15:52:00.496156+0000",
        "action": "authenticate",
        "outcome": "failure",
        "observer": {
            "id": "08d95fde312344debbcc00106b139995",
            "typeURI": "service/security"
        },
        "initiator": {
            "id": "78fab588d9d34900a6c59fc22f3a2049",
            "typeURI": "service/security/account/user",
            "host": {
                "address": "***",
                "agent": "openstacksdk/1.3.0 keystoneauth1/5.2.1 python-requests/2.31.0 CPython/3.11.4"
            },
            "request_id": "req-0126be50-db3a-4975-a814-d717a5fdfdf8",
            "user_id": "***",
            "username": "***"
        },
        "target": {
            "id": "fc220ce7-2787-5152-bf81-9943cb8ee24b",
            "typeURI": "service/security/account/user"
        }
    },
    "publisher_id": "identity.stephenfin-devstack",
    "event_type": "identity.authenticate"
}

This can be extremely valuable for debugging since notifications will expose things that aren’t necessarily obvious from logs.

Multipoint VXLAN Tunnels

Mon, 26 Jun 2023 00:00:00 +0000

I recently had to deploy a multi-node OpenStack cluster in Azure. Like most cloud platforms, Azure does not expose L2 networks and implements MAC spoofing protection, preventing VMs from advertising IPs that have not been assigned to them. However, we want a provider network in our deployment in order to provide external network connectivity to OpenStack VMs, which meant we needed some kind of L2 network. Enter overlay networks, specifically VXLAN network, which will let us provide a virtual L2 domain for our created VMs. We ended up not needing this as we only deployed a single networking controller node, but it was an interesting exercise all the same. I’ve documented the steps I needed to take to get this functioning below.

The scenario

We have three hosts: a bastion and two OpenStack hosts. We would like to configure a VXLAN mesh between all three VMs.

Bastion: 192.168.226.4 (VXLAN IP: 192.168.234.1/23)
Host 00: 192.168.226.5 (VXLAN IP: 192.168.234.2/23)
Host 01: 192.168.226.6 (VXLAN IP: 192.168.234.3/23)

The solution

On bastion:

ip link add provider0 type vxlan id 16 local 192.168.226.4 dstport 0
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.5 dev provider0  # host-00
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.6 dev provider0  # host-01
ip addr add 192.168.234.1/23 dev provider0
ip link set up dev provider0

On host-00:

ip link add provider0 type vxlan id 16 local 192.168.226.5 dstport 0
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.4 dev provider0  # bastion
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.6 dev provider0  # host-01
ip addr add 192.168.234.2/23 dev provider0
ip link set up dev provider0

On host-01:

ip link add provider0 type vxlan id 16 local 192.168.226.6 dstport 0
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.4 dev provider0  # bastion
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.5 dev provider0  # host-00
ip addr add 192.168.234.3/23 dev provider0
ip link set up dev provider0

The (alternative) solution

If using Debian or Ubuntu, you can achieve part of this using netplan. This can be helpful when using e.g. Ansible. If using netplan, simply omit the network.tunnels.{name}.remote setting, like so:

network:
  tunnels:
    provider0:
      mode: vxlan
      id: 16
      local: 192.168.226.4
      addresses: [ 192.168.234.1/23 ]
  version: 2

You then need to run the bridge fdb commands manually as it doesn’t seem to be possible to configure this via netplan.

The explanation

Typically, you would create a VXLAN tunnel endpoint (VTEP) using a command like so:

ip link add provider0 type vxlan id 16 local 192.168.226.4 remote 192.168.226.5 dstport 0
ip addr add 192.168.234.1/23 dev provider0

However, the ip command does not allow you to specify multiple remote stanzas. As a result, you need to omit this and manually specify forwarding database entries using the bridge fdb command.

BONUS! Using GRE instead

This wasn’t an option for me because GRE is apparently blocked by Azure. In theory though, there’s no reason this shouldn’t work otherwise.

On bastion:

ip tunnel add provider0 mode gre local 192.168.226.4 key 1234
ip addr add 192.168.234.1/23 dev provider0
ip neighbor add 192.168.234.2 lladdr 192.168.226.5 dev provider0  # host-00
ip neighbor add 192.168.234.3 lladdr 192.168.226.6 dev provider0  # host-01
ip link set dev provider0 up

On host-00:

ip tunnel add provider0 mode gre local 192.168.226.5 key 1234
ip addr add 192.168.234.2/23 dev provider0
ip neighbor add 192.168.234.1 lladdr 192.168.226.4 dev provider0  # bastion
ip neighbor add 192.168.234.3 lladdr 192.168.226.6 dev provider0  # host-01
ip link set dev provider0 up

On host-01:

ip tunnel add provider0 mode gre local 192.168.226.6 key 1234
ip addr add 192.168.234.3/23 dev provider0
ip neighbor add 192.168.234.1 lladdr 192.168.226.4 dev provider0  # bastion
ip neighbor add 192.168.234.2 lladdr 192.168.226.5 dev provider0  # host-00
ip link set dev provider0 up

References

Deploying MetalLB with BGP on Openstack (Part 2)

Tue, 23 May 2023 00:00:00 +0000

As noted previously, one of the goals for an upcoming OpenShift release is to formally support MetalLB and the MetalLB operator on the OpenStack platform. In part one of the series, we configured an environment with OpenStack, OpenShift and BGP software routers. Now, in part two, we’re going to focus on installing and configuring MetalLB itself.

Install MetalLB

The first step on the path to using MetalLB is actually installing it. As a reminder, we want to use MetalLB in BGP mode. This necessitates things like routers that speak BGP and an OpenStack deployment that is configured to talk to these routers. These were all discussed in part one of this series, and if you followed along with this then you will currently have a deployment that looks like this:

Our deployment configured

If this is not the case, you probably want to read that part first. However, assuming it is, we can now proceed with installation. As a Kubernetes-native project, MetalLB comes with all the usual mechanisms for installation of Kubernetes components. Plain manifests are provided, as are Helm Charts and an operator, the MetalLB Operator. The various installation mechanisms are all discussed in the MetalLB installation guide.

Because operators are awesome, I opted to use the MetalLB Operator to deploy MetalLB and manage its lifecycle. The MetalLB Operator is available on OperatorHub at operatorhub.io/operator/metallb-operator but when I was testing, the main branch contained a feature I wanted, namely the ability to configure a loadBalancerClass. As a result, I opted to deploy MetalLB Operator from source. The MetalLB Operator provides a very helpful Make target to do this, which you can use:

❯ git clone https://github.com/metallb/metallb-operator
❯ cd metallb-operator
❯ make deploy-openshift

Deployment takes a while but once finished we can validate that everything is running:

❯ oc get all -n metallb-system

Installation complete! Now onto configuration.

Initial MetalLB configuration

With the MetalLB Operator installed, it’s time to configure MetalLB. By using MetalLB Operator we gain the ability to manage configuration of MetalLB itself via CRs. First up is the MetalLB CR. This is primary configuration mechanism and is the thing that enables MetalLB itself. There should only be one of them, which you can create like so:

❯ cat << EOF | oc apply -f -
apiVersion: metallb.io/v1beta1
kind: MetalLB
metadata:
    name: metallb
    namespace: metallb-system
spec:
    loadBalancerClass: 'metallb.universe.tf/metallb'
    nodeSelector:
        kubernetes.io/os: linux
        node-role.kubernetes.io/worker: ""
EOF

There are a couple of things to note here. Firstly, we are configuring spec.loadBalancerClass. This is necessary because dev-install deploys Octavia by default and cloud-provider-openstack is using this for load balancing by default. By setting this, we have the ability to later configure Services to use MetalLB rather than Octavia. Secondly, we are restricting the speakers to run on worker nodes by configuring spec.nodeSelector as there’s simply no need to run them on the master nodes.

Once the CR is created, we can validate that it exists:

❯ oc get -n metallb-system metallb metallb

Assuming so, you can ensure the loadBalancerClass attribute is in effect by inspecting the underlying service containers:

❯ oc get -n metallb-system pods
NAME                                                  READY   STATUS    RESTARTS      AGE
controller-7df7bbcffb-8cqzb                           1/1     Running   0             5m1s
metallb-operator-controller-manager-c44c967b9-l6rvx   1/1     Running   0             13h
metallb-operator-webhook-server-6fdccfb5c5-k8b2m      1/1     Running   1 (13h ago)   13h
speaker-bg6pq                                         4/4     Running   0             5m1s
speaker-q6dmg                                         4/4     Running   0             5m1s
speaker-sjmtc                                         4/4     Running   0             5m1s
speaker-z9zrr                                         4/4     Running   0             5m1s

❯ oc get -n metallb-system pod controller-7df7bbcffb-8cqzb -o jsonpath="{.spec.containers[0].args}" | yq -P
- --port=7472
- --log-level=info
- --cert-service-name=webhook-service
- --lb-class=metallb.universe.tf/metallb
- --webhook-mode=disabled

❯ oc get -n metallb-system pod speaker-bg6pq -o jsonpath="{.spec.containers[0].args}" | yq -P
- --port=7472
- --log-level=info
- --lb-class=metallb.universe.tf/metallb

This all looks good. With this, the initial configuration of MetalLB is also complete. Now to the next step: configuring MetalLB for our BGP environment.

Configuring MetalLB to talk to the router

MetalLB in BGP mode requires a few bits of information. It needs a list of IP addresses that it can hand out, it needs information about the routers that it should peer with, and it needs to be told which IP addresses it can advertise via BGP. This configuration is all done using CRs, namely the IPAddressPool, BGPPeer, and BGPAdvertisement CRs. First up, the IP address pools.

cat << EOF | oc apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  namespace: metallb-system
  name: ipaddresspool
spec:
  addresses:
    - 192.168.50.1-192.168.50.254
EOF

I’ve picked a totally arbitrary IP address range, ensuring it doesn’t overlap with any other IP address range on the network. Obviously if you have specific IP addresses you wish to use, you should configure these instead. I’ve also only configured one. This should be loads for our purposes.

Next, the BGP peers. While we have multiple leaf routers, our master and worker nodes are all talking to the rack1-gateway router. As a result, we really only need to create a BGP peer for this router.

cat << EOF | oc apply -f -
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: rack1-bgp-peer
  namespace: metallb-system
spec:
  myASN: 64998
  peerASN: 64999
  peerAddress: 192.168.10.1
  password: "f00barZ"
EOF

The poc-bgp project we used to configure our BGP software router uses the 64999 ASN for all of the leaf routers and configured a password of f00barZ. Since we’re pairing with one of these leaf routers, rack1-gateway, we needed to configure these. We chose the 64998 ASN for ourselves.

Finally, the BGP advertisement. Because MetalLB supports both a BGP and an L2 mode, it is possible that you could have some IP addresses that are meant to be assigned using BGP and other IP addresses that are meant to be assigned using ARP. In our case, we are only configuring MetalLB in BGP mode and only have a single IP address pool, which means we can advertise all IPs via BGP.

cat << EOF | oc apply -f -
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: bgpadvertisement
  namespace: metallb-system
EOF

With those three CRs created, we can validate that they have been in fact created.

❯ oc get -n metallb-system ipaddresspool
NAME            AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
ipaddresspool   true          false             ["192.168.50.1-192.168.50.254"]

❯ oc get -n metallb-system bgppeer
NAME             ADDRESS        ASN     BFD PROFILE   MULTI HOPS
rack1-bgp-peer   192.168.10.1   64999

❯ oc get -n metallb-system bgpadvertisement
NAME               IPADDRESSPOOLS   IPADDRESSPOOL SELECTORS   PEERS
bgpadvertisement

Wonderful!

Dealing with port security issues

With the above steps completed, our configuration of MetalLB is complete. However, if you were to create a Service with type=LoadBalancer now, you would find it doesn’t actually work. This is because Neutron has MAC spoofing protection that is enabled by default. The fact that our worker node is advertising IP addresses that neutron does not know about triggers this protection and results in packets getting dropped as they egress our worker node.

To work around this issue you have two options: you can disable port security, or you make use of neutron’s allowed-address-pairs extension to allow additional IPs, subnets, and MAC addresses, other than the fixed IP and MAC address associated with the port, to act as source addresses for traffic leaving the port/virtual interface. Let’s start with the former.

Disable port security

Disabling port security requires removing any existing allowed address pairs, removing any security groups, and finally disabling port security in general. If you you opt to do this, you will need to do this it for all worker node ports. As we only only have one worker node here, there is only one port to worry about, but you can trivially script the removal of port security for a larger number of ports using openstackclient or something like Ansible. For example:

ports=(
    stephenfin-qfnvm-worker-0-97fkv
)

for port in "${ports[@]}"; do
    openstack port set --no-allowed-address "$port"
    openstack port set --no-security-group "$port"
    openstack port set --disable-port-security "$port"
done

This is definitely the simpler option, though you will obviously be reliant on something else to provide network security. Let’s look at the alternative option.

Configure allowed address pairs

The allowed-address-pairs extension can be used to allow egress traffic from a VM with an IP outside of the one configured on the neutron port. From the docs:

The allowed-address-pairs extension adds an allowed_address_pairs attribute to ports. The value of allowed_address_pairs is an array of allowed address pair objects, each having an ip_address and a mac_address. The set of allowed address pairs defines IP and MAC address that the port can use when sending packets if port_security_enabled is true (see the port-security extension). Note that while the ip_address is required in each allowed address pair, the mac_address is optional and will be taken from the port if not specified.

If you inspect the ports assigned to the master and worker nodes, you’ll note that we already have some allowed address pairs defined. OpenShift on OpenStack uses this feature to enable load balancing of the ingress and API VIPs and we can opt to use it for load balancing of services also. Unfortunately, there is no easy way to create allowed address pairs for all IP addresses in a given subnet, nor to create them in a manner where they apply to all hosts. As a result, applying this will be tedious and scalability may well become a concern, particularly where there is a large number of worker nodes or a large range of IP addresses (via IPAddressPool). Since we only have one node and will only create one example, we can at least try it out.

❯ openstack port set --allowed-address=ip-address=192.168.50.1 stephenfin-qfnvm-worker-0-97fkv-0

If you wanted to create multiple services, you would need to create an allowed address pair for each and every IP address specified in the IPAddressPool(s).

This is the more involved option but either option should work. In any case, with port security issues mitigated, we are finally in a position to validate behavior.

Testing it out

Let’s create a simple “Hello, world” example to test this out. We’ll use the e2e-test-images/agnhost image to do this:

❯ oc new-project test-metallb
❯ oc create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname
❯ cat  << EOF | oc apply -f -
---
apiVersion: v1
kind: Service
metadata:
  name: test-frr
spec:
  selector:
    app: hello-node
  ports:
    - port: 80
      protocol: TCP
      targetPort: 9376
  loadBalancerClass: metallb.universe.tf/metallb
  type: LoadBalancer
EOF

This should look very familiar if you’ve ever worked with services. In fact, there’s nothing unusual here aside from our use of spec.type=LoadBalancer and the declaration of spec.loadBalancerClass. The former ensure we actually use a load balancer while the latter ensures that the load balancer used is MetalLB rather than Octavia.

Once created, inspect the service to find the IP address assigned:

❯ oc get service test-frr
NAME       TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE
test-frr   LoadBalancer   172.30.130.128   192.168.50.1   80:32519/TCP   2m13s

From this, we can see that the service has been assigned IP 192.168.50.1. If we curl this, we should get something back. Let’s test it out.

❯ curl 192.168.50.1
hello-node-855787d74c-fkbt5

And it works. Good job, people!

Wrap Up

In this post, we deployed MetalLB via the MetalLB Operator, configured it a minimal manner, and worked around the port security issues that using an external routing solution with neutron presents. As you can see though, port security issues aside, the actual process of deploying and configuring MetalLB is rather effortless.

In a future post, I plan to outline some of the steps I used to debug and resolve issues I had deploying this initial configuration (there were a few). For now, I hope this was helpful to someone.

Deploying MetalLB with BGP on Openstack (Part 1)

Mon, 22 May 2023 00:00:00 +0000

One of the goals for an upcoming OpenShift release is to formally support MetalLB and the MetalLB operator on the OpenStack platform. While MetalLB is mainly targeted at bare metal deployments, it also has value on some on-prem platforms such as OpenStack; for example, if you can’t or don’t want to deploy the OpenStack Load Balancing service (Octavia). I’ve been investigating how this would work and this post consists of my testing notes, along with some asides to help flesh things out. I took a few shortcuts, particularly when it comes to initial deployment, so I don’t know how broadly useful this will be. However, there are very few blogs talking about MetalLB in BGP mode on OpenStack so I imagine there’s something of use here.

One thing to note from the get-go is that this focuses on using MetalLB as a load balancer for applications or data plane - that is, Service instances with type=LoadBalancer. Like OpenShift on bare metal, OpenShift on OpenStack uses keepalived and HAProxy to load balance the API and ingress VIPs (which you can see in action by inspecting the pods in openshift-openstack-infra namespace). There are no current plans to provide a mechanism for using MetalLB to load balance the control plane.

This is the first part of two and focuses on deploying OpenStack, configuring a BGP environment, before deploying an OpenShift cluster. The end result should look something like this:

Our proposed deployment

This is likely the most involved section to do from scratch owing to the variety and varying complexity of options available for configuring OpenStack, BGP, and OpenShift. It is also heavily dependent on the hardware you have available to you. As a result, you may wish to ignore this part entirely and figure out your own mechanism for getting an environment that looks like the above.

OpenStack

Before we even start thinking about OpenShift, we need our OpenStack platform to run things on. You may well have an OpenStack platform already, in which case you can skip this section entirely. Similarly, you may have a preferred tool or workflow for deploying OpenStack clusters, meaning again you can skip this section entirely and do your own thing. The important thing is that whatever environment you have must have enough capacity to run 4 small instances for BGP routing (a flavour with 1GB RAM, 1 vCPU, and 10GB disk will suffice) and 4 much larger instances for OpenShift nodes (16GB RAM, 4 vCPU and 40GB disk minimum).

In my case, I was deploying OpenStack from scratch onto a single beefy bare metal machine (A Dell PowerEdge R640 with 2 Intel Xeon Silver 4216 CPUs, 192 GB RAM and 512GB SSD + 2TB HDD). To do so, I used github.com/shiftstack/dev-install, an opinionated wrapper around TripleO Standalone designed for hosting OpenShift clusters, to deploy an OSP 16.2 cloud (OpenStack Train). dev-install is available on GitHub:

❯ git clone https://github.com/shiftstack/dev-install
❯ cd dev-install

Using dev-install is relatively easy: all it needs is a local_overrides.yaml file that contains information about the IP address and hostname of the server, the OSP release you wish to use, and any other overrides necessary. I didn’t need anything special so there were no overrides in my case. This meant my local_overrides.yaml file looked something like this:

hostname: acme
public_api: 10.1.240.35
standalone_host: acme.shiftstack.test
rhos_release: 16.2
clouddomain: shiftstack.test

With the local_overrides.yaml file in place, you can kick off installation:

❯ make config host=acme.shiftstack.test
❯ make osp_full overrides=local_overrides.yaml

Installation should take about 30 minutes and two new entries, standalone and standalone_openshift, will be added to your clouds.yaml file upon completion.

Once installation is completed, there is one final step necessary: starting sshuttle. dev-install creates a hostonly network that is not routable outside the host. As a result, you need something to proxy requests for the associated hostonly-subnet subnet to the host and dev-install opts to use sshuttle for this. You can start sshuttle using the wrapper script provided:

❯ ./scripts/sshuttle-standalone.sh

BGP

Next, we need to configure a BGP environment. I didn’t have access to some ToR switches with BGP support so we opted to emulate it using frr, a software routing solution. To do this, I used github.com/shiftstack/poc-bgp, a set of Ansible playbooks which deployed four instances on my new OpenStack cloud: a spine router instance and three leaf router instances. Once again, there are other software routing solutions available and you might even have access to real hardware, so if you opt for another approach you could obviously skip this section (though if you do, you will likely need to modify your MetalLB configuration when we get to that step in the next post). Assuming you opt to use poc-bgp, you’ll need a CentOS 9 Stream image to exist on the cloud as well as a key pair so that you can SSH into the instances it creates. You can create the image like so:

❯ wget https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20230417.0.x86_64.qcow2
❯ openstack image create --public --disk-format qcow2 --file CentOS-Stream-GenericCloud-9-20230417.0.x86_64.qcow2 centos9-stream

Similarly, you can create the key pair like so:

❯ openstack keypair create --public-key ~/.ssh/id_ed25519.pub default

Once these resources exists, configuration and deployment is otherwise relatively uncomplex.

❯ git clone https://github.com/shiftstack/poc-bgp
❯ cd poc-bgp
❯ cat << EOF > localvars.yaml
cloud_name: standalone_openshift
image: centos9-stream
instance_name: stephenfin-poc-bgp
external_network: hostonly
keypair: default
dns_nameservers: 1.1.1.1
flavor: m1.tiny
EOF
❯ make deploy

Once deployed you will end up with 4 instances and 7 networks (6 plus the hostonly network):

❯ openstack server list
+--------------------------------------+----------------------------------+--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+
| ID                                   | Name                             | Status | Networks                                                                                                                                                                   | Image          | Flavor    |
+--------------------------------------+----------------------------------+--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+
| a3375961-0821-4135-b064-b884460d11c2 | stephenfin-poc-bgp-spine-gateway | ACTIVE | hostonly=192.168.25.191, 2001:db8::100; stephenfin-poc-bgp-rack1-patch=192.168.0.1; stephenfin-poc-bgp-rack2-patch=192.168.0.5; stephenfin-poc-bgp-rack3-patch=192.168.0.9 | centos9-stream | m1.tiny   |
| f1e3b324-bf60-440e-808a-3e548f571f1a | stephenfin-poc-bgp-rack1-gateway | ACTIVE | stephenfin-poc-bgp-rack1-leaf=192.168.10.1; stephenfin-poc-bgp-rack1-patch=192.168.0.2                                                                                     | centos9-stream | m1.tiny   |
| 2b6051cb-f98a-4112-8096-c5c96c56f05a | stephenfin-poc-bgp-rack2-gateway | ACTIVE | stephenfin-poc-bgp-rack2-leaf=192.168.20.1; stephenfin-poc-bgp-rack2-patch=192.168.0.6                                                                                     | centos9-stream | m1.tiny   |
| 2d14cda5-377a-4383-b6e8-feb50455fad0 | stephenfin-poc-bgp-rack3-gateway | ACTIVE | stephenfin-poc-bgp-rack3-leaf=192.168.30.1; stephenfin-poc-bgp-rack3-patch=192.168.0.10                                                                                    | centos9-stream | m1.tiny   |
+--------------------------------------+----------------------------------+--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+-----------+

❯ openstack network list
+--------------------------------------+--------------------------------+----------------------------------------------------------------------------+
| ID                                   | Name                           | Subnets                                                                    |
+--------------------------------------+--------------------------------+----------------------------------------------------------------------------+
| 19708577-3757-4956-903e-9656229e1286 | stephenfin-poc-bgp-rack1-leaf  | 6152b856-781b-4fc9-9979-e4cfc9e282b8                                       |
| 2a28f908-e9ec-4fb5-a4cb-09cd5dae8723 | stephenfin-poc-bgp-rack3-patch | 0f15d6ed-0b8a-435e-90d0-83edb2164100                                       |
| 2cd308e3-8370-425b-af1c-0ce5562c2b36 | stephenfin-poc-bgp-rack1-patch | f22c9d7e-9c7f-4985-be7c-63eb05014544                                       |
| 3060e6b0-c2c3-4f59-ac13-9d07a7a963f1 | stephenfin-poc-bgp-rack2-leaf  | 824cfd85-f89b-4a07-a330-3c3adb55b0e3                                       |
| 538a2436-07c5-48f1-8cd0-39931244bcb7 | stephenfin-poc-bgp-rack3-leaf  | 3109bff8-fdd5-4824-ab2d-58184096e35c                                       |
| 91efea92-eaf8-4146-ac4c-48d3edfedfe4 | hostonly                       | 607ba284-3762-4667-a2fa-2a7f31de6f35, b147f959-3122-4ad9-aaaa-6ff1af41c8df |
| b15da053-0625-4d33-ad6a-036dd76cfc8d | stephenfin-poc-bgp-rack2-patch | 71a6eeef-2c3d-45e2-80a9-efff6e15a289                                       |
+--------------------------------------+--------------------------------+----------------------------------------------------------------------------+

The spine-gateway instance is connected to the three rackN-gateway instances via separate patch networks, while the three rackN-gateway instances have their own rackN-leaf leaf network. We’re going to use one of the latter as our machine subnet when installing OpenShift shortly.

We have some final steps to do here. First, we need to configure the subnets created by poc-bgp to keep IP address < .10 free. This is necessary because we are going to use some of these IP addresses for OpenShift VIPs shortly.

❯ openstack subnet set --no-allocation-pool --allocation-pool "start=192.168.10.10,end=192.168.10.239" stephenfin-poc-bgp-rack1-subnet
❯ openstack subnet set --no-allocation-pool --allocation-pool "start=192.168.20.10,end=192.168.20.239" stephenfin-poc-bgp-rack2-subnet
❯ openstack subnet set --no-allocation-pool --allocation-pool "start=192.168.30.10,end=192.168.30.239" stephenfin-poc-bgp-rack3-subnet

In addition, we need another instance of sshuttle to configure traffic for these new networks to route through the spine-gateway host. This will look something like so:

❯ sshuttle -r cloud-user@${GATEWAY_HOST_IP} 192.168.10.0/24 192.168.20.0/24 192.168.30.0/24

where ${GATEWAY_HOST_IP} is the IP of the spine-gateway host (192.168.25.191 in my case).

With our BGP routers in place, we can move onto the final stage of set up: installing OpenShift.

OpenShift

The last step of initial set up is installing OpenShift. I used openshift-install to do this, deploying a 4.12 OpenShift cluster on my OpenStack cloud. When deploying OpenShift, you’ll want to pay special attention to the networking configuration. As noted above, we want to use one of the rackN-leaf networks and I chose rack1-leaf arbitrarily. You should use IPs from whatever subnet you choose for your API and Ingress VIPs, picking address in the < .10 range we previously set aside. This means you should end up with configuration similar to the following in your install-config.yaml:

---
# ...
platform:
  openstack:
    # ...
    machinesSubnet: 6152b856-781b-4fc9-9979-e4cfc9e282b8  # stephenfin-poc-bgp-rack1-subnet
    apiVIPs:
      - 192.168.10.5  # from stephenfin-poc-bgp-rack1-subnet, < .10
    ingressVIPs:
      - 192.168.10.7  # from stephenfin-poc-bgp-rack1-subnet, < .10
networking:
  clusterNetworks:
    - cidr: 10.128.0.0/14
      hostSubnetLength: 9
  serviceCIDR: 172.30.0.0/16
  machineNetwork:
    - cidr: 192.168.10.0/24
    - cidr: 192.168.25.0/24

In addition, assuming you don’t have a local DNS server configured, you will need to modify your /etc/hosts file to specify the hostnames and corresponding IP addresses of your OpenShift cluster. This is described in the OpenShift Installer docs, but in summary you’ll want something like this:

# openshift shiftstack nodes
192.168.10.5 api.stephenfin.openshift.shiftstack.test
192.168.10.7 console-openshift-console.apps.stephenfin.openshift.shiftstack.test
192.168.10.7 integrated-oauth-server-openshift-authentication.apps.stephenfin.openshift.shiftstack.test
192.168.10.7 oauth-openshift.apps.stephenfin.openshift.shiftstack.test
192.168.10.7 prometheus-k8s-openshift-monitoring.apps.stephenfin.openshift.shiftstack.test
192.168.10.7 grafana-openshift-monitoring.apps.stephenfin.openshift.shiftstack.test
# End of openshift nodes

You should obviously change the hostnames and IP addresses to match whatever you configured in your install-config.yaml file.

Once you have these steps completed, you can kick of installation:

openshift-install --log-level debug create cluster

Wait for that to complete, at which point you should have a fully functioning OpenShift deployment that you can interact with using oc or kubectl.

Wrap Up

In this post, we deployed an OpenStack deployment, configured BGP software routers, and deployed OpenShift on OpenStack. In the next post, we will work to install MetalLB itself. Stay tuned.

Appendix: Configuration files

Here are the full configuration files used for my deployment.

dev-install

local-overrides.yaml:

hostname: acme
public_api: 10.1.240.35
standalone_host: acme.shiftstack.test
rhos_release: 16.2
clouddomain: shiftstack.test

poc-bgp

local_vars.yaml:

cloud_name: standalone_openshift
image: centos9-stream
instance_name: stephenfin-poc-bgp
external_network: hostonly
keypair: default
dns_nameservers: 1.1.1.1
flavor: m1.tiny

openshift-installer

install-config.yaml:

---
apiVersion: v1
baseDomain: openshift.shiftstack.test
metadata:
  name: stephenfin
controlPlane:
  name: master
  platform:
    openstack:
      type: m1.xlarge
  replicas: 3
compute:
  - name: worker
    platform:
      openstack:
        type: m1.xlarge
    replicas: 1
platform:
  openstack:
    defaultMachinePlatform:
      type: m1.xlarge
    clusterOSImage: rhcos-4.12
    cloud: standalone_openshift
    machinesSubnet: 6152b856-781b-4fc9-9979-e4cfc9e282b8
    apiVIPs:
      - 192.168.10.5
    ingressVIPs:
      - 192.168.10.7
networking:
  clusterNetworks:
    - cidr: 10.128.0.0/14
      hostSubnetLength: 9
  serviceCIDR: 172.30.0.0/16
  machineNetwork:
    - cidr: 192.168.10.0/24
    - cidr: 192.168.25.0/24
pullSecret: <redacted>

Installing OSC With pipx

Thu, 27 Apr 2023 00:00:00 +0000

Another one that really belongs in a separate “tips & tricks” section of the blog, which unfortunately does not yet exist. I use openstackclient (OSC) pretty extensively day-to-day. Fedora provides a relatively up-to-date version of the package, but OSC has been evolving very rapidly of late and Fedora’s packagers just can’t be that fast 😁. The solution, therefore, is to install from pip. Installing things globally, even if that’s global to the user, is a bad idea though: there are too many opportunities for dependency updates to break other packages. I want to use a virtualenv. The obvious solution is to use pipx, which is specifically designed for this use case. From the homepage:

pipx is a tool to help you install and run end-user applications written in Python. It’s roughly similar to macOS’s brew, JavaScript’s npx, and Linux’s apt.

It’s closely related to pip. In fact, it uses pip, but is focused on installing and managing Python packages that can be run from the command line directly as applications.

pipx itself is available in the Fedora repos so if Fedora is you distro of choice then you can install it with dnf:

❯ sudo dnf install pipx

Once that’s in place, you can install OSC with pipx:

❯ pipx install python-openstackclient

Unlike openstacksdk, however, the python-openstackclient package doesn’t provide support for OpenStack services beyond the core ones: Identity (keystone), Compute (nova), Image (glance), Block Storage (cinder), Network (neutron) and Object Storage (swift). For services like the Share Filesystem-as-a-Service (manila), Load Balancer-as-a-Service (octavia) or Placement service, you need to install additional packages (python-manilaclient, python-octaviaclient, and osc-placement, respectively).

❯ openstack loadbalancer list
openstack: 'loadbalancer list' is not an openstack command. See 'openstack --help'.
Did you mean one of these?
  container create
  container delete
  container list
  container save
  container set
  container show
  container unset

It took me a beat to figure out but pipx provides an easy mechanism to do this: the inject command. For example, to install python-octaviaclient in the same virtualenv, run:

❯ pipx inject python-openstackclient python-octaviaclient

You can repeat this for additional dependencies:

❯ pipx inject python-openstackclient osc-placement python-neutronclient python-manilaclient python-ironicclient python-barbicanclient python-designateclient

❯ pipx list --include-injected
venvs are in /home/stephenfin/.local/pipx/venvs
apps are exposed on your $PATH at /home/stephenfin/.local/bin
   package python-openstackclient 6.2.0, installed using Python 3.11.2
    - openstack
    Injected Packages:
      - osc-placement 4.1.0
      - python-barbicanclient 5.5.0
      - python-designateclient 5.2.0
      - python-ironicclient 5.1.0
      - python-manilaclient 4.3.0
      - python-neutronclient 9.0.0
      - python-octaviaclient 3.4.0

With that done, you should have access to all the commands. 💪

Playing with OpenStack Nova APIs from a Python Shell

Wed, 22 Mar 2023 00:00:00 +0000

A quick hack. If you want to call various OpenStack Nova APIs outside the context of a service like nova-compute, there’s a good chance you’ll need some configuration. If you have deployment, you’ll likely have a configuration file with all the necessary details. DevStack places its Nova configuration files at /etc/nova. Nova provides APIs, which are tiny wrappers around oslo.db functionality, for loading these.

>>> import nova.conf
>>> from nova import config
>>> config.parse_args('foo --config-file /etc/nova/nova.conf'.split())
>>> CONF = nova.conf.CONF

With you configuration, you can now do things that require this configuration. For example, let’s look at our databases using SQLAlchemy.

>>> from nova.db.main import api
>>> engine = api.get_engine()
>>> import sqlalchemy
>>> sqlalchemy.inspect(engine).has_table('alembic_version')
True
>>> sqlalchemy.inspect(engine).has_table('migrate_version')
False

This can be handy to get you out of a hole, or to hack on stuff in a development environment.

Hello, Gophercloud

Thu, 16 Feb 2023 00:00:00 +0000

I’ve been playing around with Gophercloud over the past few days. While it’s nowhere near as complete as openstacksdk (which, coincidentally, recently hit the big 1.0.0 milestone 🎉) or legacy Python-based service clients yet, it is clearly a great tool to have in your arsenal and is the library underlying the Terraform OpenStack Provider, OpenStack platform support in openshift-install, and various OpenStack-specific Kubernetes components, among others.

In the interests of easing ramp-up for anyone else trying to get started with Gophercloud, here’s an simple “hello, world” example - listing servers - using both Gophercloud and openstacksdk.

Gophercloud (Go)

Create a new directory containing a single file, main.go, and add the following to it.

package main

import (
	"context"
	"fmt"

	"github.com/gophercloud/gophercloud/v2/openstack/compute/v2/servers"
	"github.com/gophercloud/utils/v2/openstack/clientconfig"
)

func main() {
	opts := new(clientconfig.ClientOpts)
	// We could configure the cloud manually but we don't. Instead we'll leave it unset causing gophercloud to load the
	// cloud from the 'OS_CLOUD' environment variable. We could also simplify this further and pass 'nil' below instead
	// of generating and passing a 'ClientOpts' object but this shows how you _could_ configure things if you so chose.
	// opts.Cloud = "devstack-admin"

	client, err := clientconfig.NewServiceClient(context.TODO(), "compute", opts)
	if err != nil {
		fmt.Println(err)
		return
	}

	pager, err := servers.List(client, nil).AllPages(context.TODO())
	if err != nil {
		fmt.Println(err)
		return
	}

	servers, err := servers.ExtractServers(pager)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println("servers:")
	for i, server := range servers {
		fmt.Printf("  server %d: id=%s\n", i, server.ID)
	}
}

Now generate the go.mod file.

❯ go mod init gophercloud-hello-world
❯ go mod tidy

Finally, you can run the command.

❯ export OS_CLOUD=devstack-admin  # or another entry from your local 'clouds.yaml'
❯ go run .

This should dump something like the following.

servers:
  server 0: id=395b60b4-73be-4ba4-a8a8-fdaa3fce57d5

This example also makes use of the gophercloud/utils library, to simplify authentication and the creation of clients. gophercloud itself also provides support for clouds.yaml files, but its slightly more verbose. If you wanted to skip utils, you could rewrite this like so:

package main

import (
	"context"
	"fmt"

	"github.com/gophercloud/gophercloud/v2"
	"github.com/gophercloud/gophercloud/v2/openstack"
	"github.com/gophercloud/gophercloud/v2/openstack/config"
	"github.com/gophercloud/gophercloud/v2/openstack/config/clouds"
	"github.com/gophercloud/gophercloud/v2/openstack/compute/v2/servers"
)

func main() {
	ctx := context.TODO()

	ao, eo, tlsConfig, err := clouds.Parse()
	if err != nil {
		fmt.Println(err)
		return
	}

	providerClient, err := config.NewProviderClient(ctx, ao, config.WithTLSConfig(tlsConfig))
	if err != nil {
		fmt.Println(err)
		return
	}

	client, err := openstack.NewComputeV2(ctx, providerClient, eo)

	// rest of code here ...
}

If you are working on Gophercloud itself, it can be helpful to use scripts like this to test your changes. If doing this, you can rely on the replace directive in go.mod to ensure your local clone of Gophercloud is used.

replace (
	github.com/gophercloud/gophercloud/v2 => /home/user/code/gophercloud
	github.com/gophercloud/utils/v2 => /home/user/code/utils
)

More information here.

openstacksdk (Python)

Create a new directory containing a single file, main.py, and add the following to it.

import openstack

def main():
    # We could configure the cloud manually but we don't. Instead we'll leave it unset causing openstacksdk to load the
    # cloud from the 'OS_CLOUD' environment variable.
    # conn = openstack.connect(cloud='devstack-admin')
    conn = openstack.connect()

    servers = conn.compute.servers()

    print('servers:')
    for i, server in enumerate(servers):
        print(f'  server {i}: id={server.id}')

if __name__ == '__main__':
    main()

Now create a virtual environment and install the dependencies.

❯ virtualenv .venv
❯ source .venv/bin/activate
❯ pip install openstacksdk

Finally, you can run the command.

❯ export OS_CLOUD=devstack-admin  # or another entry from your local 'clouds.yaml'
❯ python main.py

This should dump identical output to the Go-based script.

servers:
  server 0: id=395b60b4-73be-4ba4-a8a8-fdaa3fce57d5

API Versioning in Openstack

Fri, 03 Feb 2023 00:00:00 +0000

The API versioning schemes in use in OpenStack vary between services and have evolved since the early days of OpenStack. There are two types of API versioning to consider: the major version and the minor version. Today, most OpenStack services have settled on a single major API version and have chosen to evolve the API without bumping the major API version any further. There are three API minor versioning schemes in common use.

Per-release versions

This is used by the Image service (glance) and DNS service (designate). All changes to the API during a given release cycle are gathered into a single new API version. As such, the API version will increase at most once per release. You can continue to request older versions.

For example, consider the Image (glance) API.

Release	Supported 2.x API versions
Grizzly	2.0 - 2.1
Havana	2.0 - 2.2
Kilo	2.0 - 2.3
…	…

In theory, this means the API is purely additive and the new version can be used as a signal that an API addition is available. Glance takes this a step further and includes or excludes API versions depending on whether an optional feature is enabled or not.

Single version

This is used by the Identity service (keystone). It is similar to the per-release versions but unlike those, only a single API version is exposed. Like those, however, this version can be used as signal that a feature is available. Once again, this API schema is really intended to be purely additive, since there’s no way to opt into old behavior. Keystone, for example, has never removed APIs or made non-additive changes like changing the type of a field or removing it.

Microversions

This is used by multiple services including the Compute service (nova), Block Storage service (cinder), and Shared Filesystem service (manila). Each change to the API will result in a new API version. As such, the API version can increase multiple times per release. You can continue to request older versions.

For example, consider the Compute (nova) API.

Release	Supported 2.x API versions
Kilo	2.1 - 2.3
Liberty	2.1 - 2.12
Mitaka	2.1 - 2.25
…	…

Unlike the previous examples, this API versioning schema allows for practically any change, including changing the type of a field, adding and removing new fields, and adding or removing new APIS. If you want the old behavior, you simply request an older microversion.

Extensions

This is used by the Networking service (neutron). It’s a versioning scheme that doesn’t use API versions. Instead, it exposes a list of available extensions. An extension can add, remove or modify features and vendor-specific functionality to the API. This can include API resources/routes as well as new fields in API requests and responses. If you want to depend on a feature added by an extension, you should check if the extension is present.

Zero-downtime Upgrades With Alembic and SQLAlchemy

Sun, 13 Nov 2022 00:00:00 +0000

This is the blog form of a talk I delivered at PyCon Ireland 2022. You can find the slides here and the source code here.

One thing to note before we begin is that while I mention OpenStack services below, this article isn’t actually about OpenStack 😉 We’re merely using it as a good example. The problems that OpenStack needs to address such as scaling out services in larger deployments are common to most distributed services.

Background

In the beginning, there were only two OpenStack services: Swift, which was responsible for object-storage, and Nova, which handled everything else. Over time, the list of services has grown enormously as Nova has shed some of its former responsibilities to other services and the amounts of things an OpenStack cloud can do has grown. However, due to many projects’ shared heritage (where they were split out from or modeled on Nova) and general design decisions made in the early days of OpenStack, even the newest OpenStack services tend to have a couple of commonalities in their design. Two of these are relevant to this article. Firstly, many of the services are designed to be horizontally scalable and almost always use a message broker to achieve this. This message broker is typically RabbitMQ and it provides a way for multiple instances of a service to communicate with each other. Secondly, without fail, OpenStack services will use a traditional relational DB to store data. The database backend is usually MySQL (although PostgreSQL is also an option) while SQLAlchemy is used for them ORM.

These two design decisions are relevant because they contain an inherent conflict. There can be many instances of a service (e.g. nova-conductor or nova-api) but they will all share a single database. If the schema of the database changes then each of these services will need to understand the new format. Historically in OpenStack, the way to do this was to closely couple the source code and database schema. An upgrade to a new version of OpenStack would necessitate taking all services down, upgrading the service code and the database schema, and then restarting the services. It works but it requires potentially large API downtime: the table lock that MySQL grabs while migrating the schema will block all reads and writes until the migration is complete, and for large installations where tables can have many millions of rows that migration can take minutes if not hours. This isn’t good enough for most production enterprise clouds, let alone specific applications like Telco where SLAs are something that the government cares about. What we want is the ability to do rolling upgrades. Enter expand-contract.

The Expand-Migrate-Contract pattern

We can start exploring the expand-contract pattern by thinking about what happens to the data in an application during a migration.

Data can be added. For example, adding a new name field to a user account model.
Data can change. For example, combining separate existing first_name and last_name fields into a single name field.
Data can be deleted. For example, dropping the old first_name and last_name fields.

The basic idea behind the expand-migrate-contract migration pattern is to split the schema migrations into two groups: additive changes, which will handle the added data, and contractive/destructive changes, which will handle the removed data. The changed data is an example of a data migration and either the application or another set of migrations handles this. If you’re a visual person, you can visualize this like so:

A visualization of expand-migrate-contract

By applying the expand-contract migration, we can move from an upgrade pattern like this:

Stop all services
Upgrade the services and apply all database schema migrations.
Wait X minutes/hours/days for the schema migration to complete.
Restart all services

To this:

Apply the “expand” database schema migration
Upgrade the service instances one-by-one until all services have been upgraded
(Optional) Apply database data migrations
(At some future point) Apply the “contract” database schema migrations

There are often a couple of additional steps that your application will need to do, but the basic idea here is that by adapting the expand-migrate-contract pattern in your application, you can both allow your database schema to evolve while avoiding undesirable downtime.

Now that we have all this background out of the way, where do Alembic and SQLAlchemy fit into this equation? I’m glad you asked!

Initial steps

So before we start talking about alternative database upgrade plans, we need to have a platform to build on. To this end, let’s create an initial project structure and populate it with some initial SQLAlchemy models. I’ve based this on a slightly modified version of the sample models found in the SQLAlchemy quick start guide.

❯ mkdir -p alembic-expand-contract-demo/alembic_expand_contract_demo
❯ cd alembic-expand-contract-demo
❯ touch alembic_expand_contract_demo/__init__.py alembic_expand_contract_demo/models.py alembic_expand_contract_demo/README.md

Dump the following into the newly created alembic_expand_contract_demo/models.py file:

from sqlalchemy import Column
from sqlalchemy import ForeignKey
from sqlalchemy import Integer
from sqlalchemy import String
from sqlalchemy.orm import declarative_base
from sqlalchemy.orm import relationship

Base = declarative_base()


class User(Base):
    __tablename__ = "user_account"
    id = Column(Integer, primary_key=True)
    first_name = Column(String(30))
    last_name = Column(String(30))
    addresses = relationship(
        "Address", back_populates="user", cascade="all, delete-orphan"
    )

    def __repr__(self):
        return f"User(id={self.id!r})"


class Address(Base):
    __tablename__ = "address"
    id = Column(Integer, primary_key=True)
    email_address = Column(String, nullable=False)
    user_id = Column(Integer, ForeignKey("user_account.id"), nullable=False)
    user = relationship("User", back_populates="addresses")

    def __repr__(self):
        return f"Address(id={self.id!r})"

We’ll also create our requirements.txt file and install those requirements.

❯ cat > requirements.txt << EOF
sqlalchemy
alembic
EOF
❯(.venv) virtualenv .venv
❯(.venv) source .venv/bin/activate
❯(.venv) pip install -r requirements.txt

With our initial models in place, we can now proceed to creating the initial migration directory structure.

❯(.venv) alembic init alembic_expand_contract_demo/migrations

You should now have a directory structure like so:

.
├── alembic_expand_contract_demo
│   ├── __init__.py
│   ├── migrations
│   │   ├── env.py
│   │   ├── README
│   │   ├── script.py.mako
│   │   └── versions
│   ├── models.py
├── alembic.ini
├── README.md
└── requirements.txt

As suggested by the wizard, you’ll want to make a few changes to the alembic.ini file created in the root of your directory. In particular, you’ll want to configure the [alembic] script_location and [alembic] sqlalchemy.url settings. I used the following:

[alembic]
script_location = %(here)s/alembic_expand_contract_demo/migrations
sqlalchemy.url = sqlite:///alembic-expand-contract-demo.db

You’ll also need to configure the env.py file found in the new alembic_expand_contract_demo/migrations directory to point the target_metadata value to your base model. For example:

from alembic_expand_contract_demo import models

# ...

target_metadata = models.Base.metadata

Finally, we’ll modify the migration script template script.py.mako in the alembic_expand_contract_demo/migrations directory to remove the downgrade step: downgrades aren’t all that useful in a production environment (back up your data!) and add needless complexity here:

--- alembic_expand_contract_demo/migrations/script.py.mako
+++ alembic_expand_contract_demo/migrations/script.py.mako
@@ -2,9 +2,8 @@
 
 Revision ID: ${up_revision}
 Revises: ${down_revision | comma,n}
-Create Date: ${create_date}
-
 """
+
 from alembic import op
 import sqlalchemy as sa
 ${imports if imports else ""}
@@ -18,7 +17,3 @@ depends_on = ${repr(depends_on)}
 
 def upgrade() -> None:
     ${upgrades if upgrades else "pass"}
-
-
-def downgrade() -> None:
-    ${downgrades if downgrades else "pass"}

With all this in place, we can now create an initial empty database and generate our first migration (Alembic needs a database to compare against to generate the updated schema).

❯ alembic upgrade head

Once this is run, you’ll note a new alembic-expand-contract-demo.db SQLite database in your root directory. A quick look at the schema of this database shows an effectively empty table with only Alembic’s own alembic_version table present. This is expected: while we have models in place, we don’t yet have a Alembic migration to add them.

❯ sqlite3 alembic-expand-contract-demo.db '.dump'
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE alembic_version (
        version_num VARCHAR(32) NOT NULL,
        CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num)
);
COMMIT;

Since we’d already wired everything up, we can now simply run the revision sub-command with the --autogenerate option and our migrations will be auto-generated:

❯ alembic revision --autogenerate --message='Initial models'
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added table 'user_account'
INFO  [alembic.autogenerate.compare] Detected added table 'address'
  Generating /home/stephenfin/demos/alembic-expand-contract-demo/alembic_expand_contract_demo/migrations/versions/a185a7b16d70_initial_models.py ...  done

This generates a migration that will look something like the following, once comments are addressed and formatting cleaned up:

"""Initial models

Revision ID: 6cb93d555e2b
Revises:
"""

from alembic import op
import sqlalchemy as sa

# revision identifiers, used by Alembic.
revision = '6cb93d555e2b'
down_revision = None
branch_labels = None
depends_on = None


def upgrade() -> None:
    op.create_table(
        'user_account',
        sa.Column('id', sa.Integer(), nullable=False),
        sa.Column('first_name', sa.String(length=30), nullable=True),
        sa.Column('last_name', sa.String(length=30), nullable=True),
        sa.PrimaryKeyConstraint('id'),
    )
    op.create_table(
        'address',
        sa.Column('id', sa.Integer(), nullable=False),
        sa.Column('email_address', sa.String(), nullable=False),
        sa.Column('user_id', sa.Integer(), nullable=False),
        sa.ForeignKeyConstraint(
            ['user_id'],
            ['user_account.id'],
        ),
        sa.PrimaryKeyConstraint('id'),
    )


def downgrade() -> None:
    op.drop_table('address')
    op.drop_table('user_account')

Apply this and we should see the new tables in the database schema.

❯ alembic upgrade head
❯ sqlite3 alembic-expand-contract-demo.db '.dump'
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE alembic_version (
        version_num VARCHAR(32) NOT NULL,
        CONSTRAINT alembic_version_pkc PRIMARY KEY (version_num)
);
INSERT INTO alembic_version VALUES('6cb93d555e2b');
CREATE TABLE user_account (
        id INTEGER NOT NULL,
        first_name VARCHAR(30),
        last_name VARCHAR(30),
        PRIMARY KEY (id)
);
CREATE TABLE address (
        id INTEGER NOT NULL,
        email_address VARCHAR NOT NULL,
        user_id INTEGER NOT NULL,
        PRIMARY KEY (id),
        FOREIGN KEY(user_id) REFERENCES user_account (id)
);
COMMIT;

Nothing complicated so far. Onto the meaty part.

First pass at expand-contract using Alembic

We now have an initial application. Time to start thinking about how to upgrade it. Looking at our models, we’ll note that we have separate first_name and last_name fields.

class User(Base):
    __tablename__ = "user_account"
    id = Column(Integer, primary_key=True)
    first_name = Column(String(30))
    last_name = Column(String(30))
    addresses = relationship(
        "Address", back_populates="user", cascade="all, delete-orphan"
    )

    def __repr__(self):
        return f"User(id={self.id!r})"

Let’s say we wanted to merge these two fields into a single combined field called name. For example:

@@ -17,8 +17,7 @@ Base = declarative_base()
 class User(Base):
     __tablename__ = "user_account"
     id = Column(Integer, primary_key=True)
-    first_name = Column(String(30))
-    last_name = Column(String(30))
+    name = Column(String)
     addresses = relationship(
         "Address", back_populates="user", cascade="all, delete-orphan"
     )

We can auto-generate our migration once again:

❯ alembic revision --message='Merge user names'

This will give us a migration like so:

"""Merge user names

Revision ID: 0585005489f0
Revises: 6cb93d555e2b
"""

from alembic import op
import sqlalchemy as sa


# revision identifiers, used by Alembic.
revision = '0585005489f0'
down_revision = '6cb93d555e2b'
branch_labels = None
depends_on = None


def upgrade() -> None:
    op.add_column(
        'user_account', sa.Column('name', sa.String(), nullable=True)
    )
    op.drop_column('user_account', 'last_name')
    op.drop_column('user_account', 'first_name')


def downgrade() -> None:
    op.add_column(
        'user_account',
        sa.Column('first_name', sa.VARCHAR(length=30), nullable=True),
    )
    op.add_column(
        'user_account',
        sa.Column('last_name', sa.VARCHAR(length=30), nullable=True),
    )
    op.drop_column('user_account', 'name')

Clearly this isn’t very useful as it would result in data loss: we need to migrate data before we can remove the old column. The easy way to address this would be to modify the migration to also migrate data as an intermediary step. For example:

"""Merge user names

Revision ID: 0585005489f0
Revises: 6cb93d555e2b
"""

from alembic import op
import sqlalchemy as sa

# revision identifiers, used by Alembic.
revision = '0585005489f0'
down_revision = '6cb93d555e2b'
branch_labels = None
depends_on = None


def upgrade() -> None:
    # schema migrations - expand
    op.add_column(
        'user_account',
        sa.Column('name', sa.String(), nullable=True),
    )

    # data migrations
    bind = op.get_bind()

    user_account_table = sa.Table(
        'user_account',
        sa.MetaData(),
        sa.Column('id', sa.Integer(), nullable=False),
        sa.Column('first_name', sa.String(length=30), nullable=True),
        sa.Column('last_name', sa.String(length=30), nullable=True),
        sa.Column('name', sa.String, nullable=True),
    )

    names = bind.execute(
        sa.select(
            [
                user_account_table.c.id,
                user_account_table.c.first_name,
                user_account_table.c.last_name,
            ]
        )
    ).fetchall()
    for id, first_name, last_name in names:
        bind.execute(
            user_account_table.update()
            .where(user_account_table.c.id == id)
            .values(
                name=f'{first_name} {last_name}',
            )
        )

    # schema migrations - contract
    op.drop_column('user_account', 'first_name')
    op.drop_column('user_account', 'last_name')


def downgrade() -> None:
    raise Exception("Irreversible migration")

While this works, as noted at the outset this approach doesn’t work for scaled applications or applications with uptime guarantees: your applications would need to be taken offline while this database schema was applied, which itself is an operation that could take some time. Let’s try an expand-contract approach instead. First, we’ll do this manually. Instead of having one large migration file, we’ll create three separate migration files (or two, if your application is going to take care of data migrations at runtime).

❯ alembic revision --message='Merge user names (expand)'
❯ alembic revision --message='Merge user names (migrate)'
❯ alembic revision --message='Merge user names (contract)'

In the “expand” migration, we add the new column:

"""Merge user names (expand)

Revision ID: 9c36df1b3f62
Revises: 6cb93d555e2b
"""

from alembic import op
import sqlalchemy as sa

# revision identifiers, used by Alembic.
revision = '9c36df1b3f62'
down_revision = '6cb93d555e2b'
branch_labels = None
depends_on = None


def upgrade() -> None:
    op.add_column(
        'user_account',
        sa.Column('name', sa.String(), nullable=True),
    )

In the “migrate” migration, we migrate our data:

"""Merge user names (migrate)

Revision ID: c9afc30a70ee
Revises: 9c36df1b3f62
"""

from alembic import op
import sqlalchemy as sa

# revision identifiers, used by Alembic.
revision = 'c9afc30a70ee'
down_revision = '9c36df1b3f62'
branch_labels = None
depends_on = None


def upgrade() -> None:
    # data migrations
    bind = op.get_bind()

    user_account_table = sa.Table(
        'user_account',
        sa.MetaData(),
        sa.Column('id', sa.Integer(), nullable=False),
        sa.Column('first_name', sa.String(length=30), nullable=True),
        sa.Column('last_name', sa.String(length=30), nullable=True),
        sa.Column('name', sa.String, nullable=True),
    )

    names = bind.execute(
        sa.select(
            [
                user_account_table.c.id,
                user_account_table.c.first_name,
                user_account_table.c.last_name,
            ]
        )
    ).fetchall()
    for id, first_name, last_name in names:
        bind.execute(
            user_account_table.update()
            .where(user_account_table.c.id == id)
            .values(
                name=f'{first_name} {last_name}',
            )
        )

Finally, in the “contract” migration, we remove the old columns:

"""Merge user names (contract)

Revision ID: e629a4ea0677
Revises: c9afc30a70ee
"""

from alembic import op

# revision identifiers, used by Alembic.
revision = 'e629a4ea0677'
down_revision = 'c9afc30a70ee'
branch_labels = None
depends_on = None


def upgrade() -> None:
    # schema migrations - contract
    op.drop_column('user_account', 'first_name')
    op.drop_column('user_account', 'last_name')

This is better. However, it does mean you can’t simply run alembic upgrade head as this would apply all migrations in one go, totally defeating the point. What we want is a way to only apply the expand migrations initially, waiting to apply the other migrations later once all our services have been updated. This leads us to…

Branches

Per the Alembic docs:

A branch describes a point in a migration stream when two or more versions refer to the same parent migration as their ancestor.

They’re intended for scenarios where “two divergent source trees, both containing Alembic revision files created independently within those source trees, are merged together into one”. This could includes things like feature branches, where multiple developers are working on separate features and merge everything back in around the same time. However, we can also use them to separate our expand migrations from our contract migrations. Let’s look at how we’d do that. First the expand migration:

--- alembic_expand_contract_demo/migrations/versions/9c36df1b3f62_merge_user_names_expand.py
+++ alembic_expand_contract_demo/migrations/versions/9c36df1b3f62_merge_user_names_expand.py
@@ -11,7 +11,7 @@ import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = '9c36df1b3f62'
 down_revision = '6cb93d555e2b'
-branch_labels = None
+branch_labels = ('expand',)
 depends_on = None

All that we’ve done here is added a new expand branch label. Nothing odd there. Next up, the expand migration. As noted above, we’re actually going to skip this going forward since it’s not totally relevant, so let’s delete it and create a separate tool to manage the migrations.

❯ rm alembic_expand_contract_demo/migrations/versions/c9afc30a70ee_merge_user_names_migrate.py
❯ touch data-migration.py

# SPDX-License-Identifier: MIT

import sqlalchemy as sa


def migrate_data():
    engine = sa.create_engine('sqlite:///alembic-expand-contract-demo.db')

    user_account_table = sa.Table(
        'user_account',
        sa.MetaData(),
        sa.Column('id', sa.Integer(), nullable=False),
        sa.Column('first_name', sa.String(length=30), nullable=True),
        sa.Column('last_name', sa.String(length=30), nullable=True),
        sa.Column('name', sa.String, nullable=True),
    )

    with engine.connect() as conn:
        names = conn.execute(
            sa.select(
                [
                    user_account_table.c.id,
                    user_account_table.c.first_name,
                    user_account_table.c.last_name,
                ]
            )
        ).fetchall()
        for id, first_name, last_name in names:
            conn.execute(
                user_account_table.update()
                .where(user_account_table.c.id == id)
                .values(
                    name=f'{first_name} {last_name}',
                )
            )


if __name__ == '__main__':
    migrate_data()

Finally, the contract migration:

--- alembic_expand_contract_demo/migrations/versions/e629a4ea0677_merge_user_names_contract.py
+++ alembic_expand_contract_demo/migrations/versions/e629a4ea0677_merge_user_names_contract.py
@@ -9,8 +9,8 @@ from alembic import op

 # revision identifiers, used by Alembic.
 revision = 'e629a4ea0677'
-down_revision = 'c9afc30a70ee'
-branch_labels = None
+down_revision = '9c36df1b3f62'
+branch_labels = ('contract',)
 depends_on = None

Again, nothing too odd here. We add the contract branch label and change the down revision to reflect the fact that we removed the data migration.

With these changes in place, we now have the ability to apply the branches separately.

❯ alembic upgrade expand
❯ python data-migration.py
❯ alembic upgrade contract

If you wanted to add additional migrations to either branch, you can use the --branch-label and --depends-on options to set the branch and dependency migration, respectively. For example:

❯ alembic revision \
    --branch-label expand \
    --depends-on 9c36df1b3f62 \
    --message "Another expand type change"

This is a further improvement from where we were, but we’re still missing a huge feature: auto-generation. While we can pass the --autogenerate flag alembic revision, the generated migrations need a lot of work including manually splitting the operations into expand and contract branch migrations. It would be great if Alembic could do this for us. Turns out it can, but it needs a bit of work.

Extending auto-generation to support expand-contract

What we need is a hook point to modify the migrations generated by Alembic so that we can separate them into expand migrations and contract migrations. To do this, Alembic provides process_revision_directives attribute to the context.configure API call found in env.py. From the Alembic docs:

process_revision_directives - a callable function that will be passed a structure representing the end result of an autogenerate or plain “revision” operation, which can be manipulated to affect how the alembic revision command ultimately outputs new revision scripts.

Before we configure that attribute, let’s create the callable function itself. To do this, we’re going to make use of the alembic.util.Dispatcher class. This is a class that is normally responsible for generating per-backend SQL statements for each migration operation. Instead of using this to generate SQL statements, we can use it to sift through the operations generated by Alembic so we can split them into two phases. Let’s look at how we can do this in practice. Create an new file, alembic_expand_contract_demo/migrations/autogen.py and dump the following into it. We’ll discuss the contents of the file step-by-step in a bit.

# SPDX-License-Identifier: MIT

from alembic.operations import ops
from alembic.util import Dispatcher
from alembic.util import rev_id as new_rev_id

_ec_dispatcher = Dispatcher()


def process_revision_directives(context, revision, directives):
    directives[:] = list(_assign_directives(context, directives))


def _assign_directives(context, directives, phase=None):
    for directive in directives:
        decider = _ec_dispatcher.dispatch(directive)
        if phase is None:
            phases = ('expand', 'contract')
        else:
            phases = (phase,)
        for phase in phases:
            decided = decider(context, directive, phase)
            if decided:
                yield decided


@_ec_dispatcher.dispatch_for(ops.MigrationScript)
def _migration_script_ops(context, directive, phase):
    op = ops.MigrationScript(
        new_rev_id(),
        ops.UpgradeOps(
            ops=list(
                _assign_directives(context, directive.upgrade_ops.ops, phase)
            )
        ),
        ops.DowngradeOps(ops=[]),
        message=directive.message,
        head=f'{phase}@head',
    )
    if not op.upgrade_ops.is_empty():
        return op


@_ec_dispatcher.dispatch_for(ops.ModifyTableOps)
def _modify_table_ops(context, directive, phase):
    op = ops.ModifyTableOps(
        directive.table_name, 
        ops=list(_assign_directives(context, directive.ops, phase)),
        schema=directive.schema,
    )
    if not op.is_empty():
        return op


@_ec_dispatcher.dispatch_for(ops.AddConstraintOp)
@_ec_dispatcher.dispatch_for(ops.CreateIndexOp)
@_ec_dispatcher.dispatch_for(ops.CreateTableOp)
@_ec_dispatcher.dispatch_for(ops.AddColumnOp)
def _expands(context, directive, phase):
    if phase == 'expand':
        return directive
    else:
        return None


@_ec_dispatcher.dispatch_for(ops.DropConstraintOp)
@_ec_dispatcher.dispatch_for(ops.DropIndexOp)
@_ec_dispatcher.dispatch_for(ops.DropTableOp)
@_ec_dispatcher.dispatch_for(ops.DropColumnOp)
def _contracts(context, directive, phase):
    if phase == 'contract':
        return directive
    else:
        return None


@_ec_dispatcher.dispatch_for(ops.AlterColumnOp)
def _alter_column(context, directive, phase):
    is_expand = phase == 'expand'
    if is_expand and directive.modify_nullable is True:
        return directive
    elif not is_expand and directive.modify_nullable is False:
        return directive
    else:
        raise NotImplementedError(
            "Don't know if operation is an expand or contract at the moment: "
            "%s" % directive
        )

With that file in place, let’s take a look at what it’s doing. First, the process_revision_directives and _assign_directives functions:

_ec_dispatcher = Dispatcher()


def process_revision_directives(context, revision, directives):
    directives[:] = list(_assign_directives(context, directives))


def _assign_directives(context, directives, phase=None):
    for directive in directives:
        decider = _ec_dispatcher.dispatch(directive)
        if phase is None:
            phases = ('expand', 'contract')
        else:
            phases = (phase,)
        for phase in phases:
            decided = decider(context, directive, phase)
            if decided:
                yield decided

Hopefully this should be mostly self-explanatory. process_revision_directives is initial “entry point” and all it does is overwrites the provided list of directives with our modified set. _assign_directives, on the other hand, does the bulk heavy lifting. It’s responsible for iterating through the generated directives calling the dispatcher for each phase that we’re examining.

Next, we look to look at the dispatchers themselves. For each type of operation, we need to write a separate dispatcher. The top-level operation is the generation of the migration script itself a.k.a. the MigrationScript operation. Let’s look at this first.

@_ec_dispatcher.dispatch_for(ops.MigrationScript)
def _migration_script_ops(context, directive, phase):
    op = ops.MigrationScript(
        new_rev_id(),
        ops.UpgradeOps(
            ops=list(
                _assign_directives(context, directive.upgrade_ops.ops, phase)
            )
        ),
        ops.DowngradeOps(ops=[]),
        message=directive.message,
        head=f'{phase}@head',
    )
    if not op.upgrade_ops.is_empty():
        return op

Again, hopefully there’s nothing too complicated. This dispatcher generates a migration script container a filtered list of upgrade directives (and no downgrade directives). This filtered list is filtered by phase, and we also specify a head that corresponds to the phase (expand@head for the expand phase).

Moving on, in addition to the MigrationScript operation, we have one other “meta” operation to concern ourselves with: ModifyTableOps. Fortunately, this isn’t too complicated. We basically ensure that all operations against a table are filtered by phase as we’d expect:

@_ec_dispatcher.dispatch_for(ops.ModifyTableOps)
def _modify_table_ops(context, directive, phase):
    op = ops.ModifyTableOps(
        directive.table_name, 
        ops=list(_assign_directives(context, directive.ops, phase)),
        schema=directive.schema,
    )
    if not op.is_empty():
        return op

Now for the first “real” operations: the expand operations. These are AddConstraintOp, CreateIndexOp, CreateTableOp, and AddColumnOp. Let’s look at the dispatcher for these.

@_ec_dispatcher.dispatch_for(ops.AddConstraintOp)
@_ec_dispatcher.dispatch_for(ops.CreateIndexOp)
@_ec_dispatcher.dispatch_for(ops.CreateTableOp)
@_ec_dispatcher.dispatch_for(ops.AddColumnOp)
def _expands(context, directive, phase):
    if phase == 'expand':
        return directive
    else:
        return None

Simply put, if we’re in expand phase and encounter one of these operations, proceed as normal. If we’re in the contract phase, skip it. As you’d imagine, the dispatcher for the contract options - DropConstraintOp, DropIndexOp, DropTableOp, and DropColumnOp - works pretty similarly:

@_ec_dispatcher.dispatch_for(ops.DropConstraintOp)
@_ec_dispatcher.dispatch_for(ops.DropIndexOp)
@_ec_dispatcher.dispatch_for(ops.DropTableOp)
@_ec_dispatcher.dispatch_for(ops.DropColumnOp)
def _contracts(context, directive, phase):
    if phase == 'contract':
        return directive
    else:
        return None

With the plain old create and destroy operations out of the way, we only have to take care of the remaining “modify” or “alter” operations. This consists of modifications to columns a.k.a. AlterColumnOp. The AlterColumnOp dispatcher is trickier that the others, so pay attention here:

@_ec_dispatcher.dispatch_for(ops.AlterColumnOp)
def _alter_column(context, directive, phase):
    is_expand = phase == 'expand'
    if is_expand and directive.modify_nullable is True:
        return directive
    elif not is_expand and directive.modify_nullable is False:
        return directive
    else:
        raise NotImplementedError(
            "Don't know if operation is an expand or contract at the moment: "
            "%s" % directive
        )

The only modification we support here is the modifying NULL constraints on columns. There are other modifications you can make, such as modifying the type or a server default but these operations should really involve a bit of manual intervention to handle correctly. What we’ve said here is that if we’re in the expand phase then settings a column to nullable is a-okay. Likewise, if we’re in the contract phase then setting a column to non-nullable is also permissible. All other cases are forbidden though and will raise a NotImplementedError exception.

With this in place, the final step is to enable the processor by passing the process_revision_directives argument to the context.configure API call in env.py:

--- alembic_expand_contract_demo/migrations/env.py
+++ alembic_expand_contract_demo/migrations/env.py
@@ -5,6 +5,7 @@ from sqlalchemy import engine_from_config
 from sqlalchemy import pool

 from alembic_expand_contract_demo import models
+from alembic_expand_contract_demo.migrations import autogen

 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
@@ -68,6 +69,7 @@ def run_migrations_online() -> None:
         context.configure(
             connection=connection,
             target_metadata=target_metadata,
+            process_revision_directives=autogen.process_revision_directives,
         )

         with context.begin_transaction():

Testing things out

Now that we have all of our machinery in place, it’s time to take things for a spin. Let’s test out auto-generation using similar commands to those we used as the beginning of the article.

❯ alembic revision --autogenerate --message='Merge user names'

The first thing we notice is that there are no longer one but two generations generated. This looks promising and looking at the migrations themselves things look even better. First the expand migration:

❯ cat alembic_expand_contract_demo/migrations/versions/1630422c66fe_merge_user_names.py

"""Merge user names

Revision ID: 1630422c66fe
Revises: 9c36df1b3f62
"""

from alembic import op
import sqlalchemy as sa

# revision identifiers, used by Alembic.
revision = '1630422c66fe'
down_revision = '9c36df1b3f62'
branch_labels = None
depends_on = None


def upgrade() -> None:
    op.add_column(
        'user_account', sa.Column('name', sa.String(), nullable=True)
    )

Now the contract migration:

❯ cat alembic_expand_contract_demo/migrations/versions/a142d030afc5_merge_user_names.py

"""Merge user names

Revision ID: a142d030afc5
Revises: e629a4ea0677
"""

from alembic import op

# revision identifiers, used by Alembic.
revision = 'a142d030afc5'
down_revision = 'e629a4ea0677'
branch_labels = None
depends_on = None


def upgrade() -> None:
    op.drop_column('user_account', 'last_name')
    op.drop_column('user_account', 'first_name')

Applying these works exactly as expected:

❯ alembic upgrade expand
❯ alembic upgrade contract

Hurrah, everything worked (hopefully)! However, this is only the first step. While this is a marked improvement on where we were, there are a few improvements we could yet make. For one, the names of the files don’t reveal information about whether a change is an expand or contract migration. You can figure this out by looking at the migrations themselves but it would be much nicer if we could somehow indicate this through the filenames or, better yet, by separating them into separate folders. The branches here are also long-running and it would be nice if we could apply, say, all expand migrations up to a specific release along with any contract migrations from the previous release (which should be okay to apply assuming you’ve upgraded all services to the previous release version and have been running it for some time to migrate data). Both of these requests (and more!) can be achieved, but that’s outside the scope of this article. I’d highly recommend looking at the source code of the OpenStack Neutron project, which implements this functionality in a full-featured manner.

Quotas in OpenStack

Fri, 07 Oct 2022 00:00:00 +0000

I’ve been playing around with quotas in OpenStack again. Every time I do, I encounter another strange bit of behavior that catches me out. This time, I’ve decided to write down these strange things so at least I have a reference to go back to at some point in the future. I should probably get these notes into the docs for nova, cinder and neutron at some point…

Project quotas and default quotas

Quotas are a complicated area with a lot of baggage. Broadly speaking, there are two types of quota: default quotas and project-specific quotas. The default quotas are applied to projects unless specifically overridden by project quotas. In addition, for nova and cinder, there are two types of default quota: API-configured default quotas and statically configured default quotas defined in config files (so nova.conf or cinder.conf respectively). Neutron only supports statically configured quotas (neutron.conf). Project specific quotas take precedence over API-configured default quotas (where available), which in turn take priority over statically configured default quotas. We can visualize this as such:

nova:    conf default quotas < API default quotas < project quotas
cinder:  conf default quotas < API default quotas < project quotas
neutron: conf default quotas           <            project quotas

User-specific quotas

Nova has another type of quota: user-specific quotas. There’s only one type of user-specific quota in nova - keypairs and they’re not something that nova is likely to continue supporting in the long-term. As you’d expect, the primary difference between between user-specific quotas and project-specific quotas is that user-specific quotas are tied to the user instead of the project. This means you can specify that user foo has e.g. a keypair quota of 5, while user bar has a keypair quota of 6.

Quota classes

Finally, nova and cinder also have the concept of quota classes. The idea behind these was to allow for a two-level hierarchy of quotas, where you could define different “classes” of default quota and specific what class a project would get. For example, you could have three quota classes - gold, silver and bronze - and a project would get assigned to one of these quota classes depending on how much the customer was paying. This was seen as an easier alternative to setting defaults for each project individually. However, actually using quota classes required a separate out-of-tree service that would set a quota_class attribute in the request context. Rackspace apparently had one such service called Turnstile that did this but no one else appears to have implemented something like this and efforts to remove the need for an external service never went anywhere. In effect, this feature was never fully implemented and both nova and cinder only support the default quota class while neutron never even tried implementing this. It’s an irrelevance nowadays.

Quota usage

Obviously applying quotas to projects and users is one thing, but we actually need to track that usage somewhere. This also happens at the project level. The nova, cinder and neutron projects all track both a reserved value and an in_use value. In nova’s case, the reserved value was previously used to reserve a resource at the API layer (nova-api) before committing them at the compute layer (nova-compute). Nova moved away from this model in the 16.0.0 (Pike) release as part of the work to introduce cells v2, and the reserved value will now always be 0. This effort was tracked in the Count resources to check quota in API for cells spec.

In the case of nova, much of this information is gleaned either from the database or the placement service, depending on configuration. I suspect most services use a similar model. This usage information is available to the user via various APIs or openstackclient.

Quota drivers

The nova, cinder and neutron projects all have the concept of quota drivers. By default, all projects use a DB-based quota driver but both nova and neutron offer alternative drivers. I haven’t gone into detail on these here since it’s not entirely relevant to this discussion. You should refer to the nova and neutron configuration documentation for more information if interested though.

How do I use the damn thing?

The best way to interact with quotas is using a tool that tries to abstract all of the above craziness from you. To this end, I’d recommend openstackclient. While OSC has supported quotas for years, recent versions of this (v6.1.0 or later) have improved the UX further and contain some important feature additions, like the ability to view quota usage for the cinder service.

Firstly, let’s create a project-specific quota for, say, the number of instances.

❯ openstack quota set --instances 5 $OS_PROJECT_NAME

Now we can inspect those quotas:

❯ openstack quota show $OS_PROJECT_NAME
+-----------------------+-------+
| Resource              | Limit |
+-----------------------+-------+
| cores                 |    20 |
| instances             |    10 |
| ram                   | 51200 |
| volumes               |    10 |
| snapshots             |    43 |
| gigabytes             |  1000 |
| backups               |    10 |
| volumes_lvmdriver-1   |    -1 |
| gigabytes_lvmdriver-1 |    -1 |
| snapshots_lvmdriver-1 |    -1 |
| volumes___DEFAULT__   |    -1 |
| gigabytes___DEFAULT__ |    -1 |
| snapshots___DEFAULT__ |    -1 |
| groups                |    10 |
| networks              |   100 |
| ports                 |   500 |
| rbac_policies         |    10 |
| routers               |    21 |
| subnets               |   100 |
| subnet_pools          |    -1 |
| fixed-ips             |    -1 |
| injected-file-size    | 10240 |
| injected-path-size    |   255 |
| injected-files        |     5 |
| key-pairs             |    33 |
| properties            |   128 |
| server-groups         |    10 |
| server-group-members  |    10 |
| floating-ips          |    50 |
| secgroup-rules        |   100 |
| secgroups             |    10 |
| backup-gigabytes      |  1000 |
| per-volume-gigabytes  |    -1 |
+-----------------------+-------+

You can also include usage information if you want:

❯ openstack quota show --usage $OS_PROJECT_NAME
+-----------------------+-------+--------+----------+
| Resource              | Limit | In Use | Reserved |
+-----------------------+-------+--------+----------+
| cores                 |    20 |      2 |        0 |
| instances             |    10 |      2 |        0 |
| ram                   | 51200 |   4096 |        0 |
| volumes               |    10 |      1 |        0 |
| snapshots             |    43 |      0 |        0 |
| gigabytes             |  1000 |      5 |        0 |
| backups               |    10 |      0 |        0 |
| volumes_lvmdriver-1   |    -1 |      1 |        0 |
| gigabytes_lvmdriver-1 |    -1 |      5 |        0 |
| snapshots_lvmdriver-1 |    -1 |      0 |        0 |
| volumes___DEFAULT__   |    -1 |      0 |        0 |
| gigabytes___DEFAULT__ |    -1 |      0 |        0 |
| snapshots___DEFAULT__ |    -1 |      0 |        0 |
| groups                |    10 |      0 |        0 |
| networks              |   100 |      2 |        0 |
| ports                 |   500 |      4 |        0 |
| rbac_policies         |    10 |      4 |        0 |
| routers               |    21 |      0 |        0 |
| subnets               |   100 |      3 |        0 |
| subnet_pools          |    -1 |      2 |        0 |
| fixed-ips             |    -1 |      0 |        0 |
| injected-file-size    | 10240 |      0 |        0 |
| injected-path-size    |   255 |      0 |        0 |
| injected-files        |     5 |      0 |        0 |
| key-pairs             |    33 |      0 |        0 |
| properties            |   128 |      0 |        0 |
| server-groups         |    10 |      0 |        0 |
| server-group-members  |    10 |      0 |        0 |
| floating-ips          |    50 |      0 |        0 |
| secgroup-rules        |   100 |      4 |        0 |
| secgroups             |    10 |      1 |        0 |
| backup-gigabytes      |  1000 |      0 |        0 |
| per-volume-gigabytes  |    -1 |      0 |        0 |
+-----------------------+-------+--------+----------+

Default quotas are applied to each project unless there are project-specific quotas to override them. These can be inspected also:

❯ openstack quota show --default
+-----------------------+-------+
| Resource              | Limit |
+-----------------------+-------+
| cores                 |    20 |
| instances             |    10 |
| ram                   | 51200 |
| volumes               |    10 |
| snapshots             |    43 |
| gigabytes             |  1000 |
| backups               |    10 |
| volumes_lvmdriver-1   |    -1 |
| gigabytes_lvmdriver-1 |    -1 |
| snapshots_lvmdriver-1 |    -1 |
| volumes___DEFAULT__   |    -1 |
| gigabytes___DEFAULT__ |    -1 |
| snapshots___DEFAULT__ |    -1 |
| groups                |    10 |
| networks              |   100 |
| ports                 |   500 |
| rbac_policies         |    10 |
| routers               |    10 |
| subnets               |   100 |
| subnet_pools          |    -1 |
| fixed-ips             |    -1 |
| injected-file-size    | 10240 |
| injected-path-size    |   255 |
| injected-files        |     5 |
| key-pairs             |    33 |
| properties            |   128 |
| server-groups         |    10 |
| server-group-members  |    10 |
| floating-ips          |    50 |
| secgroup-rules        |   100 |
| secgroups             |    10 |
| backup-gigabytes      |  1000 |
| per-volume-gigabytes  |    -1 |
+-----------------------+-------+

You can list quotas for multiple projects. Note that when doing this, only the projects with project-specific quotas are shown to avoid dumping potentially thousands of lines of duplicate, useless info to the terminal.

❯ openstack quota list --compute
+----------------------------------+-------+-----------+----------------+-----------------------------+--------------------------+-----------+-----------+----------------+-------+---------------+----------------------+
| Project ID                       | Cores | Fixed IPs | Injected Files | Injected File Content Bytes | Injected File Path Bytes | Instances | Key Pairs | Metadata Items |   Ram | Server Groups | Server Group Members |
+----------------------------------+-------+-----------+----------------+-----------------------------+--------------------------+-----------+-----------+----------------+-------+---------------+----------------------+
| 700a8fa37f154153809be9d1814d8625 |    20 |        -1 |              5 |                       10240 |                      255 |         5 |        33 |            128 | 51200 |            10 |                   10 |
+----------------------------------+-------+-----------+----------------+-----------------------------+--------------------------+-----------+-----------+----------------+-------+---------------+----------------------+

Finally, you can unset configured quotas:

❯ openstack quota delete $OS_PROJECT_NAME

Addendum! Unified limits

If you consider the above rather confusing, you should try to using it in real-world environments! The deficiencies with the existing quota model were brought up in multiple discussions with operators and end users over the years, and the end result of these discussions was a concept known as unified limits. As the name would suggest, the aim of this feature is to provide a unified model for maintaining resource limits across multiple services. Good documentation exists to describe the benefits of this approach over the status quo. Adding support for unified limits requires significant effort on the end of the individual projects, and so far only the compute project appears to have started down this path, introducing preliminary support for unified limits during the yoga release. This remains in development and I’m not sure when we plan to finish it. The above guide will probably be useful for some time to come 😅

SQLAlchemy Relationships Without Foreign Keys (And How to Replace 'backref' With 'back_populates')

Fri, 07 Oct 2022 00:00:00 +0000

When implementing relationships in SQLAlchemy models, you will generally need to cross-reference one model from another, ideally in a bidirectional manner. Historically, the mechanism to do this has been the relationship.backref keyword. However, as noted in the SQLAlchemy documentation, this keyword should now be considered legacy and the relationship.back_populates migration preferred instead. There are a couple of reasons for this. Quoting from the docs:

The relationship.backref keyword should be considered legacy, and use of relationship.back_populates with explicit relationship() constructs should be preferred. Using individual relationship() constructs provides advantages including that both ORM mapped classes will include their attributes up front as the class is constructed, rather than as a deferred step, and configuration is more straightforward as all arguments are explicit. New PEP 484 features in SQLAlchemy 2.0 also take advantage of attributes being explicitly present in source code rather than using dynamic attribute generation.

The SQLAlchemy docs include a guide on migrating tables, (as well as a great overview on relationships in general), however, this guide assumes you are using foreign key constraints in your models and using the auto-incrementing primary key field as your reference. To be honest, you probably should be using foreign keys since without these you have no way to enforce referential integrity but say you can’t use them or don’t want to, how does one migrate? The below examples should hopefully guide you in migrating these.

One-to-Many

The one-to-many pattern is the simpler one to migrate. Consider the following example:

import uuid

import sqlalchemy as sa
from sqlalchemy import orm


BASE = orm.declarative_base()


class User(BASE):
    __tablename__ = 'user'

    id = sa.Column(sa.Integer, primary_key=True)
    uuid = sa.Column(sa.String(36), nullable=False)
    name = sa.Column(sa.String)


class Address(BASE):
    __tablename__ = 'addresses'
    __tableargs__ = (
        sa.Index('addresses_user_uuid_idx', 'user_uuid'),
    )

    id = sa.Column(sa.Integer, primary_key=True)
    user_uuid = sa.Column(sa.String(36), nullable=False)

    user = orm.relationship(
        'User',
        backref='addresses',
        primaryjoin='Address.user_uuid == User.uuid',
        foreign_keys=user_uuid,
    )


def main():
    engine = sa.create_engine('sqlite://')
    session = orm.sessionmaker(bind=engine)()
    BASE.metadata.create_all(engine)

    user = User(name='John Doe', uuid=str(uuid.uuid4()))

    session.add(user)
    session.commit()

    address = Address(user_uuid=user.uuid)
    session.add(address)
    session.commit()

    print('# Users')
    for user in session.query(User).all():
        print(f'User: name={user.name}')
    print()

    print('# Addresses')
    for address in session.query(Address).all():
        print(f'Address: user={address.user_uuid}')
    print()


if __name__ == '__main__':
    main()

Before looking at migrating from backref to back_populates, it’s worth understanding what’s going on here. You’ll note that we have a relationship from the Address model back to the User model. This has the backref keyword set on it, allowing you to reference related addresses from instances of User, however it also has primaryjoin and foreign_keys keywords set. These are necessary because the user_uuid column does not have a foreign key constraint present. You can confirm this by inspecting the schemas generated by SQLAlchemy.

❯ python
>>> import test_o2m  # name of module saved locally
>>> from sqlalchemy import schema
>>> print(schema.CreateTable(test_o2m.User.__table__))

CREATE TABLE "user" (
        id INTEGER NOT NULL,
        uuid VARCHAR(36) NOT NULL,
        name VARCHAR,
        PRIMARY KEY (id)
)


>>> print(schema.CreateTable(test_o2m.Address.__table__))

CREATE TABLE addresses (
        id INTEGER NOT NULL,
        user_uuid VARCHAR(36) NOT NULL,
        PRIMARY KEY (id)
)

Because we don’t have a foreign key present, SQLAlchemy isn’t able to determine the join condition. This necessitates the addition of the primaryjoin and foreign_keys keywords to guide SQLAlchemy on resolving these relations. Removing these keywords will yield an error message.

sqlalchemy.exc.NoForeignKeysError: Can't find any foreign key relationships between 'addresses' and 'user'.

An alternative to the foreign_keys argument is to annotate the join condition with the foreign() annotation.

user = orm.relationship(
    'User',
    backref='addresses',
    primaryjoin='Address.user_uuid == foreign(User.uuid)',
)

The only other unusual thing here is that we’re using a field other than id to reference things, namely a uuid field on both models. There’s no real reason to do this here but this was the case in the models that I had to migrate and it makes things a little more complicated and “real-world’y”.

Now that we understand what’s going, it’s time to look at switching from backref to back_populates. It turns out to be pretty simple for one-to-many models like this.

import uuid

import sqlalchemy as sa
from sqlalchemy import orm


BASE = orm.declarative_base()


class User(BASE):
    __tablename__ = 'user'

    id = sa.Column(sa.Integer, primary_key=True)
    uuid = sa.Column(sa.String(36), nullable=False)
    name = sa.Column(sa.String)

    addresses = orm.relationship(
        'Address',
        primaryjoin='User.uuid == Address.user_uuid',
        back_populates='user',
        foreign_keys='Address.user_uuid',
    )


class Address(BASE):
    __tablename__ = 'addresses'
    __tableargs__ = (
        sa.Index('addresses_user_uuid_idx', 'user_uuid'),
    )

    id = sa.Column(sa.Integer, primary_key=True)
    user_uuid = sa.Column(sa.String(36), nullable=False)

    user = orm.relationship(
        'User',
        primaryjoin='Address.user_uuid == User.uuid',
        back_populates='addresses',
        foreign_keys=user_uuid,
    )


def main():
    engine = sa.create_engine('sqlite://')
    session = orm.sessionmaker(bind=engine)()
    BASE.metadata.create_all(engine)

    user = User(name='John Doe', uuid=str(uuid.uuid4()))

    session.add(user)
    session.commit()

    address = Address(user_uuid=user.uuid)
    session.add(address)
    session.commit()

    print('# Users')
    for user in session.query(User).all():
        print(f'User: name={user.name}')
    print()

    print('# Addresses')
    for address in session.query(Address).all():
        print(f'Address: user={address.user_uuid}')
    print()


if __name__ == '__main__':
    main()

Looking at the diff for these two files, we see the following:

diff --git test_o2m.py test_o2m.py
index 4f23221..f865045 100644
--- test_o2m.py
+++ test_o2m.py
@@ -14,6 +14,13 @@ class User(BASE):
     uuid = sa.Column(sa.String(36), nullable=False)
     name = sa.Column(sa.String)

+    addresses = orm.relationship(
+        'Address',
+        primaryjoin='User.uuid == Address.user_uuid',
+        back_populates='user',
+        foreign_keys='Address.user_uuid',
+    )
+

 class Address(BASE):
     __tablename__ = 'addresses'
@@ -26,8 +33,8 @@ class Address(BASE):

     user = orm.relationship(
         'User',
-        backref='addresses',
         primaryjoin='Address.user_uuid == User.uuid',
+        back_populates='addresses',
         foreign_keys=user_uuid,
     )

So for the Address model, we have simply replaced the backref keyword with back_populates. For the User model, we have added a new relationship. This is mostly the inverse of the relationship on the Address model and only the foreign_key argument points to the same field (a foreign field in the Address model case).

Overall, this isn’t a difficult migration. Migrating many-to-many models could be a little trickier though. Let’s see.

Many-to-Many

Now consider the following example:

import uuid

import sqlalchemy as sa
from sqlalchemy import orm


BASE = orm.declarative_base()


association_table = sa.Table(
    'association',
    BASE.metadata,
    sa.Column('parent_uuid', sa.String(36), primary_key=True),
    sa.Column('child_uuid', sa.String(36), primary_key=True),
)


class Parent(BASE):
    __tablename__ = 'parents'
    __table_args__ = (
        sa.Index('parents_uuid_idx', 'uuid', unique=True),
    )

    id = sa.Column(sa.Integer, primary_key=True)
    uuid = sa.Column(sa.String(36), nullable=False)
    name = sa.Column(sa.String)


class Child(BASE):
    __tablename__ = 'children'
    __table_args__ = (
        sa.Index('children_uuid_idx', 'uuid', unique=True),
    )

    id = sa.Column(sa.Integer, primary_key=True)
    uuid = sa.Column(sa.String(36), nullable=False)
    name = sa.Column(sa.String)

    parents = orm.relationship(
        'Parent',
        secondary='association',
        primaryjoin='Child.uuid == association.c.child_uuid',
        secondaryjoin='association.c.parent_uuid == Parent.uuid',
        backref='children',
    )


def main():
    engine = sa.create_engine('sqlite://')
    session = orm.sessionmaker(bind=engine)()
    BASE.metadata.create_all(engine)

    parent = Parent(name='John Doe', uuid=str(uuid.uuid4()))
    session.add(parent)
    session.commit()

    child = Child(name='Jimmy Doe', uuid=str(uuid.uuid4()))
    session.add(child)
    session.commit()

    parent.children.append(child)

    print('# Parents')
    for parent in session.query(Parent).all():
        children = [x.name for x in parent.children]
        print(f'Parent: name={parent.name}, children={children}')
    print()

    print('# Children')
    for child in session.query(Child).all():
        parents = [x.name for x in child.parents]
        print(f'Child: name={child.name}, parents={parents}')
    print()


if __name__ == '__main__':
    main()

Again, there are a couple of things to consider here. Firstly, since this is a many-to-many relationship, we need a so-called join table (also known as a association table in SQLAlchemy or a through table in Django). The necessitates the use of the secondary keyword to indicate the association table. Next up, once again we do not have a foreign key constraint present meaning we need to inform SQLAlchemy how the join should be managed. The use of the association table means we must specify not only primaryjoin but also secondaryjoin: the former describes the relationship from Child to the association table, while the latter describes the relationship from the association table to Parent. Finally, once again we’re using a uuid field for referencing to makes things a little more like the real world.

Now to switch from backref to back_populates once again.

import uuid

import sqlalchemy as sa
from sqlalchemy import orm


BASE = orm.declarative_base()


association_table = sa.Table(
    'association',
    BASE.metadata,
    sa.Column('parent_uuid', sa.String(36), primary_key=True),
    sa.Column('child_uuid', sa.String(36), primary_key=True),
)


class Parent(BASE):
    __tablename__ = 'parents'
    __table_args__ = (
        sa.Index('parents_uuid_idx', 'uuid', unique=True),
    )

    id = sa.Column(sa.Integer, primary_key=True)
    uuid = sa.Column(sa.String(36), nullable=False)
    name = sa.Column(sa.String)

    children = orm.relationship(
        'Child',
        secondary='association',
        primaryjoin='Parent.uuid == association.c.parent_uuid',
        secondaryjoin='association.c.child_uuid == Child.uuid',
        back_populates='parents',
    )


class Child(BASE):
    __tablename__ = 'children'
    __table_args__ = (
        sa.Index('children_uuid_idx', 'uuid', unique=True),
    )

    id = sa.Column(sa.Integer, primary_key=True)
    uuid = sa.Column(sa.String(36), nullable=False)
    name = sa.Column(sa.String)

    parents = orm.relationship(
        'Parent',
        secondary='association',
        primaryjoin='Child.uuid == association.c.child_uuid',
        secondaryjoin='association.c.parent_uuid == Parent.uuid',
        back_populates='children',
    )


def main():
    engine = sa.create_engine('sqlite://')
    session = orm.sessionmaker(bind=engine)()
    BASE.metadata.create_all(engine)

    parent = Parent(name='John Doe', uuid=str(uuid.uuid4()))
    session.add(parent)
    session.commit()

    child = Child(name='Jimmy Doe', uuid=str(uuid.uuid4()))
    session.add(child)
    session.commit()

    parent.children.append(child)

    print('# Parents')
    for parent in session.query(Parent).all():
        children = [x.name for x in parent.children]
        print(f'Parent: name={parent.name}, children={children}')
    print()

    print('# Children')
    for child in session.query(Child).all():
        parents = [x.name for x in child.parents]
        print(f'Child: name={child.name}, parents={parents}')
    print()


if __name__ == '__main__':
    main()

Looking at the diff for these two files, we see the following:

diff --git test_m2m.py test_m2m.py
index 948942f..d1dc0d4 100644
--- test_m2m.py
+++ test_m2m.py
@@ -25,6 +25,14 @@ class Parent(BASE):
     uuid = sa.Column(sa.String(36), nullable=False)
     name = sa.Column(sa.String)

+    children = orm.relationship(
+        'Child',
+        secondary='association',
+        primaryjoin='Parent.uuid == association.c.parent_uuid',
+        secondaryjoin='association.c.child_uuid == Child.uuid',
+        back_populates='parents',
+    )
+

 class Child(BASE):
     __tablename__ = 'children'
@@ -41,7 +49,7 @@ class Child(BASE):
         secondary='association',
         primaryjoin='Child.uuid == association.c.child_uuid',
         secondaryjoin='association.c.parent_uuid == Parent.uuid',
-        backref='children',
+        back_populates='children',
     )

This is nearly identical to the changes required for migrating the one-to-many models. Once again, we simply replace backref with back_populates on the relationship that is defined. We then define a new relationship on the Parent model which has an inverse of the primary and secondary joins given for the Child model relationship.

Summary

So that’s how you migrate from backref to back_populates on models without foreign key constraints present. This likely isn’t broadly useful but hopefully it will help someone that encounters this.

The code for this blog post can be found on GitHub.