Quotas in OpenStack

I’ve been playing around with quotas in OpenStack again. Every time I do, I encounter another strange bit of behavior that catches me out. This time, I’ve decided to write down these strange things so at least I have a reference to go back to at some point in the future. I should probably get these notes into the docs for nova, cinder and neutron at some point…

Project quotas and default quotas

Quotas are a complicated area with a lot of baggage. Broadly speaking, there are two types of quota: default quotas and project-specific quotas. The default quotas are applied to projects unless specifically overridden by project quotas. In addition, for nova and cinder, there are two types of default quota: API-configured default quotas and statically configured default quotas defined in config files (so nova.conf or cinder.conf respectively). Neutron only supports statically configured quotas (neutron.conf). Project specific quotas take precedence over API-configured default quotas (where available), which in turn take priority over statically configured default quotas. We can visualize this as such:

nova:    conf default quotas < API default quotas < project quotas
cinder:  conf default quotas < API default quotas < project quotas
neutron: conf default quotas           <            project quotas

User-specific quotas

Nova has another type of quota: user-specific quotas. There’s only one type of user-specific quota in nova - keypairs and they’re not something that nova is likely to continue supporting in the long-term. As you’d expect, the primary difference between between user-specific quotas and project-specific quotas is that user-specific quotas are tied to the user instead of the project. This means you can specify that user foo has e.g. a keypair quota of 5, while user bar has a keypair quota of 6.

Quota classes

Finally, nova and cinder also have the concept of quota classes. The idea behind these was to allow for a two-level hierarchy of quotas, where you could define different “classes” of default quota and specific what class a project would get. For example, you could have three quota classes - gold, silver and bronze - and a project would get assigned to one of these quota classes depending on how much the customer was paying. This was seen as an easier alternative to setting defaults for each project individually. However, actually using quota classes required a separate out-of-tree service that would set a quota_class attribute in the request context. Rackspace apparently had one such service called Turnstile that did this but no one else appears to have implemented something like this and efforts to remove the need for an external service never went anywhere. In effect, this feature was never fully implemented and both nova and cinder only support the default quota class while neutron never even tried implementing this. It’s an irrelevance nowadays.

Quota usage

Obviously applying quotas to projects and users is one thing, but we actually need to track that usage somewhere. This also happens at the project level. The nova, cinder and neutron projects all track both a reserved value and an in_use value. In nova’s case, the reserved value was previously used to reserve a resource at the API layer (nova-api) before committing them at the compute layer (nova-compute). Nova moved away from this model in the 16.0.0 (Pike) release as part of the work to introduce cells v2, and the reserved value will now always be 0. This effort was tracked in the Count resources to check quota in API for cells spec.

In the case of nova, much of this information is gleaned either from the database or the placement service, depending on configuration. I suspect most services use a similar model. This usage information is available to the user via various APIs or openstackclient.

Quota drivers

The nova, cinder and neutron projects all have the concept of quota drivers. By default, all projects use a DB-based quota driver but both nova and neutron offer alternative drivers. I haven’t gone into detail on these here since it’s not entirely relevant to this discussion. You should refer to the nova and neutron configuration documentation for more information if interested though.

How do I use the damn thing?

The best way to interact with quotas is using a tool that tries to abstract all of the above craziness from you. To this end, I’d recommend openstackclient. While OSC has supported quotas for years, recent versions of this (v6.1.0 or later) have improved the UX further and contain some important feature additions, like the ability to view quota usage for the cinder service.

Firstly, let’s create a project-specific quota for, say, the number of instances.

❯ openstack quota set --instances 5 $OS_PROJECT_NAME

Now we can inspect those quotas:

❯ openstack quota show $OS_PROJECT_NAME
+-----------------------+-------+
| Resource              | Limit |
+-----------------------+-------+
| cores                 |    20 |
| instances             |    10 |
| ram                   | 51200 |
| volumes               |    10 |
| snapshots             |    43 |
| gigabytes             |  1000 |
| backups               |    10 |
| volumes_lvmdriver-1   |    -1 |
| gigabytes_lvmdriver-1 |    -1 |
| snapshots_lvmdriver-1 |    -1 |
| volumes___DEFAULT__   |    -1 |
| gigabytes___DEFAULT__ |    -1 |
| snapshots___DEFAULT__ |    -1 |
| groups                |    10 |
| networks              |   100 |
| ports                 |   500 |
| rbac_policies         |    10 |
| routers               |    21 |
| subnets               |   100 |
| subnet_pools          |    -1 |
| fixed-ips             |    -1 |
| injected-file-size    | 10240 |
| injected-path-size    |   255 |
| injected-files        |     5 |
| key-pairs             |    33 |
| properties            |   128 |
| server-groups         |    10 |
| server-group-members  |    10 |
| floating-ips          |    50 |
| secgroup-rules        |   100 |
| secgroups             |    10 |
| backup-gigabytes      |  1000 |
| per-volume-gigabytes  |    -1 |
+-----------------------+-------+

You can also include usage information if you want:

❯ openstack quota show --usage $OS_PROJECT_NAME
+-----------------------+-------+--------+----------+
| Resource              | Limit | In Use | Reserved |
+-----------------------+-------+--------+----------+
| cores                 |    20 |      2 |        0 |
| instances             |    10 |      2 |        0 |
| ram                   | 51200 |   4096 |        0 |
| volumes               |    10 |      1 |        0 |
| snapshots             |    43 |      0 |        0 |
| gigabytes             |  1000 |      5 |        0 |
| backups               |    10 |      0 |        0 |
| volumes_lvmdriver-1   |    -1 |      1 |        0 |
| gigabytes_lvmdriver-1 |    -1 |      5 |        0 |
| snapshots_lvmdriver-1 |    -1 |      0 |        0 |
| volumes___DEFAULT__   |    -1 |      0 |        0 |
| gigabytes___DEFAULT__ |    -1 |      0 |        0 |
| snapshots___DEFAULT__ |    -1 |      0 |        0 |
| groups                |    10 |      0 |        0 |
| networks              |   100 |      2 |        0 |
| ports                 |   500 |      4 |        0 |
| rbac_policies         |    10 |      4 |        0 |
| routers               |    21 |      0 |        0 |
| subnets               |   100 |      3 |        0 |
| subnet_pools          |    -1 |      2 |        0 |
| fixed-ips             |    -1 |      0 |        0 |
| injected-file-size    | 10240 |      0 |        0 |
| injected-path-size    |   255 |      0 |        0 |
| injected-files        |     5 |      0 |        0 |
| key-pairs             |    33 |      0 |        0 |
| properties            |   128 |      0 |        0 |
| server-groups         |    10 |      0 |        0 |
| server-group-members  |    10 |      0 |        0 |
| floating-ips          |    50 |      0 |        0 |
| secgroup-rules        |   100 |      4 |        0 |
| secgroups             |    10 |      1 |        0 |
| backup-gigabytes      |  1000 |      0 |        0 |
| per-volume-gigabytes  |    -1 |      0 |        0 |
+-----------------------+-------+--------+----------+

Default quotas are applied to each project unless there are project-specific quotas to override them. These can be inspected also:

❯ openstack quota show --default
+-----------------------+-------+
| Resource              | Limit |
+-----------------------+-------+
| cores                 |    20 |
| instances             |    10 |
| ram                   | 51200 |
| volumes               |    10 |
| snapshots             |    43 |
| gigabytes             |  1000 |
| backups               |    10 |
| volumes_lvmdriver-1   |    -1 |
| gigabytes_lvmdriver-1 |    -1 |
| snapshots_lvmdriver-1 |    -1 |
| volumes___DEFAULT__   |    -1 |
| gigabytes___DEFAULT__ |    -1 |
| snapshots___DEFAULT__ |    -1 |
| groups                |    10 |
| networks              |   100 |
| ports                 |   500 |
| rbac_policies         |    10 |
| routers               |    10 |
| subnets               |   100 |
| subnet_pools          |    -1 |
| fixed-ips             |    -1 |
| injected-file-size    | 10240 |
| injected-path-size    |   255 |
| injected-files        |     5 |
| key-pairs             |    33 |
| properties            |   128 |
| server-groups         |    10 |
| server-group-members  |    10 |
| floating-ips          |    50 |
| secgroup-rules        |   100 |
| secgroups             |    10 |
| backup-gigabytes      |  1000 |
| per-volume-gigabytes  |    -1 |
+-----------------------+-------+

You can list quotas for multiple projects. Note that when doing this, only the projects with project-specific quotas are shown to avoid dumping potentially thousands of lines of duplicate, useless info to the terminal.

❯ openstack quota list --compute
+----------------------------------+-------+-----------+----------------+-----------------------------+--------------------------+-----------+-----------+----------------+-------+---------------+----------------------+
| Project ID                       | Cores | Fixed IPs | Injected Files | Injected File Content Bytes | Injected File Path Bytes | Instances | Key Pairs | Metadata Items |   Ram | Server Groups | Server Group Members |
+----------------------------------+-------+-----------+----------------+-----------------------------+--------------------------+-----------+-----------+----------------+-------+---------------+----------------------+
| 700a8fa37f154153809be9d1814d8625 |    20 |        -1 |              5 |                       10240 |                      255 |         5 |        33 |            128 | 51200 |            10 |                   10 |
+----------------------------------+-------+-----------+----------------+-----------------------------+--------------------------+-----------+-----------+----------------+-------+---------------+----------------------+

Finally, you can unset configured quotas:

❯ openstack quota delete $OS_PROJECT_NAME

Addendum! Unified limits

If you consider the above rather confusing, you should try to using it in real-world environments! The deficiencies with the existing quota model were brought up in multiple discussions with operators and end users over the years, and the end result of these discussions was a concept known as unified limits. As the name would suggest, the aim of this feature is to provide a unified model for maintaining resource limits across multiple services. Good documentation exists to describe the benefits of this approach over the status quo. Adding support for unified limits requires significant effort on the end of the individual projects, and so far only the compute project appears to have started down this path, introducing preliminary support for unified limits during the yoga release. This remains in development and I’m not sure when we plan to finish it. The above guide will probably be useful for some time to come 😅

comments powered by Disqus