github twitter linkedin email
CPU pinning with image metadata

OpenStack Nova allows the specification of image metadata to provide per-instance metadata. This is similar to the flavor extra specs. The CPU pinning feature makes use of this to allow for configuration of CPU policies and CPU thread policies via such metdata.

Someone suggested to me that CPU policies many not be being applied correctly when stored as image metadata rather than flavor properties. I decided to investigate this claim.

Commit 8bafc9 of Nova was used for this testing. Much of this feature has been documented upstreamed since conducting this testing.

Steps

Create a custom image with metadata

The first step is to create a new image and save metadata against this image. To do this, we’ll duplicate an existing image and modify this duplicate. Begin by duplicating the image:

$ openstack image list
+--------------------------------------+---------------------------------+--------+
| ID                                   | Name                            | Status |
+--------------------------------------+---------------------------------+--------+
| a3450a21-f012-4807-a514-0838750d430e | cirros-0.3.4-x86_64-uec         | active |
| 742bc914-e60c-4eab-9e5b-0a1c19ec1a84 | cirros-0.3.4-x86_64-uec-ramdisk | active |
| 3e507ab7-080d-4eea-845a-69950ec139b8 | cirros-0.3.4-x86_64-uec-kernel  | active |
+--------------------------------------+---------------------------------+--------+

$ openstack image save --file cirros.img a3450a21-f012-4807-a514-0838750d430e

$ openstack image create --file cirros.img cirros-0.3.4-x86_64-pinned
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| checksum         | eb9139e4942121f22bbc2afc0400b2a4                     |
| container_format | bare                                                 |
| created_at       | 2016-02-16T09:40:12Z                                 |
| disk_format      | raw                                                  |
| file             | /v2/images/e3e7d5a5-b044-493a-b3e1-fa9ba6225ed6/file |
| id               | e3e7d5a5-b044-493a-b3e1-fa9ba6225ed6                 |
| min_disk         | 0                                                    |
| min_ram          | 0                                                    |
| name             | cirros-0.3.4-x86_64-pinned                           |
| owner            | c1e8c7671b0f4016a9250a8787de6930                     |
| protected        | False                                                |
| schema           | /v2/schemas/image                                    |
| size             | 25165824                                             |
| status           | active                                               |
| tags             |                                                      |
| updated_at       | 2016-02-16T09:40:12Z                                 |
| virtual_size     | None                                                 |
| visibility       | private                                              |
+------------------+------------------------------------------------------+

Next up, add the metadata to the image. We’ll only worry about CPU pinning, by way of CPU policies, ignoring CPU thread pinning for now.

$ openstack image set cirros-0.3.4-x86_64-pinned --property hw_cpu_policy=dedicated

$ openstack image list
+--------------------------------------+---------------------------------+--------+
| ID                                   | Name                            | Status |
+--------------------------------------+---------------------------------+--------+
| e3e7d5a5-b044-493a-b3e1-fa9ba6225ed6 | cirros-0.3.4-x86_64-pinned      | active |
| a3450a21-f012-4807-a514-0838750d430e | cirros-0.3.4-x86_64-uec         | active |
| 742bc914-e60c-4eab-9e5b-0a1c19ec1a84 | cirros-0.3.4-x86_64-uec-ramdisk | active |
| 3e507ab7-080d-4eea-845a-69950ec139b8 | cirros-0.3.4-x86_64-uec-kernel  | active |
+--------------------------------------+---------------------------------+--------+

$ openstack image show cirros-0.3.4-x86_64-pinned
+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| checksum         | eb9139e4942121f22bbc2afc0400b2a4                     |
| container_format | bare                                                 |
| created_at       | 2016-02-16T09:40:12Z                                 |
| disk_format      | raw                                                  |
| file             | /v2/images/e3e7d5a5-b044-493a-b3e1-fa9ba6225ed6/file |
| id               | e3e7d5a5-b044-493a-b3e1-fa9ba6225ed6                 |
| min_disk         | 0                                                    |
| min_ram          | 0                                                    |
| name             | cirros-0.3.4-x86_64-pinned                           |
| owner            | c1e8c7671b0f4016a9250a8787de6930                     |
| properties       | hw_cpu_policy='dedicated'                            |
| protected        | False                                                |
| schema           | /v2/schemas/image                                    |
| size             | 25165824                                             |
| status           | active                                               |
| tags             |                                                      |
| updated_at       | 2016-02-16T09:41:42Z                                 |
| virtual_size     | None                                                 |
| visibility       | private                                              |
+------------------+------------------------------------------------------+

Create a new instance

Now create the flavor. There’s no need to specify any metadata on the flavors: it’s already specified in the image:

$ openstack flavor list
+----+-----------+-------+------+-----------+-------+-----------+
| ID | Name      |   RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------+-------+------+-----------+-------+-----------+
| 1  | m1.tiny   |   512 |    1 |         0 |     1 | True      |
| 2  | m1.small  |  2048 |   20 |         0 |     1 | True      |
| 3  | m1.medium |  4096 |   40 |         0 |     2 | True      |
| 4  | m1.large  |  8192 |   80 |         0 |     4 | True      |
| 42 | m1.nano   |    64 |    0 |         0 |     1 | True      |
| 5  | m1.xlarge | 16384 |  160 |         0 |     8 | True      |
| 84 | m1.micro  |   128 |    0 |         0 |     1 | True      |
+----+-----------+-------+------+-----------+-------+-----------+

$ openstack server create --flavor=m1.small --image=cirros-0.3.4-x86_64-pinned --wait test1

Validate that the instance has been created

Finally, ensure that things behave as expected. Instances should define an NUMA topology and CPU pinning in the libvirt XML file:

$ openstack server list
+--------------------------------------+-------+--------+--------------------------------------------------------+
| ID                                   | Name  | Status | Networks                                               |
+--------------------------------------+-------+--------+--------------------------------------------------------+
| 22c1afeb-06d8-4f6d-a8d8-7ea40ea9ff47 | test1 | ACTIVE | private=fd4d:adb9:5ebe:0:f816:3eff:fecd:a674, 10.0.0.3 |
+--------------------------------------+-------+--------+--------------------------------------------------------+

$ sudo virsh list
 Id    Name                           State
----------------------------------------------------
 4     instance-00000002              running

$ sudo virsh dumpxml 4
<domain type='kvm' id='4'>
  <name>instance-00000002</name>
  <vcpu placement='static'>1</vcpu>
  ...
  <cputune>
    <shares>1024</shares>
    <vcpupin vcpu='0' cpuset='1'/>
    <emulatorpin cpuset='1'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  ...
</domain>

Result

The image metadata is being loaded as expected. No issues here.

References