In a previous post, I provided a guide on configuring SR-IOV for a Mellanox ConnectX-3 NIC. I’ve since picked up a second one of these and was attempting to follow through on the same guide. However, when I attempted to “query” the device, I saw the following:
$ sudo mstconfig -d 02:00.0 query
Device #1:
----------
Device type: ConnectX3
PCI device: 02:00.0
-E- Failed to query device: 02:00.0. Unsupported FW (version 2.31.5000 or above required for CX3/PRO)
Clearly we need to update something here, hence, here’s my guide to updating
that firmware and making mstconfig happy again.
Prerequisites
As with the other post, you’re going to need a Mellanox ConnectX-3 card for this to be of any use. Once again, I’m using Ubuntu 16.04 because that’s what that OpenStack gate uses but I think most of this stuff is packaged on Fedora too.
Identify and Download Firmware
As you can see on the firmware download page, there are a variety of
ConnectX-3 models available, each with different feature sets (Ethernet vs.
Infiniband, two ports vs. four). You need to figure out which one you have.
These models are identified by their OPN and PSID. I’ve still to figure
out how to identify the former, but the latter is easily identified using the
mstflint tool. For example:
$ sudo mstflint -d 02:00.0 q
Image type: FS2
FW Version: 2.30.8000
Rom Info: type=PXE version=3.4.151 devid=4099 proto=ETH
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: 0002c90300056aa8 0002c90300056aa9 0002c90300056aaa 0002c90300056aab
MACs: 0002c921d6d0 0002c921d6d1
VSD:
PSID: MT_108012002
So with our PSID to hand - MT_108012002 in this case - we can proceed to
downloading the correct firmware from the firmware download page. There doesn’t
seem to be anyway to download by just PSID so I had to click through the list
of OPNs until I found the OPN with a PSID that matches. In my case, this was
MCX312A-XCBT for firmware version 2.42.5000. Download the relevant
firmware to your host. For example:
$ wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin.zip
Update Firmware
With our firmware update to hand, it’s time to actually update the NIC. For this section, I initially followed the updating guide provided by Mellanox. However, as before, I didn’t want to install anything that wasn’t already provided with my distro. As such the instructions provided below are a suitable alternative.
Firstly, extract the firmware you downloaded previously. For example:
$ unzip fw-ConnectX3-rel-2_42_5000-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin.zip
Once done, it’s time to flash the firmware. To do this, we once again use the
mstflint tool::
$ sudo mstflint -d 02:00.0 -i fw-ConnectX3-rel-2_42_5000-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin burn
Current FW version on flash: 2.30.8000
New FW version: 2.42.5000
Burning FS2 FW image without signatures - OK
Restoring signature - OK
Now reboot your machine so the new firmware is loaded::
$ sudo reboot
And with that, we have flashed out new firmware and can query the NIC configuration::
$ sudo mstconfig -d 02:00.0 query
Device #1:
----------
Device type: ConnectX3
PCI device: 02:00.0
Configurations: Current
SRIOV_EN 0
NUM_OF_VFS 8
LOG_BAR_SIZE 3
BOOT_OPTION_ROM_EN_P1 1
BOOT_VLAN_EN_P1 0
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 1
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 1
BOOT_VLAN_EN_P2 0
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 1
BOOT_VLAN_P2 1
Easy peasy. Now back to configuring SR-IOV.