In a previous post, I provided a guide on configuring SR-IOV for a Mellanox ConnectX-3 NIC. I’ve since picked up a second one of these and was attempting to follow through on the same guide. However, when I attempted to “query” the device, I saw the following:
$ sudo mstconfig -d 02:00.0 query Device #1: ---------- Device type: ConnectX3 PCI device: 02:00.0 -E- Failed to query device: 02:00.0. Unsupported FW (version 2.31.5000 or above required for CX3/PRO)
Clearly we need to update something here, hence, here’s my guide to updating
that firmware and making
mstconfig happy again.
As with the other post, you’re going to need a Mellanox ConnectX-3 card for this to be of any use. Once again, I’m using Ubuntu 16.04 because that’s what that OpenStack gate uses but I think most of this stuff is packaged on Fedora too.
Identify and Download Firmware
As you can see on the firmware download page, there are a variety of
ConnectX-3 models available, each with different feature sets (Ethernet vs.
Infiniband, two ports vs. four). You need to figure out which one you have.
These models are identified by their OPN and PSID. I’ve still to figure
out how to identify the former, but the latter is easily identified using the
mstflint tool. For example:
$ sudo mstflint -d 02:00.0 q Image type: FS2 FW Version: 2.30.8000 Rom Info: type=PXE version=3.4.151 devid=4099 proto=ETH Device ID: 4099 Description: Node Port1 Port2 Sys image GUIDs: 0002c90300056aa8 0002c90300056aa9 0002c90300056aaa 0002c90300056aab MACs: 0002c921d6d0 0002c921d6d1 VSD: PSID: MT_108012002
So with our PSID to hand -
MT_108012002 in this case - we can proceed to
downloading the correct firmware from the firmware download page. There doesn’t
seem to be anyway to download by just PSID so I had to click through the list
of OPNs until I found the OPN with a PSID that matches. In my case, this was
MCX312A-XCBT for firmware version
2.42.5000. Download the relevant
firmware to your host. For example:
$ wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin.zip
With our firmware update to hand, it’s time to actually update the NIC. For this section, I initially followed the updating guide provided by Mellanox. However, as before, I didn’t want to install anything that wasn’t already provided with my distro. As such the instructions provided below are a suitable alternative.
Firstly, extract the firmware you downloaded previously. For example:
$ unzip fw-ConnectX3-rel-2_42_5000-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin.zip
Once done, it’s time to flash the firmware. To do this, we once again use the
$ sudo mstflint -d 02:00.0 -i fw-ConnectX3-rel-2_42_5000-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin burn Current FW version on flash: 2.30.8000 New FW version: 2.42.5000 Burning FS2 FW image without signatures - OK Restoring signature - OK
Now reboot your machine so the new firmware is loaded::
$ sudo reboot
And with that, we have flashed out new firmware and can query the NIC configuration::
$ sudo mstconfig -d 02:00.0 query Device #1: ---------- Device type: ConnectX3 PCI device: 02:00.0 Configurations: Current SRIOV_EN 0 NUM_OF_VFS 8 LOG_BAR_SIZE 3 BOOT_OPTION_ROM_EN_P1 1 BOOT_VLAN_EN_P1 0 BOOT_RETRY_CNT_P1 0 LEGACY_BOOT_PROTOCOL_P1 1 BOOT_VLAN_P1 1 BOOT_OPTION_ROM_EN_P2 1 BOOT_VLAN_EN_P2 0 BOOT_RETRY_CNT_P2 0 LEGACY_BOOT_PROTOCOL_P2 1 BOOT_VLAN_P2 1
Easy peasy. Now back to configuring SR-IOV.