Multipoint VXLAN Tunnels

I recently had to deploy a multi-node OpenStack cluster in Azure. Like most cloud platforms, Azure does not expose L2 networks and implements MAC spoofing protection, preventing VMs from advertising IPs that have not been assigned to them. However, we want a provider network in our deployment in order to provide external network connectivity to OpenStack VMs, which meant we needed some kind of L2 network. Enter overlay networks, specifically VXLAN network, which will let us provide a virtual L2 domain for our created VMs. We ended up not needing this as we only deployed a single networking controller node, but it was an interesting exercise all the same. I’ve documented the steps I needed to take to get this functioning below.

The scenario

We have three hosts: a bastion and two OpenStack hosts. We would like to configure a VXLAN mesh between all three VMs.

Bastion: 192.168.226.4 (VXLAN IP: 192.168.234.1/23)
Host 00: 192.168.226.5 (VXLAN IP: 192.168.234.2/23)
Host 01: 192.168.226.6 (VXLAN IP: 192.168.234.3/23)

The solution

On bastion:

ip link add provider0 type vxlan id 16 local 192.168.226.4 dstport 0
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.5 dev provider0  # host-00
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.6 dev provider0  # host-01
ip addr add 192.168.234.1/23 dev provider0
ip link set up dev provider0

On host-00:

ip link add provider0 type vxlan id 16 local 192.168.226.5 dstport 0
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.4 dev provider0  # bastion
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.6 dev provider0  # host-01
ip addr add 192.168.234.2/23 dev provider0
ip link set up dev provider0

On host-01:

ip link add provider0 type vxlan id 16 local 192.168.226.6 dstport 0
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.4 dev provider0  # bastion
bridge fdb append to 00:00:00:00:00:00 dst 192.168.226.5 dev provider0  # host-00
ip addr add 192.168.234.3/23 dev provider0
ip link set up dev provider0

The (alternative) solution

If using Debian or Ubuntu, you can achieve part of this using netplan. This can be helpful when using e.g. Ansible. If using netplan, simply omit the network.tunnels.{name}.remote setting, like so:

network:
  tunnels:
    provider0:
      mode: vxlan
      id: 16
      local: 192.168.226.4
      addresses: [ 192.168.234.1/23 ]
  version: 2

You then need to run the bridge fdb commands manually as it doesn’t seem to be possible to configure this via netplan.

The explanation

Typically, you would create a VXLAN tunnel endpoint (VTEP) using a command like so:

ip link add provider0 type vxlan id 16 local 192.168.226.4 remote 192.168.226.5 dstport 0
ip addr add 192.168.234.1/23 dev provider0

However, the ip command does not allow you to specify multiple remote stanzas. As a result, you need to omit this and manually specify forwarding database entries using the bridge fdb command.

BONUS! Using GRE instead

This wasn’t an option for me because GRE is apparently blocked by Azure. In theory though, there’s no reason this shouldn’t work otherwise.