OpenShift Virtualization with Localnet Configuration

This use case demonstrates how to configure OpenShift Virtualization to leverage the existing br-ex OVS bridge for direct north/south connectivity of virtual machines.

Using an external bridge (br-ex) as a local network (localnet) allows virtual machines (VMs) in OpenShift Virtualization to directly connect to a bare metal (external) network. At a high level, the configuration involves the following steps:

1. Prepare the node networking: Use the Kubernetes NMState operator to configure each cluster node’s networking so that the existing br-ex bridge is mapped for external traffic (creating a local network on br-ex)​. This is done by applying a NodeNetworkConfigurationPolicy (NNCP) to all relevant nodes.
2. Define the local network attachment: Create a NetworkAttachmentDefinition (NAD) for the new local network. This defines how VMs can attach to the br-ex bridge network you set up on the nodes.
3. Attach VMs to the external bridge network: Configure VMs to use the new NAD as a secondary network interface. This bridges the VM’s network interface to the physical network via br-ex, giving the VM external connectivity.

Configure NodeNetworkConfigurationPolicy

Create a NodeNetworkConfigurationPolicy (NNCP) to add a localnet mapping to the br-ex bridge and apply it.

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br-ex-network
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: '' 
  desiredState:
    ovn:
      bridge-mappings:
      - localnet: br-ex-network
        bridge: br-ex 
        state: present

oc apply -f br-ex-nncp.yaml

Once the NNCP is applied, the NMState operator will configure the OVS on each node accordingly. The br-ex bridge remains in place with its primary cluster role, but now it also has an additional local network tag (e.g. br-ex-network) associated with it for use by secondary networks​. This essentially tells OVN-Kubernetes that there is a provider network (localnet) available via br-ex which can be used to bridge traffic out to the physical network.

[!NOTE]
The above method is specific to clusters using OVN-Kubernetes networking. (If the cluster were using the older openshift-sdn, this direct localnet approach would not apply, since br-ex is an OVN construct.) In an OVN-Kubernetes deployment, all nodes have a br-ex, making it convenient to leverage as a local network. If multiple NICs are available, an alternative is to create a separate OVS bridge on a spare NIC and map that as a localnet, but with a single NIC, using the existing br-ex is the recommended approach

Verify NodeNetworkConfigurationPolicy Status

Check that the policy has been applied correctly

oc get nncp
NAME            STATUS      REASON
br-ex-network   Available   SuccessfullyConfigured

Verify the node network enactments

oc get nnce
NAME                         STATUS      STATUS AGE   REASON
hub-n1.br-ex-network        Available     10s         SuccessfullyConfigured

Create NetworkAttachmentDefinition

Create a NetworkAttachmentDefinition (NAD) that will use the localnet and apply it

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: br-ex-network
  namespace: default
spec:
  config: '{
            "name":"br-ex-network",
            "type":"ovn-k8s-cni-overlay",
            "cniVersion":"0.4.0",
            "topology":"localnet",
            "netAttachDefName":"default/br-ex-network"
          }'

oc apply -f br-ex-network-nad.yaml

VLAN Configuration Example

To configure a NetworkAttachmentDefinition with a specific VLAN ID, use the vlanID property in the config. This is particularly useful for environments where network segmentation is required.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vlan-network
  namespace: default
spec:
  config: |
    {
            "cniVersion": "0.3.1",
            "name": "vlan-network",
            "type": "ovn-k8s-cni-overlay",
            "topology": "localnet",
            "vlanID": 200,
            "netAttachDefName": "default/vlan-network"
    }

oc apply -f vlan-network-nad.yaml

Adding Network Interface to Virtual Machine

To attach a VM to the localnet bridge, modify your VM definition to include an additional network interface

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: example-vm
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        devices:
          disks:
          - name: rootdisk
            disk:
              bus: virtio
          interfaces:
          - name: default
            masquerade: {}
          - name: br-ex-interface
            bridge: {}
        resources:
          requests:
            memory: 2Gi
      networks:
      - name: default
        pod: {}
      - name: br-ex-interface
        multus:
          networkName: default/br-ex-network
      volumes:
      - name: rootdisk
        containerDisk:
          image: quay.io/containerdisks/fedora:latest

Testing and Validation

Once the vm is up and runnig connect to it using openshift console or virtctl and check network interfaces

# vritctl console test-vm
$ ip addr show

Verify you have two network interfaces

• First inetrface connected to the pod network (default)
• Second inetrface connected to the br-ex network with an IP from your baremetal network

Summary

In summary, configuring br-ex as a local network in OpenShift Virtualization provides a high-performance, straightforward way to give VMs external network connectivity. The key steps involve using an NNCP to prepare the host bridges and a NAD to expose that network to VMs, resulting in VMs that are first-class citizens on the physical network – with native IPs and direct access – all while using OpenShift’s declarative configuration and existing network infrastructure.​

• Assign IP addresses to virtual machines directly from the bare metal network
• Enable direct north-south traffic without the need for NAT
• Streamline the network design by leveraging existing infrastructure

References

• Redhat Documentation – Secondary Network Config
• Redhat Documentation – Virtualization Networking