Azure Networking: Accelerated Networking

Azure has a feature called Accelerated Networking. This feature is available for specific VMs on the Azure Platform. It will create a datapath directly from VM to the Mellanox network card on Azure hosting infrastructure. This removes a virtual switch and virtual layer on the physical server. You can find more information on this link. This will greatly increase the throughput capabilities of the virtual machines.

The VM Series sizes D/DSv2 and F/FS with 2 or more vCPUs. VM Series that support hyper threading like the D/DSv3, E/ESv3, FSv2 and Ms/Mms VMs are supported with 4 or more vCPUs are supported.

Deployment

To enable you can create a NIC with Accelerated Networking during the initial creation of the VM or you can perform this after VM creation.

Azure Marketplace

In the Azure Marketplace any “Accelerated Networking supported VM will have a property to enable it during initial creation.

Azure Powershell

Via Azure Powershell you can enable “Accelerated Networking” on the NIC. A VM needs to be deallocated for this. For the whole procedure you can find more information here.

$nic = Get-AzNetworkInterface -ResourceGroupName "ResourceGroup1" -Name "NetworkInterface1"
$nic.EnableAcceleratedNetworking = 1
$nic | Set-AzNetworkInterface

Azure CLI

Azure CLI can be used as well to enable Accelerated Networking either during creation or afterwards. The VM needs to be deallocated to enable the setting.

az network nic update --accelerated-networking true --name NetworkInterface1 --resource-group MyResourceGroup1

ARM template

If you have your own ARM template to deploy your VMs you can of course enable Accelerated Networking as well. Documentation can be found here.

    {
      "apiVersion": "[variables('networkApiVersion')]",
      "type": "Microsoft.Network/networkInterfaces",
      "name": "NetworkInterface1",
      "location": "westeurope",
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "subnet": {
                "id": "[variables('subnetId')]"
              }
            }
          }
        ],
        "enableAcceleratedNetworking": "true",
        "networkSecurityGroup": {
          "id": "[variables('nsgId')]"
        }
      }
    },

Terraform template

And of course you can alse have this inside of Terraform templates as in this example. More information can be found here.

resource "azurerm_network_interface" "NetworkInterface1" {
  name                            = "NetworkInterface1"
  location                        = "westeurope"
  resource_group_name             = "JVH20-RG"
  enable_accelerated_networking   = "true"
  network_security_group_id       = "${azurerm_network_security_group.nsg.id}"

  ip_configuration {
    name                                    = "interface1"
    subnet_id                               = "${azurerm_subnet.subnet.id}"
    private_ip_address_allocation           = "dynamic"
  }
}

Verification

Once deployed the question remains ‘Do we have accelerated networking enable on the VM?”. There are a couple of quick verification steps you can perform this depends also the operating system you deployed of course.

Linux

In Linux there several locations where you can verify Accelerated Networking. First of all you can check that the Mellanox driver has been loaded in the and active:

# lspci -v
0001:00:02.0 Ethernet controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
	Subsystem: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
	Physical Slot: 1
	Flags: bus master, fast devsel, latency 0
	Memory at fe0000000 (64-bit, prefetchable) [size=8M]
	Capabilities: [60] Express Endpoint, MSI 00
	Capabilities: [9c] MSI-X: Enable+ Count=24 Masked-
	Capabilities: [40] Power Management version 0
	Kernel driver in use: mlx4_core
	Kernel modules: mlx4_core

If the driver is ready to go the network interface configuration can be verified. The output of the ‘ip addr’ command shows the typical eth0 interface and a virtual/pseudo interface. In this case it is called ‘enP1s1’ and in the output it mentions that the master interface is ‘eth0’.

ip addr
 1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
     inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
     inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
 2: eth0:  mtu 1500 qdisc mq state UP group default qlen 1000
     link/ether 00:0d:3a:04:4b:ec brd ff:ff:ff:ff:ff:ff
     inet 10.0.0.4/24 brd 10.0.0.255 scope global eth0
        valid_lft forever preferred_lft forever
     inet6 fe80::20d:3aff:fe04:4bec/64 scope link
        valid_lft forever preferred_lft forever
 3: enP1s1:  mtu 1500 qdisc mq master eth0 state UP group default qlen 1000
     link/ether 00:0d:3a:04:4b:ec brd ff:ff:ff:ff:ff:ff

Next stop is the ‘ethtool’ that will give you some final clues. First of all the speed indicates ‘40000Mb/s’.

# ethtool eth0
Settings for eth0:
 Supported ports: [ ]
 Supported link modes:   Not reported
 Supported pause frame use: No
 Supports auto-negotiation: No
 Advertised link modes:  Not reported
 Advertised pause frame use: No
 Advertised auto-negotiation: No
 Speed: 40000Mb/s
 Duplex: Full
 Port: Other
 PHYAD: 0
 Transceiver: internal
 Auto-negotiation: off
 Link detected: yes

Another option in the ethtool is look activity in the the virtual function using the command: ‘ethtool -S eth0 | grep vf_’. This indicates that traffic is flowing via the virtual/pseudo network interface.

# ethtool -S eth0 | grep vf_
     vf_rx_packets: 15675
     vf_rx_bytes: 205105559
     vf_tx_packets: 41133
     vf_tx_bytes: 13694062
     vf_tx_dropped: 0

And a final indicator is to verify that the Mellanox driver is attached to the interupts requesting the output of /proc/interrupts.

# cat /proc/interrupts
            CPU0       CPU1
   1:          9          0   IO-APIC    1-edge      i8042
   4:       2550         97   IO-APIC    4-edge      ttyS0
   8:          0          0   IO-APIC    8-edge      rtc0
   9:          0          0   IO-APIC    9-fasteoi   acpi
  12:          0          3   IO-APIC   12-edge      i8042
  14:          0          0   IO-APIC   14-edge      ata_piix
  15:        435       3238   IO-APIC   15-edge      ata_piix
  24:        320      38951  Hyper-V PCIe MSI 134250496-edge      mlx4-async@pci:0001:00:02.0
  25:        163      25656  Hyper-V PCIe MSI 134250497-edge      mlx4-1@0001:00:02.0
  26:          0      31096  Hyper-V PCIe MSI 134250498-edge      mlx4-2@0001:00:02.0

FortiGate

On the FortiGate Next-Generation Firewall it is also possible to enable Accelerated Networking on specific version (6.2.1+, 6.0.6+ and 5.6.10). Once deployed it is advised to verify that Accelerated Networking is enabled on the VM using the different debug commands on the CLI.

First of all the virtual/pseudo networking interface needs to be active.

 # fnsysctl ifconfig -a
 lo       Link encap:Local Loopback
          LOOPBACK  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0  Bytes)  TX bytes:0 (0  Bytes)
  
 dummy0    Link encap:Ethernet  HWaddr A2:F8:2C:72:0A:32
          BROADCAST NOARP  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0  Bytes)  TX bytes:0 (0  Bytes)
  
 port1     Link encap:Ethernet  HWaddr 00:0D:3A:2C:5F:16
          inet addr:172.16.136.5  Bcast:172.16.136.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2652 errors:0 dropped:0 overruns:0 frame:0
          TX packets:280 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:767281 (749.3 KB)  TX bytes:61114 (59.7 KB)
  
 port2     Link encap:Ethernet  HWaddr 00:0D:3A:2D:5D:57
          inet addr:172.16.137.5  Bcast:172.16.137.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:107 errors:0 dropped:0 overruns:0 frame:0
          TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4494 (4.4 KB)  TX bytes:22938 (22.4 KB)
  
 root      Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:796 errors:0 dropped:0 overruns:0 frame:0
          TX packets:796 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:212916 (207.9 KB)  TX bytes:212916 (207.9 KB)
  
 ssl.root  Link encap:Unknown
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:2 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0  Bytes)  TX bytes:0 (0  Bytes)
  
 JVH55-1   Link encap:Unknown
          inet addr:169.254.121.1  Mask:255.255.255.255
          UP POINTOPOINT NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:3 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0  Bytes)  TX bytes:0 (0  Bytes)
  
 JVH55-2   Link encap:Unknown
          inet addr:169.254.121.1  Mask:255.255.255.255
          UP POINTOPOINT NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:3 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0  Bytes)  TX bytes:0 (0  Bytes)
  
 vsys_ha   Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0  Bytes)  TX bytes:0 (0  Bytes)
  
 port_ha   Link encap:Ethernet  HWaddr 00:40:F4:70:84:D2
          UP BROADCAST MULTICAST  MTU:1496  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:1 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0  Bytes)  TX bytes:0 (0  Bytes)
  
 vsys_fgfm Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0  Bytes)  TX bytes:0 (0  Bytes)
  
 sriovslv0 Link encap:Ethernet  HWaddr 00:0D:3A:2D:5D:57
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1931 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0  Bytes)  TX bytes:1764473 (1.7 MB)
  
 sriovslv1 Link encap:Ethernet  HWaddr 00:0D:3A:2C:5F:16
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:7616 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13050 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5813421 (5.5 MB)  TX bytes:2628354 (2.5 MB) 

Secondly the network interface device information should show the correct speed as in linux.

 # diag hardware deviceinfo nic port1
 Name:               port1
 Driver:             hv_netvsc
 Version:
 Bus:
 Hwaddr:             00:0d:3a:2c:5f:16
 Permanent Hwaddr:00:0d:3a:2c:5f:16
 State:              up
 Link:               up
 Mtu:                1500
 Supported:
 Advertised:
 Speed:              40000full
 Auto:               disabled
 RX Ring:            10486
 TX Ring:            192
 Rx packets:                  2666
 Rx bytes:           772077
 Rx compressed:                0
 Rx dropped:                  0
 Rx errors:                   0
   Rx Length err:     0
   Rx Buf overflow:   0
   Rx Crc err:                0
   Rx Frame err:               0
   Rx Fifo overrun:   0
   Rx Missed packets:          0
 Tx packets:                  280
 Tx bytes:           61114
 Tx compressed:                0
 Tx dropped:                  0
 Tx errors:                   0
   Tx Aborted err:    0
   Tx Carrier err:    0
   Tx Fifo overrun:   0
   Tx Heartbeat err:  0
   Tx Window err:     0
 Multicasts:                  0
 Collisions:                  0 

And finally the interrupts on the unit needs to be attached to the Accelerated Networking Mellanox driver .

 # diagnose hardware sysinfo interrupts
            CPU0       CPU1       CPU2       CPU3
   0:         48          0          0          2   IO-APIC-edge      timer
   1:          0          0          0          8   IO-APIC-edge      i8042
   4:          5          1          3          3   IO-APIC-edge      serial
   8:          0          0          0          0   IO-APIC-edge      rtc
   9:          0          0          0          0   IO-APIC-fasteoi   acpi
  12:          0          0          0          3   IO-APIC-edge      i8042
  14:          0          0          0          0   IO-APIC-edge      ata_piix
  15:          0          0          0          0   IO-APIC-edge      ata_piix
  40:      14887          0          0          0   PCI-MSI-edge      mlx4-async@pci:0002:00:02.0
  41:          0        453          0          0   PCI-MSI-edge      sriovslv0-0
  42:          0          0         89          0   PCI-MSI-edge      sriovslv0-1
  43:          0          0          0        695   PCI-MSI-edge      sriovslv0-2
  44:        635          0          0          0   PCI-MSI-edge      sriovslv0-3
  45:          0      14897          0          0   PCI-MSI-edge      mlx4-async@pci:0001:00:02.0
  46:          0          0       3553          0   PCI-MSI-edge      sriovslv1-0
  47:          0          0          0       2474   PCI-MSI-edge      sriovslv1-1
  48:       3997          0          0          0   PCI-MSI-edge      sriovslv1-2
  49:          0       3273          0          0   PCI-MSI-edge      sriovslv1-3
 NMI:          0          0          0          0   Non-maskable interrupts
 LOC:     420708     407948     409165     416109   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0   Performance monitoring interrupts
 IWI:          0          0          0          0   IRQ work interrupts
 RES:       7425       2919       2856       3375   Rescheduling interrupts
 CAL:        131        134        214        233   Function call interrupts
 TLB:         57         39         61         53   TLB shootdowns
 ERR:          0
 MIS:          0 
jvhoof Written by:

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.