Port Channel for Cisco Cloud Services Platform (CSP) 2100

To improve available network bandwidth and fault tolerance, multiple virtual function (VF) network interfaces assigned to Avi Service Engines can be aggregated to form port channels or bonded interfaces.

Topology

Topology

Prerequisites

  • No port channel specific configuration is needed on the switch.
  • To achieve the intended benefit, port channel member interfaces should be associated with different physical function (PF) devices or pNICs.
  • CSP configuration
  • SRIOV should be enabled on CSP pNICs being used. The following CSP command could be used for this.
avinet-1(config)# pnic enp7s0f0 sr-iov numvfs 32 switchmode vep

Configuration

The member (slave) interfaces to be bonded together to form a port channel (bond) can be configured using Avi SE metadata (yml) file as follows.


avi.bond-ifs.SE: '1,2'

The above configuration directs the Service Engine to bond together interfaces 1 and 2 to form a port channel.

If multiple port channels are required, the indices of member interfaces for different port channels should be specified with space in-between as shown below.


avi.bond-ifs.SE: '1,2 3,4'

The above configuration makes the SE create two port channels - one with interfaces 1 and 2, and the other with interfaces 3 and 4.

Once the SE is up, the Edit Service Engine page in the Avi UI shows information about bonded and slave interfaces.

Avi Service Engine Editor

Run the CLI command show serviceengine <SE_IP_Address> interface to view the interface configuration.


[admin:10-128-2-20]: > show serviceengine 10.128.2.22 interface
+------------------------------------+-----------------------------------------------------+
| Field                              | Value                                               |
+------------------------------------+-----------------------------------------------------+
| se_uuid                            | 10-128-2-22:se-bb2c9806-fe17-4c66-9fb0-dd5b89a456f8 |
| proc_id                            | PROC_Aggregate                                      |
| vnics[1]                           |
|   vnic_name                        | eth2                                                |
|   vnic_id                          | 4                                                   |
|   mac_address                      | 02:5a:80:02:11:0c                                   |
|   interface_stats                  |                                                     |
|     ipackets                       | 69180                                               |
|     ibytes                         | 4446364                                             |
|     ierrors                        | 0                                                   |
|     rx_nombuf                      | 0                                                   |
|     opackets                       | 0                                                   |
|     obytes                         | 0                                                   |
|     oerrors                        | 0                                                   |
|     rx_kni                         | 0                                                   |
|     tx_kni                         | 2175                                                |
|     tx_kni_errs                    | 0                                                   |
|     ip_checksum_drops              | 0                                                   |
|     l4_checksum_drops              | 0                                                   |
|     tx_queue_full_retries          | 0                                                   |
|     rx_pkt_iterations              | 64223                                               |
|     rx_queue_full                  | 0                                                   |
|     rx_max_single_burst            | 22                                                  |
|     rx_mim_etype_p2s               | 0                                                   |
|     tx_mim_etype_p2s               | 0                                                   |
|     tx_mim_frags_etype_p2s         | 0                                                   |
|     rx_mim_etype_s2p               | 0                                                   |
|     tx_mim_etype_s2p               | 0                                                   |
|     tx_mim_frags_etype_s2p         | 0                                                   |
|     tx_frags_p2s                   | 0                                                   |
|     local_flow_probes_req_sent     | 0                                                   |
|     local_flow_probes_req_received | 0                                                   |
|     flow_probes_ignored_same_vnic  | 0                                                   |
|     flow_probes_ignored_in_tw      | 0                                                   |
|   vnic_parent                      | 4                                                   |
|   vnic_weight                      | -1                                                  |
|   vlan_id                          | 0                                                   |
|   num_vs_delete_drops              | 0                                                   |
|   vrf_id                           | 1                                                   |
|   vrf_ref                          | global                                              |
|   vnic_mtu                         | 1500                                                |
|   interface_disabled               | False                                               |
|   linux_intf_name                  | eth2                                                |
|   vnic_owner                       | 0                                                   |
| vnics[2]                           |                                                     |
|   vnic_name                        | eth1                                                |
|   vnic_id                          | 5                                                   |
|   mac_address                      | 02:4a:80:02:11:0c                                   |
|   interface_stats                  |                                                     |
|     ipackets                       | 611590                                              |
|     ibytes                         | 59185414                                            |
|     ierrors                        | 0                                                   |
|     rx_nombuf                      | 0                                                   |
|     opackets                       | 542345                                              |
|     obytes                         | 65622481                                            |
|     oerrors                        | 0                                                   |
|     rx_kni                         | 0                                                   |
|     tx_kni                         | 2175                                                |
|     tx_kni_errs                    | 0                                                   |
|     ip_checksum_drops              | 0                                                   |
|     l4_checksum_drops              | 0                                                   |
|     tx_queue_full_retries          | 0                                                   |
|     rx_pkt_iterations              | 337424                                              |
|     rx_queue_full                  | 0                                                   |
|     rx_max_single_burst            | 22                                                  |
|     rx_mim_etype_p2s               | 0                                                   |
|     tx_mim_etype_p2s               | 0                                                   |
|     tx_mim_frags_etype_p2s         | 0                                                   |
|     rx_mim_etype_s2p               | 0                                                   |
|     tx_mim_etype_s2p               | 0                                                   |
|     tx_mim_frags_etype_s2p         | 0                                                   |
|     tx_frags_p2s                   | 0                                                   |
|     local_flow_probes_req_sent     | 0                                                   |
|     local_flow_probes_req_received | 0                                                   |
|     flow_probes_ignored_same_vnic  | 0                                                   |
|     flow_probes_ignored_in_tw      | 0                                                   |
|   vnic_parent                      | 5                                                   |
|   vnic_weight                      | -1                                                  |
|   vlan_id                          | 0                                                   |
|   num_vs_delete_drops              | 0                                                   |
|   vrf_id                           | 1                                                   |
|   vrf_ref                          | global                                              |
|   vnic_mtu                         | 1500                                                |
|   interface_disabled               | False                                               |
|   linux_intf_name                  | eth1                                                |
|   vnic_owner                       | 0                                                   |
| vnics[3]                           |                                                     |
|   vnic_name                        | bond0                                               |
|   vnic_id                          | 32                                                  |
|   mac_address                      | 02:5a:80:02:11:0c                                   |
|   ip_info[1]                       |                                                     |
|     ip_addr                        | 10.160.100.246                                      |
|     net_mask                       | 255.255.255.0                                       |
|   ip_info[2]                       |                                                     |
|     ip_addr                        | 10.160.100.235                                      |
|     net_mask                       | 255.255.255.255                                     |
|   ip_info[3]                       |                                                     |
|     ip_addr                        | 10.160.100.236                                      |
|     net_mask                       | 255.255.255.255                                     |
|   interface_stats                  |                                                     |
|     ipackets                       | 680768                                              |
|     ibytes                         | 63631658                                            |
|     ierrors                        | 0                                                   |
|     rx_nombuf                      | 0                                                   |
|     opackets                       | 542345                                              |
|     obytes                         | 65622481                                            |
|     oerrors                        | 0                                                   |
|     rx_kni                         | 0                                                   |
|     tx_kni                         | 0                                                   |
|     tx_kni_errs                    | 0                                                   |
|     ip_checksum_drops              | 0                                                   |
|     l4_checksum_drops              | 0                                                   |
|     tx_queue_full_retries          | 0                                                   |
|     rx_pkt_iterations              | 0                                                   |
|     rx_queue_full                  | 0                                                   |
|     rx_max_single_burst            | 0                                                   |
|     rx_mim_etype_p2s               | 0                                                   |
|     tx_mim_etype_p2s               | 0                                                   |
|     tx_mim_frags_etype_p2s         | 0                                                   |
|     rx_mim_etype_s2p               | 0                                                   |
|     tx_mim_etype_s2p               | 0                                                   |
|     tx_mim_frags_etype_s2p         | 0                                                   |
|     tx_frags_p2s                   | 0                                                   |
|     local_flow_probes_req_sent     | 0
|     local_flow_probes_req_received | 0                                                   |
|     flow_probes_ignored_same_vnic  | 0                                                   |
|     flow_probes_ignored_in_tw      | 0                                                   |
|   vnic_parent                      | 32                                                  |
|   vnic_weight                      | -1                                                  |
|   vlan_id                          | 0                                                   |
|   num_vs_delete_drops              | 0                                                   |
|   vrf_id                           | 1                                                   |
|   vrf_ref                          | global                                              |
|   vnic_mtu                         | 1500                                                |
|   mbr_intfs[1]                     |                                                     |
|     if_name                        | eth2                                                |
|     linux_name                     | eth2                                                |
|     active                         | True                                                |
|   mbr_intfs[2]                     |                                                     |
|     if_name                        | eth1                                                |
|     linux_name                     | eth1                                                |
|     active                         | True                                                |
|   interface_disabled               | False                                               |
|   linux_intf_name                  | bond0                                               |
|   vnic_owner                       | 0                                                   |
+------------------------------------+-----------------------------------------------------+

Troubleshooting

1. Linux interface configuration

After port channel has been successfully configured, the Linux network interface configuration file (/etc/network/interfaces) includes information about bonded and slave interfaces as shown in the example below.

When the Avi SE metadata file has avi.bond-ifs.SE: '1,2', the SE makes Linux form a bonding interface (bond0) with interfaces eth1 and eth2 as slaves. The following configuration features in the interface configuration file (/etc/network/interfaces) in the default namespace.

auto eth1
    iface eth1 inet manual
    bond-master bond0

auto eth2
    iface eth2 inet manual
    bond-master bond0

auto bond0
    iface bond0 inet manual
    bond-mode balance-xor
    bond-miimon 100
    bond-slaves eth1 eth2

2. Interface configuration inside Avi Vantage namespace

Inside the namespace (avi_ns1), ip addr gives the following information with the bonded and slave interfaces using the same MAC address.


avi_bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet YY.YY.YY.YY/ZZ scope global avi_bond0
       valid_lft forever preferred_lft forever

avi_eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master avi_bond0 state UP group default qlen 1000
    link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff

avi_eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master avi_bond0 state UP group default qlen 1000
    link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff

3. Bond configuration inside Avi Vantage namespace

Inside the namespace (avi_ns1), /proc/net/bonding/avi_bond0 gives the information about bond configuration.


Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (xor)
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: avi_eth2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 02:5a:80:02:11:0c
Slave queue ID: 0

Slave Interface: avi_eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 02:4a:80:02:11:0c
Slave queue ID: 0
  • MII status shows the physical link status of the interfaces.
  • Starting with release 17.2.4, bond mode can be balance-rr, active-backup, or balance-xor. Previously, only xor was supported.

    Example: avi.bond-mode.SE: 'active-backup'

    This would set the port-channel mode to active-backup.

Notes:

  1. balance-xor is the default port-channel mode.
  2. If active-backup is used as the port-channel mode, then the first member vNIC is chosen as the primary link.
  • Permanent hardware addresses of the slaves are the original MAC addresses and should be different for each of the slaves.

4. Statistics

Statistics of interfaces in the output of show serviceengine <SE_IP_Address> interface can be used to see if traffic is being distributed fairly among the slave interfaces. A balanced distribution results in comparable statistics for all slave interfaces if all of them were active together during their lifetime.

5. CSP pNIC and SRIOV Configuration

The following commands in CSP shell help identify errors in pNIC and SRIOV configuration in CSP.

  • SRIOV status on pNICs helps make sure that SRIOV is on for pNICs of interest.

avinet-1# show running-config pnic sr-iov
pnic enp7s0f0
 sr-iov numvfs 32
 sr-iov switchmode veb
!
pnic enp7s0f1
 sr-iov numvfs 32
 sr-iov switchmode veb
!
avinet-1# show pnic enp7s0f0 passthrough
passthrough sriov
avinet-1# show pnic enp7s0f0 sriov_intf
sriov_intf sriov_pf
  • View the virtual function usage on a pNIC as shown below.

avinet-1# show pnic enp7s0f0 vfnics
VF              INUSE     SERVICE
IDX  NAME       FLAG      OWNER
-----------------------------------
0    enp7s16    in-use    none
1    enp7s16f2  in-use    none
2    enp7s16f4  in-use    none
3    enp7s16f6  in-use    none
4    enp7s17    in-use    none
5    enp7s17f2  in-use    none
6    enp7s17f4  in-use    none
7    enp7s17f6  not-used  none
8    enp7s18    not-used  none
9    enp7s18f2  not-used  none
10   enp7s18f4  not-used  none
11   enp7s18f6  not-used  none
12   enp7s19    not-used  none
13   enp7s19f2  not-used  none
14   enp7s19f4  not-used  none
15   enp7s19f6  not-used  none
16   enp7s20    not-used  none
17   enp7s20f2  not-used  none
18   enp7s20f4  not-used  none
19   enp7s20f6  not-used  none
20   enp7s21    not-used  none
21   enp7s21f2  not-used  none
22   enp7s21f4  not-used  none
23   enp7s21f6  not-used  none
24   enp7s22    not-used  none
25   enp7s22f2  not-used  none
26   enp7s22f4  not-used  none
27   enp7s22f6  not-used  none
28   enp7s23    not-used  none
29   enp7s23f2  not-used  none
30   enp7s23f4  not-used  none
31   enp7s23f6  not-used  none