Port Channel for Cisco Cloud Services Platform (CSP) 2100
Overview
To improve available network bandwidth and fault tolerance, multiple virtual function (VF) network interfaces assigned to Avi Service Engines can be aggregated to form port channels or bonded interfaces.
Topology
Prerequisites
- No port channel specific configuration is needed on the switch.
- To achieve the intended benefit, port channel member interfaces should be associated with different physical function (PF) devices or pNICs.
- CSP configuration
- SRIOV should be enabled on CSP pNICs being used. The following CSP command could be used for this.
avinet-1(config)# pnic enp7s0f0 sr-iov numvfs 32 switchmode vep
Configuration
The member (secondary) interfaces to be bonded together to form a port channel (bond) can be configured using Avi SE metadata (yml) file as follows.
avi.bond-ifs.SE: '1,2'
The above configuration directs the Service Engine to bond together interfaces 1 and 2 to form a port channel.
If multiple port channels are required, the indices of member interfaces for different port channels should be specified with space in-between as shown below.
avi.bond-ifs.SE: '1,2 3,4'
The above configuration makes the SE create two port channels - one with interfaces 1 and 2, and the other with interfaces 3 and 4.
Once the SE is up, the Edit Service Engine page in the Avi UI shows information about bonded and secondary interfaces.
Run the CLI command show serviceengine <SE_IP_Address> interface to view the interface configuration.
[admin:10-128-2-20]: > show serviceengine 10.128.2.22 interface
+------------------------------------+-----------------------------------------------------+
| Field | Value |
+------------------------------------+-----------------------------------------------------+
| se_uuid | 10-128-2-22:se-bb2c9806-fe17-4c66-9fb0-dd5b89a456f8 |
| proc_id | PROC_Aggregate |
| vnics[1] |
| vnic_name | eth2 |
| vnic_id | 4 |
| mac_address | 02:5a:80:02:11:0c |
| interface_stats | |
| ipackets | 69180 |
| ibytes | 4446364 |
| ierrors | 0 |
| rx_nombuf | 0 |
| opackets | 0 |
| obytes | 0 |
| oerrors | 0 |
| rx_kni | 0 |
| tx_kni | 2175 |
| tx_kni_errs | 0 |
| ip_checksum_drops | 0 |
| l4_checksum_drops | 0 |
| tx_queue_full_retries | 0 |
| rx_pkt_iterations | 64223 |
| rx_queue_full | 0 |
| rx_max_single_burst | 22 |
| rx_mim_etype_p2s | 0 |
| tx_mim_etype_p2s | 0 |
| tx_mim_frags_etype_p2s | 0 |
| rx_mim_etype_s2p | 0 |
| tx_mim_etype_s2p | 0 |
| tx_mim_frags_etype_s2p | 0 |
| tx_frags_p2s | 0 |
| local_flow_probes_req_sent | 0 |
| local_flow_probes_req_received | 0 |
| flow_probes_ignored_same_vnic | 0 |
| flow_probes_ignored_in_tw | 0 |
| vnic_parent | 4 |
| vnic_weight | -1 |
| vlan_id | 0 |
| num_vs_delete_drops | 0 |
| vrf_id | 1 |
| vrf_ref | global |
| vnic_mtu | 1500 |
| interface_disabled | False |
| linux_intf_name | eth2 |
| vnic_owner | 0 |
| vnics[2] | |
| vnic_name | eth1 |
| vnic_id | 5 |
| mac_address | 02:4a:80:02:11:0c |
| interface_stats | |
| ipackets | 611590 |
| ibytes | 59185414 |
| ierrors | 0 |
| rx_nombuf | 0 |
| opackets | 542345 |
| obytes | 65622481 |
| oerrors | 0 |
| rx_kni | 0 |
| tx_kni | 2175 |
| tx_kni_errs | 0 |
| ip_checksum_drops | 0 |
| l4_checksum_drops | 0 |
| tx_queue_full_retries | 0 |
| rx_pkt_iterations | 337424 |
| rx_queue_full | 0 |
| rx_max_single_burst | 22 |
| rx_mim_etype_p2s | 0 |
| tx_mim_etype_p2s | 0 |
| tx_mim_frags_etype_p2s | 0 |
| rx_mim_etype_s2p | 0 |
| tx_mim_etype_s2p | 0 |
| tx_mim_frags_etype_s2p | 0 |
| tx_frags_p2s | 0 |
| local_flow_probes_req_sent | 0 |
| local_flow_probes_req_received | 0 |
| flow_probes_ignored_same_vnic | 0 |
| flow_probes_ignored_in_tw | 0 |
| vnic_parent | 5 |
| vnic_weight | -1 |
| vlan_id | 0 |
| num_vs_delete_drops | 0 |
| vrf_id | 1 |
| vrf_ref | global |
| vnic_mtu | 1500 |
| interface_disabled | False |
| linux_intf_name | eth1 |
| vnic_owner | 0 |
| vnics[3] | |
| vnic_name | bond0 |
| vnic_id | 32 |
| mac_address | 02:5a:80:02:11:0c |
| ip_info[1] | |
| ip_addr | 10.160.100.246 |
| net_mask | 255.255.255.0 |
| ip_info[2] | |
| ip_addr | 10.160.100.235 |
| net_mask | 255.255.255.255 |
| ip_info[3] | |
| ip_addr | 10.160.100.236 |
| net_mask | 255.255.255.255 |
| interface_stats | |
| ipackets | 680768 |
| ibytes | 63631658 |
| ierrors | 0 |
| rx_nombuf | 0 |
| opackets | 542345 |
| obytes | 65622481 |
| oerrors | 0 |
| rx_kni | 0 |
| tx_kni | 0 |
| tx_kni_errs | 0 |
| ip_checksum_drops | 0 |
| l4_checksum_drops | 0 |
| tx_queue_full_retries | 0 |
| rx_pkt_iterations | 0 |
| rx_queue_full | 0 |
| rx_max_single_burst | 0 |
| rx_mim_etype_p2s | 0 |
| tx_mim_etype_p2s | 0 |
| tx_mim_frags_etype_p2s | 0 |
| rx_mim_etype_s2p | 0 |
| tx_mim_etype_s2p | 0 |
| tx_mim_frags_etype_s2p | 0 |
| tx_frags_p2s | 0 |
| local_flow_probes_req_sent | 0
| local_flow_probes_req_received | 0 |
| flow_probes_ignored_same_vnic | 0 |
| flow_probes_ignored_in_tw | 0 |
| vnic_parent | 32 |
| vnic_weight | -1 |
| vlan_id | 0 |
| num_vs_delete_drops | 0 |
| vrf_id | 1 |
| vrf_ref | global |
| vnic_mtu | 1500 |
| mbr_intfs[1] | |
| if_name | eth2 |
| linux_name | eth2 |
| active | True |
| mbr_intfs[2] | |
| if_name | eth1 |
| linux_name | eth1 |
| active | True |
| interface_disabled | False |
| linux_intf_name | bond0 |
| vnic_owner | 0 |
+------------------------------------+-----------------------------------------------------+
Troubleshooting
1. Linux interface configuration
After port channel has been successfully configured, the Linux network interface configuration file (/etc/network/interfaces) includes information about bonded and secondary interfaces as shown in the example below.
When the Avi SE metadata file has avi.bond-ifs.SE: '1,2', the SE makes Linux form a bonding interface (bond0) with interfaces eth1 and eth2 as secondary. The following configuration features in the interface configuration file (/etc/network/interfaces) in the default namespace.
auto eth1
iface eth1 inet manual
bond-master bond0
auto eth2
iface eth2 inet manual
bond-master bond0
auto bond0
iface bond0 inet manual
bond-mode balance-xor
bond-miimon 100
bond-slaves eth1 eth2
2. Interface configuration inside Avi Vantage namespace
Inside the namespace (avi_ns1), ip addr gives the following information with the bonded and secondary interfaces using the same MAC address.
avi_bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
inet YY.YY.YY.YY/ZZ scope global avi_bond0
valid_lft forever preferred_lft forever
avi_eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master avi_bond0 state UP group default qlen 1000
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
avi_eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master avi_bond0 state UP group default qlen 1000
link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
3. Bond configuration inside Avi Vantage namespace
Inside the namespace (avi_ns1), /proc/net/bonding/avi_bond0 gives the information about bond configuration.
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: load balancing (xor)
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: avi_eth2
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 02:5a:80:02:11:0c
Slave queue ID: 0
Slave Interface: avi_eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 02:4a:80:02:11:0c
Slave queue ID: 0
- MII status shows the physical link status of the interfaces.
- Starting with release 17.2.4, bond mode can be
balance-rr,active-backup, orbalance-xor. Previously, only xor was supported.
Example:avi.bond-mode.SE: 'active-backup'
This would set the port-channel mode toactive-backup.
Notes:
balance-xoris the default port-channel mode.- If
active-backupis used as the port-channel mode, then the first member vNIC is chosen as the primary link.
- Permanent hardware addresses of the secondaries are the original MAC addresses and should be different for each of the secondaries.
4. Statistics
Statistics of interfaces in the output of show service engine <SE_IP_Address> interface can be used to see if traffic is being distributed fairly among the secondary interfaces. A balanced distribution results in comparable statistics for all secondary interfaces if all of them were active together during their lifetime.
5. CSP pNIC and SRIOV Configuration
The following commands in CSP shell help identify errors in pNIC and SRIOV configuration in CSP.
- SRIOV status on pNICs helps make sure that SRIOV is on for pNICs of interest.
avinet-1# show running-config pnic sr-iov
pnic enp7s0f0
sr-iov numvfs 32
sr-iov switchmode veb
!
pnic enp7s0f1
sr-iov numvfs 32
sr-iov switchmode veb
!
avinet-1# show pnic enp7s0f0 passthrough
passthrough sriov
avinet-1# show pnic enp7s0f0 sriov_intf
sriov_intf sriov_pf
- View the virtual function usage on a pNIC as shown below.
avinet-1# show pnic enp7s0f0 vfnics
VF INUSE SERVICE
IDX NAME FLAG OWNER
-----------------------------------
0 enp7s16 in-use none
1 enp7s16f2 in-use none
2 enp7s16f4 in-use none
3 enp7s16f6 in-use none
4 enp7s17 in-use none
5 enp7s17f2 in-use none
6 enp7s17f4 in-use none
7 enp7s17f6 not-used none
8 enp7s18 not-used none
9 enp7s18f2 not-used none
10 enp7s18f4 not-used none
11 enp7s18f6 not-used none
12 enp7s19 not-used none
13 enp7s19f2 not-used none
14 enp7s19f4 not-used none
15 enp7s19f6 not-used none
16 enp7s20 not-used none
17 enp7s20f2 not-used none
18 enp7s20f4 not-used none
19 enp7s20f6 not-used none
20 enp7s21 not-used none
21 enp7s21f2 not-used none
22 enp7s21f4 not-used none
23 enp7s21f6 not-used none
24 enp7s22 not-used none
25 enp7s22f2 not-used none
26 enp7s22f4 not-used none
27 enp7s22f6 not-used none
28 enp7s23 not-used none
29 enp7s23f2 not-used none
30 enp7s23f4 not-used none
31 enp7s23f6 not-used none