Packet Capture

Overview

Virtual service logs are available on Avi Vantage for troubleshooting of connection or traffic data. However, some troubleshooting requires full visibility into the packet transmission. Avi Vantage provides a packet capture feature, which runs TCPdump against a designated virtual service or an Avi Service Engine.

Virtual services may exist on a single Avi Service Engine (SE) or may be scaled out across multiple active Avi SEs. Traffic captures will automatically be executed on all Avi SEs actively handling traffic for the virtual service. Once the captures are complete, the Avi SEs forward their pcap files to the Controller, which aggregates and sorts the data into a single file containing client and server sides of Vantage.

Note: Starting with release 18.1.2, this feature is supported for IPv6 in Avi Vantage.

Configuring Avi Vantage for Packet Capture

The following sections describe how to configure and start a packet capture.

Note: It is highly recommended to set a limit to the duration of the capture. This limit may be either the maximum number of packets to receive or the duration of time, in minutes. Once the limit is reached, the capture will terminate and be sent to the Avi Controller for processing and should be available shortly after.

Capturing Traffic for a Virtual Service Using Avi UI

Navigate to Operations > Traffic Capture. The Capture Settings section shows the parameters defined for any captures that are currently in progress. To begin a new capture, select the edit icon.

packet-capture2

  • Select Virtual Service:  The capture is executed against traffic for a virtual service. The capture includes both the client-to-SE and SE-to-server sides of the connection. It will automatically be performed on all Avi SEs handling traffic for the virtual service.
  • Filter to Capture
    • All Traffic: Captures all traffic.
    • Choose Client IP, IP Range, Subnet Mask: Captures traffic only for the specified IP address, list or range of IP addresses, or subnet. The IP addresses can be client or server addresses.
      • To specify a list, use a space between each address. For example: 10.1.1.1 10.1.1.99 192.168.8.200
      • To specify a range, use the following format: 10.1.1.1-10.1.1.255
      • To specify a subnet, use the following format: 10.1.1.1/24.
    • Number of Packets: Maximum number of packets to capture, per core.
    • Duration: Number of minutes during which to capture packets.
    • Size of Packets: Number of bytes to capture per packet. This is similar to the snaplen option in TCPdump. To capture entire packets, enter 0.
  • Advanced Settings
    • Health Monitor Options: Indicates whether to capture packets from health checks.
      • None: Excludes health monitoring traffic (the default)
      • Include: Includes health monitoring traffic
      • Only: Captures only health monitoring traffic

Note: If ‘Include’ or ‘Only’ are selected, then Avi Vantage will capture all SE health monitor traffic for the SEs in which the VS is hosted, including other VS/pool health monitor traffic.

When the capture is started, the Capture Settings section indicates the progress of the capture. Once the capture is complete, it may still take a few minutes for the new capture to show up in the Completed Captures on Avi UI, as the Avi Controller may need to collate data from multiple Avi SEs and format the data, which is output as a pcap file.

Note: Enabling session key capture is documented in this article

Completed Captures

Once a traffic capture has completed, it will show in the Completed Captures table. This table shows the date, virtual service name, and size. The far right column of the table contains an export icon, which allows administrators to download the pcap file. This file type can be viewed by common traffic capture utilities such as Wireshark.

Capturing Traffic for a Virtual Service Using AVI CLI

To capture packets using the Avi CLI, log into the shell prompt, then enter the packet capture sub-mode for the desired virtual service:

debug virtualservice Test-virtual service
Updating an existing object. Currently, the object is:
+-------+--------------------+
| Field | Value |
+-------+--------------------+
| uuid | virtualservice-0-1 |
| name | Test-virtual service |
+-------+--------------------+

Parameters may be defined for the packet capture. By default, the capture is performed within the context of the selected virtual service. It is also performed on all Avi SEs that are handling the virtual service traffic and includes the packets from the client and server side of the Avi SE.

capture_params duration Time, in minutes. Default is unlimited.
capture_params num_pkts Maximum number of packets to collect. Default is unlimited.
capture_params pkt_size Packet size, or snap length, to capture. Default is unlimited.
debug_ip addrs IPv4 Address format
debug_ip prefixes IPv4 Prefix format <x.x.x.x/x>
debug_virtual service_hm_include Include health monitor packets in the capture
debug_virtual service_hm_none Omit health monitor packets from the capture (the default)
debug_virtual service_hm_only Capture only health monitor packets

The debug_ip command enters a sub-mode. This allows multiple IP addresses or IP subnets to be entered. (Omit the debug_ip option for for subsequent entries.) Save to commit the desired IPs and return to the previous menu.

Note:  By default, no maximum packets or duration of time to be captured are defined. It is recommended to include a maximum packet capture as shown in the following example. Without a limit, the capture will run until filling the Avi SE drive, potentially disrupting service.

Specify parameters, including the maximum number of packets to capture:

debugvirtualservice> capture_params num_pkts 1000
debugvirtualservice> debug_ip addrs 10.10.10.10
debugvirtualservice:debug_ip> save

Begin capturing based on the previously configured parameters:

debugvirtualservice> capture
debugvirtualservice> save
+----------------+--------------------+
| Field | Value |
+----------------+--------------------+
| uuid | virtualservice-0-1 |
| name | Test-VS |
| debug_ip | |
| addrs[1] | 10.10.10.10 |
| capture | True |
| capture_params | |
| duration | 0 mins |
| num_pkts | 1000 |
+----------------+--------------------+

Re-enter the packet capture sub-mode and stop an ongoing packet capture:

debug virtualservice Test-virtual service
debugvirtualservice> no capture
debugvirtualservice> save

Exporting Packet Capture for Analysis

Export the packet capture to a remote system that can view it via a tool such as TCPdump or Wireshark:

show debug virtualservice Test-virtual service capture
Please specify the destination directory: /tmp
Downloaded the attachment to /tmp/virtual service_virtualservice.20141205_192033.pcap
bash
scp /tmp/virtual service_virtualservice.192033.pcap user@10.1.1.1:/tmp

Capturing Traffic for an Avi SE using Avi CLI

Starting with Avi Vantage release 18.2.1, packet capture is available for Avi Service Engines as well.

Login to the shell using Avi CLI and then enter into the packet capture sub-mode for the Avi Service Engine:

To start packet capture for an Avi SE, use the debug serviceengine <SE IP address> command.


[admin:cntrl1]: > debug serviceengine 10.10.22.107
Updating an existing object. Currently, the object is:
+------------+--------------------------+
| Field      | Value                    |
+------------+--------------------------+
| uuid       | se-10.10.22.107-avitag-1 |
| name       | 10.10.22.107             |
| tenant_ref | admin                    |
+------------+--------------------------+

Filtering Packet Capture Based on IP Address

Execute the debug_ip addrs <IP address for filter> command from the debugserviceengine mode to filter the SE packet capture for the specific IP address.


 [admin:cntrl1]: debugserviceengine> debug_ip addrs 10.10.10.10
 [admin:cntrl1]: debugserviceengine:debug_ip>
 [admin:cntrl1]: debugserviceengine:debug_ip> save
 [admin:cntrl1]: debugserviceengine> where
 Tenant: admin
 +----------------+--------------------------+
 | Field          | Value                    |
 +----------------+--------------------------+
 | uuid           | se-10.10.22.107-avitag-1 |
 | name           | 10.10.22.107             |
 | debug_ip       |                          |
 |   addrs[1]     | 10.10.10.10              |
 | tenant_ref     | admin                    |
 +----------------+--------------------------+
 

Filtering Packet Capture Based on Duration for the Capture

Execute the capture_params duration <duration in minutes> command from the debugserviceengine mode to capture packets for the specified duration.


 [admin:cntrl1]: debugserviceengine> capture_params duration 10
 [admin:cntrl1]: debugserviceengine> where
 Tenant: admin
 +----------------+--------------------------+
 | Field          | Value                    |
 +----------------+--------------------------+
 | uuid           | se-10.10.22.107-avitag-1 |
 | name           | 10.10.22.107             |
 | debug_ip       |                          |
 |   addrs[1]     | 10.10.10.10              |
 | capture_params |                          |
 |   duration     | 10 min                   |
 | tenant_ref     | admin                    |
 +----------------+--------------------------+

Filtering Packet Capture Based on Packet Size for the Capture

Execute the capture_params pkt_size <packet size in bytes> command from the debugserviceengine mode to capture traffic of the desired packet size.




[admin:cntrl1]: debugserviceengine> capture_params pkt_size 0
[admin:cntrl1]: debugserviceengine> where
Tenant: admin
+----------------+--------------------------+
| Field          | Value                    |
+----------------+--------------------------+
| uuid           | se-10.10.22.107-avitag-1 |
| name           | 10.10.22.107             |
| debug_ip       |                          |
|   addrs[1]     | 10.10.10.10              |
| capture_params |                          |
|   pkt_size     | 0 bytes                  |
|   duration     | 10 min                   |
| tenant_ref     | admin                    |
+----------------+--------------------------+
[admin:cntrl1]: debugserviceengine> save
[admin:cntrl1]:

The above packet filter captures traffic for the Service Engine 10.10.22.107 with the IP address 10.10.10.10 for a duration of 10 minutes with packet size 0.

To stop the ongoing packet capture, re-enter the packet capture sub-mode and execute no capture command.


[admin:cntrl1]: debug service engine 10.10.22.107
[admin:cntrl1]: debugserviceengine> no capture
[admin:cntrl1]: debugserviceengine> save

For analysis, export the packet capture to a remote system and use tools such as TCPdump or Wireshark for further analysis. For more information on exporting packet capture, refer to Exporting Packet Capture for Analysis section of this article.