Streaming Avi Vantage Client Logs to an External Server

Avi Vantage has a built-in indexing and searching service that provides analytics of the application traffic, as well as Avi Vantage system and configuration events. Some customers wish to incorporate the data into a pre-existing log management system (e.g., Splunk, Sumo Logic, or rsyslog/elasticsearch, etc.).

Starting in version 17.1, Avi Vantage can stream application logs directly to an external server. (We are referring to the logs typically visible in the Avi UI as shown below.) The logs are streamed as UDP messages directly from the Avi Service Engines. Customers can provide external server information in a new option under Analytics Profile, client_log_streaming_config. Traffic logs of any virtual service that uses such an analytics profile are automatically streamed from the Service Engine(s) on which that VS has been placed. Service Engines use their management interface to connect to a configured external server.

application log GUI display

Enabling Application Log Streaming via Avi CLI

Create a new AnalyticsProfile object or edit an existing one and set the following fields under the client_log_streaming_config subsection for streaming application logs:

  • external_server: The destination server IP address or hostname. If a name is provided, this should be resolvable on Avi Service Engines.
  • external_server_port: The destination server’s service port. The default for this is 514.
  • log_types_to_send: Type of logs to stream to the external server. Default is logs_all, i.e., send all logs. Other options are:
    • logs_significant_only: Only significant logs
    • logs_udf_only: Only logs that match any client log filters or rules with logging enabled
    • logs_udf_significant: Significant logs as well as logs that match any client log filters or rules with logging enabled
  • max_logs_per_second: Maximum number of logs per second streamed to the external server. By default, 100 logs per second are streamed. Set this to zero(0) to not enforce any limit.

    Caution: Please see the notes in “Rate Limiting” section below before making any changes to this variable.

    [admin:node-1]: > configure analyticsprofile streaming-profile
    [admin:node-1]: analyticsprofile> client_log_streaming_config
    [admin:node-1]: analyticsprofile:client_log_streaming_config> external_server
    [admin:node-1]: analyticsprofile:client_log_streaming_config> log_types_to_send logs_significant_only
    [admin:node-1]: analyticsprofile:client_log_streaming_config> max_logs_per_second 20
    [admin:node-1]: analyticsprofile:client_log_streaming_config> save
    [admin:node-1]: analyticsprofile> save
    | client_log_streaming_config                     |                                                       |
    |   external_server                               |                                          |
    |   external_server_port                          | 514                                                   |
    |   log_types_to_send                             | LOGS_SIGNIFICANT_ONLY                                 |
    |   max_logs_per_second                           | 20                                                    |
    [admin:node-1]: >

After making the changes above, traffic logs of any virtual service associated with this analytics profile will be streamed to the configured external server.

Enabling Application Log Streaming via Avi UI

Log into the Controller with sufficient administrative privilege to perform the following steps.

  • Navigate to Templates -> Profiles -> Analytics.
  • Create a new or select some pre-existing analytics profile to edit. The relevant settings for log streaming are at the very bottom.


Check the Stream Logs to an External Server option, default for which is OFF.

  • Complete the form, and click Save.


  • Apply the profile to those virtual services for which log data is to be streamed to the external server.

Rate limiting

As mentioned above, SEs use their management interface to stream application logs to a configured external server. Since the SE uses the same network interface to synchronize with the Avi Controller, it is necessary to ensure streaming log traffic does not interfere with the management traffic. To that end, Avi Vantage limits the rate of the streaming traffic to some number of log entries streamed per second. The default limit is 100 log entries per second. Though this rate can be changed in the configuration, one should be mindful that streaming logs consumes both SE CPU cycles and bandwidth on the management network.

Formatting of the Streamed Messages

Each log is streamed as a JSON-formatted string with no line-breaks.

Example layout:

{"adf": 1, "virtualservice": "virtualservice-4abd93ed-9d89-4ca2-813f-f1706285d7c7", "report_timestamp": "2017-05-01T15:10:08.798592", "service_engine": "", "vcpu_id": 1, "log_id": 5, "client_ip": "", "client_src_port": 41392, "client_dest_port": 9000, "client_rtt": 1, "http_version": "1.1", "method": "GET", "uri_path": "/notexist.html", "referer": "", "user_agent": "L7ProxyTest", "xff": "", "host": "", "persistent_session_id": 3472328296917460336, "response_content_type": "text/html", "request_length": 299, "cacheable": 1, "pool": "pool-16fd2f0c-01db-467a-b673-6faa076b9142", "pool_name": "l7pool1", "server_ip": "", "server_name": "", "server_conn_src_ip": "", "server_dest_port": 80, "server_src_port": 49003, "server_rtt": 16, "server_response_length": 1395, "server_response_code": 404, "server_response_time_first_byte": 1, "server_response_time_last_byte": 1, "response_length": 1397, "response_code": 404, "response_time_first_byte": 1, "response_time_last_byte": 1, "compression": "NO_COMPRESSION_CAN_BE_COMPRESSED", "client_insights": "NO_INSIGHTS_NOT_SAMPLED_TYPE", "request_headers": 689219, "response_headers": 13, "request_state": "AVI_HTTP_REQUEST_STATE_SEND_TO_CLIENT", "significant_log": "[ADF_RESPONSE_CODE_4XX]", "headers_sent_to_server": "X-Forwarded-For:  Host:  Accept-Encoding: identity  Accept: */*  User-Agent: L7ProxyTest  referer:  Authorization: Basic YXZpdXNlcjphdml1c2Vy    ", "headers_received_from_server": "Server: nginx/1.2.1  Date: Mon, 01 May 2017 15:15:24 GMT  Content-Type: text/html  Content-Length: 1242  Connection: keep-alive  ", "server_connection_reused": 1, "vs_ip": "", "body_updated": "NOT_UPDATED", "vs_name": "l7vs1"}

Every log contains a field named report_timestamp, that denotes the time at which that log was generated at the corresponding Service Engine.

Splunk as the external server

Splunk can be configured to receive UDP messages on port 514. Please refer to the documentation.

 ./splunk add udp 514 -sourcetype syslog 

We recommend using syslog as the source type to properly interpret the single-line JSON string streamed for each log.

By default, Splunk would timestamp each received log with a timestamp corresponding to the time at which Splunk received that log. To force Splunk to use the report_timestamp in the log content as the timestamp for the log, please set the following configuration in props.conf :

TIME_PREFIX = \"report_timestamp\":\ \"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%5N

Please refer to the documentation for more details.

Screenshot from a Splunk Server:

Example of Avi Vantage Streaming to Splunk