Avi Vantage Prometheus Integration
Prometheus is an open-source systems monitoring and alerting toolkit that joined the Cloud Native Computing Foundation (CNCF) in 2016. Unlike other common monitoring solutions, Prometheus metrics collection uses a pull method utilizing HTTP.
For more information on Prometheus, refer to the Prometheus home page
This article explains the integration of Avi Vantage with Prometheus, and how to authenticate Prometheus scrape requests.
Prometheus API Spec
As of Avi Vantage release 18.2.6, the Avi Controller provides first class REST APIs to pull metrics (data in a format that can be ingested by Prometheus) for the following:
- The Controller cluster
- Virtual services
- Service engines
GET APIs to Fetch Prometheus Metrics
The GET API to fetch Prometheus metrics is as below:
entity_type is the controller, virtualservice, serviceengine or pool.
Valid query parameters are as follows:
entity_id- Depending on the entity type it can be a comma separated list of Controller node IDs, pool UUIDs, service engine UUIDs, virtual service UUIDs
entity_name- Depending on the
entity_nameis a comma separated list of valid pool, virtual service or service engine object names
metric_id- A comma separated list of valid metric IDs, defaults to all possible metric IDs. Use the API
GET/api/analytics/metric_idto find the valid metric IDs. The complete set of metrics is provided here.
tenant- A comma separated list of tenant names, by default set to admin
description- Attaches a line of description before every metric as a comment, This parameter is by default set to false
Using the API with Prometheus
Prometheus uses a pull method utilizing HTTP for metrics collection. The users provide HTTP endpoints to fetch data from, and other parameters like scraping interval, additional query params, port, authorization details etc.
Prometheus provides two ways to authorize the HTTP scrape requests:
- Basic auth
- Bearer token/Bearer token file
The Prometheus endpoints exposed as APIs from the Controller require authentication, similar to all other existing Controller APIs.
Avi Vantage provides two ways to authenticate prometheus scrape requests to the Controller:
1. Setting Basic Auth for Controller APIs
i. API calls from a client to the Avi Vantage Controller can be setup to use Basic Auth as the authentication mechanism for Controller APIs. To know how to enable basic auth on the Controller , refer to HTTP Basic Auth for API Queries.
ii. Use the
basic_auth parameter under
scrape_config to setup basic auth for prometheus scrape requests in the prometheus.yml file.
A sample prometheus.yml job config using a
basic_auth enabled controller setup is shown below:
- job_name: 'avi_vs' scrape_interval: 60s metrics_path: '/api/analytics/prometheus-metrics/virtualservice' params: entity_name: ['prod-vs1,prod-vs2'] tenant: ['admin,tenant1'] scheme: 'https' tls_config: insecure_skip_verify: true basic_auth: username: '[username]' password: '[password]' static_configs: - targets: ['[avi_controller_cluster]']
Note: Prometheus allows users to pass CA filepath, client certificate, and key files for TLS configuration under the
tls_config parameter. In the sample config, the certificate check has been skipped using the
2. Using avi-api-proxy
avi-api-proxy is a dockerized proxy container, built over Avi’s SDK which is responsible for handling authentication and session management with Avi Controller’s API server. All the HTTP REST API calls (GET, POST, PUT, DELETE) to
avi-api-proxy are forwarded to the Controller API server with valid authorization headers and cookies. The response from the Controller is returned back to the requesting client application, in this case the Prometheus setup.
A sample prometheus.yml job config using
avi-api-proxy is shown below:
- job_name: 'avi_vs' scrape_interval: 60s metrics_path: '/api/analytics/prometheus-metrics/virtualservice' scheme: 'http' params: entity_name: ['prod-vs1,prod-vs2'] tenant: ['admin,tenant1'] static_configs: - targets: ['127.0.0.1:8080']
avi-api-proxy is up and running, no API authentication details are provided to the Prometheus configuration file (refer to the sample prometheus.yml job config using
avi-api-proxy). The authentication to the Avi Controller API server is fully managed by
Note: The target for the scrape requests specified here is the address and port where
avi-api-proxy is running. In the sample provided, the
avi-api-proxy runs on the same network namespace as the Prometheus setup, on port 8080, hence the target 127.0.0.1:8080.
Click here to find the avi-api-proxy docker container image.
Note: The container requires the Avi Controller cluster address, username and password to get started.
Filtering Metrics Using Params
Add query params for an API endpoint in the prometheus config yaml using either one of the following:
metrics_pathfield : For instance
metrics_pathcan be specified with query params as follows:
paramskey: A more readable alternative for provided query params in a key value yaml format.
To minimize the response time of the API, it is advised to use the provided filters.
In case no params are provided, the API fetches data for all entity objects belonging to the admin tenant, consisting of all metric IDs, without description strings.
A sample prometheus job config displaying how to use query params is shown below. The following job config, fetches
l4_server.avg_connections_dropped metrics, for all pool objects in the admin and default tenant namespace, with a metric description line before the metric data.
- job_name: 'avi_pool' scrape_interval: 60s metrics_path: '/api/analytics/prometheus-metrics/pool' scheme: 'http' params: tenant: ['admin,default'] metric_id: ['l4_server.avg_bandwidth,l4_server.avg_connections_dropped'] description: 'true' static_configs: - targets: ['127.0.0.1:8080']
Further, to get metrics for specific pool objects in the admin and default tenants, use entity_name or entity_id params as follows:
params: entity_name: ['vs1-pool1,vs1-pool2,vs2-pool1'] tenant: ['admin,default'] metric_id: ['l4_server.avg_bandwidth,l4_server.avg_connections_dropped'] description: 'true'
Note: Avi Vantage provides a comprehensive set of metrics related to virtual service, pool, and service engine objects. When such objects are large in number, the Avi Controller might take some time to compute the analytics data and return the response. Setting a low
scrape_interval value might result in API timeouts. In such cases, use filter params in order to get any relevant data and set the
scrape_interval value to a minimum of 60 seconds in the Prometheus config yaml.
What Advantages does the Proxy Provide?
Changing the system settings to allow basic authentication, enables Basic Auth on all controller APIs. This might not be the intended behaviour in some scenarios.
Applications communicate with the proxy over standard HTTP REST protocol similar to what is provided by the Avi Controller. The proxy is integrated with Avi’s go SDK to provide a cookie based authentication and session management.
The proxy is instantiated using the Avi Controller credentials, and any application running the proxy (preferably in the same network namespace), and should be able to communicate to the Avi Controller via
The proxy provides the following advantages:
- Session management - The proxy uses Avi’s go SDK to manage session cookies, set auth headers, refresh tokens, and reuse TLS connection to reduce TLS handshakes.
- Retry requests - In case of server errors, the SDK tries to retry failed requests.
- Ease of use - No system settings change is required.
The connection between
avi-api-proxy and the application (in this case, Prometheus) is an open connection, and hence it is advised to run
avi-api-proxy in the same network namespace as the application, for the communication to happen over localhost.