Avi Vantage Design Considerations with Cisco ACI

Overview

Cisco Application Centric Infrastructure (ACI) is a software defined networking solution offered by Cisco for data centers and clouds which helps in increasing operational efficiency, delivering network automation, and improving security for any combination of on-premises data centers, private, and public clouds.

The Avi Vantage Platform provides enterprise-grade distributed ADC and iWAF (Intelligent Web Application Firewall) solutions for on-premises and public-cloud infrastructure. Avi Vantage also provides inbuilt analytics that enhances the end-user application experience as well as ease of operations for network administrators.

For complete information on Avi Vantage architecture, please refer to Avi Vantage Architectural Overview.

This document discusses options for deploying Cisco ACI integration with Avi Vantage in several host infrastructures such as, VMware, Cisco CSP, etc., along with the deployment best practices. This document does not discuss the steps for deployment. For complete deployment information, refer to the Cisco ACI with Avi Vantage Deployment Guide.

Intended Audience

This document is intended for virtualization and network architects seeking to deploy Cisco ACI along with Avi Vantage solution.

Note: A solid understanding and hands-on experience with Cisco ACI and Avi Vantage are the prerequisites to understand this design guide.

Avi Vantage with ACI Integration Modes

Avi Vantage can be integrated with Cisco ACI in the following two modes:

Service manager Mode via REST API

The service manager mode with REST API provides complete automation and flexibility to insert Layer 4-Layer 7 (L4- L7) services with ease in ACI fabric. The primary advantage is the end-to-end automation offered by Cisco ACI and Avi Vantage. ACI handles the network automation, where as Avi Vantage handles the L4-L7 configuration.

Network Policy Mode

This is the traditional mode which involves no integration. Cisco ACI provides the network connectivity and contracts required by Avi Service engines to allow the traffic through.

Hosting Infrastructure

Avi Vantage can be hosted on VMware, Cisco CSP 2100 , baremetal , public clouds, and several such platforms. The following are a few host infrastructures that support ACI fabric:

Avi Vantage on VMware with Write Access

VMware deployments where Avi Controller is configured with vCenter cloud connector. The Avi Controller has write access permissions to vCenter and handles the complete automation involved in creating Service Engines and placing them in the right network. The Controller also scales the Service Engines based on the configured threshold. Refer to VMware write Access for more details.

Avi Vantage on VMware with Read/No Access

VMware deployments where Avi Controller has only read access or no access permission to the vCenter. In such deployments, Service Engines are manually deployed and the Avi Controller does not provide much automation. Refer to VMware Read/No Access for more details.

Avi Vantage on Cisco CSP 2100

Cisco CSP 2100 is a NFV platform based on Intel x86 and the KVM hypervisor. Both the Avi Controller and Avi Service Engines can be deployed on Cisco CSP 2100. Refer to Avi on Cisco CSP 2100 for more details.

Avi Vantage on Openstack

Avi Vantage integrates with OpenStack infrastructure components to provide centralized automation, monitoring, and management of application discovery and delivery. Refer to Avi on OpenStack for more details.

Integration Design Options

img1

For Avi Vantage integration with Cisco ACI, the design options are segregated among different hosting infrastructures. The following section summarizes the available hosting infrastructure and the associated integration options.

ACI Integration Modes along with Hosting Infrastructure

Hosting Infrastructure Integration Options Brief Summary
VMware write access Service manager mode via REST API
  • Workload on VMware vCenter connected as ACI VMM Domain
  • ACI provides network automation for L4-L7 services using service graphs
  • Avi Vantage provides automated configuration and provisioning for its L4-L7 services
Network policy mode
  • ACI provides network connectivity and contracts for access control
  • Avi Vantage provides automated configuration and provisioning for its L4-L7 services
VMware read access and VMware no access Network policy mode
  • Workload on any hypervisor(s), bare-metal server(s), and ACI manages reachability
  • Avi Vantage deployed in no-orchestrator mode (without automated VMware provisioning)
  • Avi Service Engines peer with ACI fabric as a BGP Layer 3-out
Cisco CSP 2100 and bare-metal servers Network policy mode
  • Avi Vantage deployed on CSP 2100 or bare-metal server(s)
  • Avi Service Engines peer with ACI fabric as a BGP Layer 3-out

Refer to the following documentation links for a detailed description on ACI integration options for each hosting infrastructure:

VMware Write Access

VMware Read and No access

Design Considerations and Limitations

Below are a few basic recommendations and best practices applicable for all design options.

Each design option also has specific recommendations mentioned under respective links. Please refer to the links in the section above for more details.

Avi Controller Considerations

The Avi Controller is a single point of management and control for the Avi Vantage system, and is typically deployed as a redundant three-node cluster.

To allow control plane communication between the Avi Controller cluster and Service Engines, open the firewall ports mentioned in the table below.

Traffic source Traffic destination Ports to allow
Avi Controller Avi Controller TCP 22 (SSH)
TCP 443 (HTTPS)
TCP 8443 (HTTPS)
TCP 5098 (SSH) (if the Controller is a docker container, SSH is on port 5098)
Avi Service Engine Avi Controller TCP 22 (SSH)
TCP 8443 (HTTPS)
UDP 123 (NTP)
TCP 5098 (SSH) (if the Controller is a docker container, SSH is on port 5098)

Note: For VMware vCenter Controller-to-ESXi hosts allow port 443.

CPU and Memory Allocations

Avi Vantage uses CPU and memory allocations as tabulated below. The sizing recommendations are based on Service Engine and virtual service scale.

CPU / Memory allocation 8 CPUs / 24 GB 16 CPUs / 32 GB 24 CPUs / 48 GB
Base processes 15 GB 20 GB 24 GB
Log analytics 9 GB 13 GB 24 GB
Virtual service scale 0-200 200-1000 1000-5000
Avi Service Engine scale 0-100 100-200 200-250

Avi Service Engine Considerations

Avi Service Engines handle all data plane operations within Avi Vantage by receiving and executing instructions from the Controller. The SEs perform load balancing and all client and server-facing network interactions.

The Cisco ACI service manager mode via REST API integration is supported only for Avi Vantage integrated with vCenter in write access mode. Refer to Installing Avi Vantage for VMware vCenter for more information on vCenter write access mode.

For network policy mode, Service Engines can be hosted on the same infrastructure as that of the Controller, or a different infrastructure. The only requirement is to ensure connectivity between the Controller and Service Engines.

Avi Controller Multi-Tenancy Considerations

Avi Vantage supports multi-tenancy with Cisco ACI integration.

For Avi Vantage to be used in APIC multiple tenants, the Avi Controller administrator should integrate the Controller with APIC in tenant common. After the integration, the Controller will create the L4-L7 logical device in tenant common. To have Avi Service Engines in multiple tenants, the APIC administrator should then export the L4-L7 logical device under tenant common to other tenants of interest.

By default, Avi Vantage will sync the APIC tenants with bridge domains in which it has been integrated. It also syncs the tenants which have the L4-L7 Avi Vantage device exported. This way, you need not configure the APIC tenants again in the Controller.

For more information on Avi Vantage tenants and on how Service Engine groups are isolated or shared across the tenants, refer to Tenants Versus SE Group Isolation.

Avi Vantage Scale Out Considerations

The Avi Vantage scale out option is used to scale out the virtual services to multiple Service Engines or migrate to new Service Engines for better resource utilization. This scale out can be triggered either automatically based on different parameters like CPU, PPS or manually by using the scale out option under virtual service.

With ACI, the bridge domain with virtual service network will have endpoint learning enabled by default, by which the ACI will map the IP address to the MAC address. So, if multiple Service Engines host the same virtual service, then the ACI fabric will see the same virtual service IP address with multiple MAC addresses on different leaf ports, leading to auto flapping in the specific network.

img2

Starting with Avi Vantage release 17.2.10, there is a workaround available for this. The Service Engines can function with SE tunnel mode enabled, where the return traffic from the secondary SE will be sent to the primary SE. This avoids all IP address to MAC address conflicts, as the traffic flow will be from a single Service Engine.

The SE Tunnel Mode option can be disabled while configuring the cloud as shown in the screenshot below.

img3

Enabling SE Tunnel Mode is recommended only for proof of concepts in the network, where Avi Vantage is been tested in an existing ACI fabric network which has a shared bridge domain for virtual service networks. We recommend disabling this setting for production deployments and follow the section below to disable endpoint leaning on virtual service bridge domain.

img4

Configure the virtual service network bridge domain with the settings below, as it will allow the scaled out traffic for virtual service on multiple Service Engines to transit through the ACI fabric seamlessly.

img5

Note: Configure this setting while creating the virtual service network bridge domain. Modifying the settings for an existing bridge domain will retain some stale entries, leading to unwanted packet drops. For more information on Avi scale out, refer to Virtual Service Scaling.

Source NAT Considerations

By default, Avi Service Engines perform source NAT. You can disable this option on the Avi Controller, if needed. We recommend using source NAT if it is not required to preserve the client IP address. Disabling source NAT will disable connection multiplexing. Refer to Connection Multiplexing for more information on connection multiplexing and its impact on other features.

Disabling source NAT on Avi Service Engines would have an impact on ACI as well, as the source IPs seen by the client network will be the same as those seen by Avi Service Engines. Enable Limit IP Learning to subnet option under the virtual service network bridge domain.

Also for deployments where source NAT is disabled, ensure that the default gateway on servers is pointed to Avi Service engines, so that the return traffic takes the path of the source traffic.

EPG Workload Considerations

For load balancing, it is recommended that the workload for each application be in its respective EPG. For instance, three tier applications with web, app, and database servers are recommended to have specific application servers in specific EPGs like web servers, app servers, etc. This ensures high level security with application communication and external routed clients-to-application communication.

Limitations

The following are a few limitations applicable to all design options:

  • Direct server return is not supported. Use NAT, if the clients and servers belong to the same network or same EPG.
  • Only one ACI fabric service manager via REST API mode integration per Avi Controller is supported .
  • Only GoTo deployment mode is supported. GoThrough, bridge, or transparent mode is not supported.