OpenShift/Kubernetes Service Configuration in Avi Vantage

Introduction

In an OpenShift or Kubernetes cloud, service creation triggers the corresponding creation of Avi Vantage VirtualService and Pool objects. Parameters governing most of the behavior of the Avi Vantage virtual service and its corresponding Avi Vantage pool members are automatically filled in from OpenShift/Kubernetes by the cloud connector. Others can be configured via annotations in the OpenShift/Kubernetes service object. The following article explains what fields are automatically derived by the cloud connector and how to stipulate additional configuration for Avi virtual services and pools.

A Note on Tenancy

OpenShift projects/Kubernetes namespaces map directly to Avi tenants. Avi tenants are automatically created and deleted following OpenShift projects/Kubernetes namespaces. Virtual services are created in the appropriate tenants. For example, services in project ‘default’ result in virtual service creation in Avi tenant ‘default.’

Automatic Configuration

The following fields in VirtualServices and Pools are automatically derived from OpenShift/Kubernetes services by the cloud connector.

Avi Vantage VirtualService Fields

Field OpenShift/Kubernetes Field Comments
tenant project/namespace
name Service name
fqdn name.subdomain ‘subdomain’ is the configured subdomain in the IPAM/DNS profile. North-south virtual services use the north-south DNS profile and east-west virtual services use the east-west DNS profile.
ip_address East-west virtual service: service cluster IP if use_service_cluster_ip_as_ew_vip is enabled in the cloud; else, auto-allocated from the east-west IPAM object

North-south virtual service: auto-allocated from the north-south IPAM object

NB: use_service_cluster_ip_as_ew_vip can be enabled if kube-proxy is disabled cluster wide.
services port field in the service
application_profile http_container_ports is a target list of HTTP/HTTPS ports in the cloud object. If all endpoints’ ports match any port in the http_container_ports list and the service protocol is ‘tcp’, the virtual service is a layer7 (HTTP/HTTPS) service. If the service protocol is ‘tcp’, but endpoints’ ports don’t match with any port in the http_container_ports list, a layer4 TCP virtual service is created. If the service protocol is ‘udp’, a layer4 UDP virtual service is created. E.g., if a deployment containers' containerPort is 8080, and some deployment containers' containerPort is 9080, and both services should be treated as HTTP/HTTPS, add 8080 and 9080 to the http_container_ports list in the cloud.
network_profile Use System-TCP-Proxy network profile for layer7 and layer4 TCP virtual services; use System-UDP-Fast-Path for layer4 UDP virtual services
pool Create a default pool corresponding to a endpoint port; if there are multiple endpoint ports, one of them will be the default pool; other ports will create additional pools.
network_security_policy A default network security policy is created for every virtual service.
http_policies An HTTP policy set is automatically created with switching rules when there are multiple endpoint ports/pools for a layer7 virtual service. For example, suppose the service object has 2 port mappings, say, (port: 80, targetPort: 8080) and (port: 443, targetPort: 8443). Two pools will be created corresponding to 8080 (pool-8080) and 8443 (pool-8443). An HTTP policy will be created with pool switching rules for port 80 to the pool-8080 and port 443 to pool-8443. For layer4 virtual services, a service_pool_select is used. See below.
microservice A microservice is automatically created for every virtual service.
service_pool_select A service_pool_select is automatically created with switching rules when there are multiple endpoint ports/pools for a layer4 virtualservice. For example, suppose the service object has 2 port mappings, say, (port: 53, targetPort: 8053, protocol: tcp) and (port: 53, targetPort: 8053, protocol: udp). Two pools will be created corresponding to 8053,tcp (pool-8053-tcp) and 8053,udp (pool-8053-udp). A service_pool_select policy will be created with pool switching rules for port 53,tcp to the pool-8053-tcp and port 53,udp to pool-8053-udp.
east_west_placement If default_service_as_east_west_service is enabled in the cloud, services are created by default as east-west virtual services. Else, the east_west_placement flag specified via annotations determines if a service is east-west or north/south. "annotations": {

"avi_proxy": "{\"virtualservice\": {\"east_west_placement\": false}}"

} creates a north-south service

Additional Virtual Service Configuration Examples

Additional configuration can be provided using annotations in the service configuration. Examples of such configuration include SSL certs, profiles, policies, health monitors, etc.

NB: The following examples assume the following cloud configuration.

  • http_container_ports is configured with 80, 8080
  • use_service_cluster_ip_as_ew_vip is enabled (kube-proxy is disabled cluster wide)
  • default_service_as_east_west_serviceis enabled (default: on)
  • A Network object is created with an IP address pool for north-south VIPs; a north-south IPAM profile is created and linked to the Network object and the cloud is linked to the north-south IPAM profile
  • A Network object is created with an IP address pool for east-west VIPs; an east-west IPAM profile is created and linked to the Network object and the cloud is linked to the east-west IPAM profile

A Layer 7 East-West HTTP Virtual Service

Sample deployment configuration in JSON format.

{
 "kind": "DeploymentConfig",
 "apiVersion": "v1",
 "metadata": {
  "name": "avitest"
 },
 "spec": {
  "template": {
   "metadata": {
    "labels": {
     "name": "avitest"
    }
   },
   "spec": {
    "containers": [
     {
      "name": "avitest",
      "image": "avinetworks/server-os",
      "ports": [
       {
        "name": "http",
        "containerPort": 8080,
        "protocol": "TCP"
       }
      ]
     }
    ]
   }
  },
  "replicas": 2,
  "selector": {
   "name": "avitest"
  }
 }
}

Sample service file to create an east-west service in JSON format.

{
 "kind": "Service",
 "apiVersion": "v1",
 "metadata": {
   "name": "avisvc",
   "labels": {
      "svc": "avisvc"
   }
  },
  "spec": {
   "ports": [{
    "name": "http",
    "port": 80,
    "targetPort": "http"
  }],
  "selector": {
   "name": "avitest"
  }
 }
}

Since default_service_as_east_west_serviceis enabled in the cloud configuration, this service creates an east-west virtual service. Since use_service_cluster_ip_as_ew_vipis enabled, this virtual service uses the same virtual IP as the cluster IP for this service.

A Layer 7 North-South HTTPS Virtual Service

Sample deployment configuration in JSON format.

{
 "kind": "DeploymentConfig",
 "apiVersion": "v1",
 "metadata": {
  "name": "avitest"
 },
 "spec": {
  "template": {
   "metadata": {
    "labels": {
     "name": "avitest"
     }
    },
    "spec": {
     "containers": [
      {
       "name": "avitest",
       "image": "avinetworks/server-os",
       "ports": [
       {
        "name": "http",
        "containerPort": 8080,
        "protocol": "TCP"
       }
      ]
     }
    ]
   }
  },
  "replicas": 2,
  "selector": {
   "name": "avitest"
  }
 }
}

Sample service file to create a north-south service in YAML format

{
 apiVersion: v1
 kind: Route
 spec:
   to:
     kind: Service
     
     name: avisvc
 metadata:
   name: avisvc
   label: avisvc
   annotations:
     avi_proxy: "{\"virtualservice\": {\"services\": [{\"port\": 443, \"enable_ssl\": true}], \"ssl_key_and_certificate_refs\": [\"app_cert\"], \"ssl_profile_ref\": \"/api/sslprofile/?name=System-Standard\"}}"
 selector:
   name: avitest
}

Annotations:

  1. services overrides the default service in the spec and configures SSL/TLS offload on the VirtualService. If SSL/TLS is configured, the default cert will be used if no cert is provided
  2. auto_allocate_ip set to true asks that a Virtual IP be allocated for this service from the north-south IPAM profile IP address pool
  3. ssl_profile_ref uses the System-Standard SSL profile for illustration purposes. If unspecified, it defaults to the System-Standard SSL profile.
  4. ssl_key_and_certificate_refs asks that a certificate called avisvccert be used for the service. NB: The certificate avisvccert should have been pre-created, else VirtualService creation will fail

A Layer 4 East-West HTTP Virtual Service

Sample deployment configuration in JSON format.

{
  "kind": "DeploymentConfig",
  "apiVersion": "v1",
  "metadata": {
    "name": "avitest"
  },
  "spec": {
    "template": {
      "metadata": {
        "labels": {
          "name": "avitest"
       }
      },
      "spec": {
        "containers": [
          {
            "name": "avitest",
            "image": "avinetworks/server-os",
            "ports": [
              {
                "name": "foo",
                "containerPort": 8080,
                "protocol": "TCP"
              },
              {
                "name": "bar",
                "containerPort": 8443,
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    },
    "replicas": 2,
    "selector": {
      "name": "avitest"
    }
  }
}

Sample service file to create an east-west service in JSON format.

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "avisvc",
    "labels": {
        "svc": "avisvc"
    }
  },
  "spec": {
    "ports": [
    {
      "name": "foo",
      "port": 80,
      "targetPort": 8080
    },
    {
      "name": "bar",
      "port": 443,
      "targetPort": "bar"
    }
    ],
    "selector": {
      "name": "avitest"
    }
  }
}

 

NB: Container port 8443 doesn’t match with any port (80, 8080) in http_container_ports. So, a layer4 TCP virtual service will be created in this case.

Exposing Just a Single L4 Port While Service Has Multiple Ports

The service has 2 ports, 443 and 80, mapped to targetPorts foo and bar. Avi annotations creates a north-south service with an auto-allocated VIP, a Layer4 (TCP) application profile and specifies just a single port. The single port is switched to pool avisvc-pool-bar-tcp. Pool names are of the form servicename-pool-targetPort-protocol.

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "avisvc",
    "labels": {
        "svc": "avisvc"
    },
    "annotations": {
        "avi_proxy": "{\"virtualservice\": {\"auto_allocate_ip\": true, \"east_west_placement\": false, \"application_profile_ref\": \"/api/applicationprofile/?name=System-L4-Application\", \"services\": [{\"port\"
: 443}], \"service_pool_select\": [{\"service_port\": 443, \"service_pool_ref\": \"/api/pool/?name=avisvc-pool-bar-tcp\"}]}}"
    }
  },
  "spec": {
    "ports": [
    {
      "name": "foo",
      "port": 443,
      "targetPort": 8080
    },
    {
      "name": "bar",
      "port": 80,
      "targetPort": "bar"
    }
    ],
    "selector": {
      "name": "avitest"
    }
  }
}

Analytics Policy to Enable Real-time Metrics and Full Client Logs

The following service file creates an east-west service with real-time metrics and full client logs enabled.

{
 "kind": "Service",
 "apiVersion": "v1",
 "metadata": {
  "name": "avisvc",
  "labels": {
     "svc": "avisvc"
  },
  "annotations": {
    "avi_proxy": "{\"virtualservice\": {\"analytics_policy\": {\"metrics_realtime_update\": {\"duration\": 0, \"enabled\":true}, \"client_insights\": \"NO_INSIGHTS\", \"full_client_logs\": {\"enabled\":true}}}}"
  }
 },
 "spec": {
  "ports": [
  {
   "name": "http",
   "port": 80,
   "targetPort": "http"
  }
  ],
  "selector": {
   "name": "avitest"
  }
 }
}

Changing the Default Load-balancing Algorithm for the Default Pool

The following annotation sets the load-balancing algorithm to round robin from the default (least connections) for the default pool of a virtual service.

"annotations": {
  "avi_proxy": "{\"pool\": {\"lb_algorithm\": \"LB_ALGORITHM_ROUND_ROBIN\"}}"
  }

The following annotation sets the load-balancing algorithm to “consistent hash” with a hash based on client source IP address.

"annotations": {
  "avi_proxy": "{\"pool\": {\"lb_algorithm\": \"LB_ALGORITHM_CONSISTENT_HASH\", \"lb_algorithm_hash\": \"LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS\"}}"
  }

Best Practices for Health Monitoring

Kubernetes provides liveness probes for Pod health-checking. See also The Lifecycle of a Pod.

  1. For health checks that Kubernetes can perform, our recommendation is to configure those via Kubernetes; unhealthy Pods will be removed from Kubernetes endpoints and subsequently from Avi Vantage's pool.
  2. In addition, an Avi layer 4 (TCP/UDP) health monitor can detect network issues, so we recommend using one. Avi uses HTTP/HTTPS health monitors for HTTP/HTTPS applications by default. Some applications may not respond or respond incorrectly to the default health monitor probe at "GET /". To avoid such false failures, enable the "Always Use Layer4 Health Monitoring" setting in the Virtual Service settings for OpenShift/Kubernetes cloud. This will use TCP Health Monitors even for HTTP/HTTPS applications, thus avoiding false positives.
  3. Some applications require custom health monitoring that Kubernetes liveness probes don't cover. A custom response or response code may have to be checked. For such cases, create Avi custom health monitors and use them via annotations as specified below.

Adding Custom Health Monitors for a Pool

The following annotation adds multiple health monitors to the default pool.

"annotations": {
  "avi_proxy": "{\"pool\": {\"health_monitor_refs\": [\"/api/healthmonitor?name=User-HM\", \"/api/healthmonitor?name=System-TCP\"]}}"
  }

Adding an Application Persistence Profile to a Pool

The following annotation adds a System-Persistence-Http-Cookie application persistence profile to the pool.

"annotations": {
  "avi_proxy": "{\"pool\": {\"application_persistence_profile_ref\": \"/api/applicationpersistenceprofile?name=System-Persistence-Http-Cookie\"}}"
  }

Session Affinity/Client Persistence Guidelines

Use the following guidelines when session affinity/client persistence is needed for applications.

 

Type of Service HTTP/HTTPS application TCP/UDP application
OpenShift Routes or North-South Services persistence profile System-Persistence-Http-Cookie or

Load balancing algorithm LB_ALGORITHM_CONSISTENT_HASH with lb_algorithm_hash of LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS, LB_ALGORITHM_CONSISTENT_HASH_URI or LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER

persistence profiles System-Persistence-Client-IP, System-Persistence-Custom-Http-Header, System-Persistence-App-Cookie

or

Load balancing algorithm LB_ALGORITHM_CONSISTENT_HASH with lb_algorithm_hash of LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS

East-West Services persistence profile System-Persistence-Http-Cookie or

Load balancing algorithm LB_ALGORITHM_CONSISTENT_HASH with lb_algorithm_hash of LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS, LB_ALGORITHM_CONSISTENT_HASH_URI or LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER

Load balancing algorithm LB_ALGORITHM_CONSISTENT_HASH with lb_algorithm_hash of LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS