OpenShift Routes Virtual Service Configuration

In an OpenShift cloud, OpenShift route creation triggers creation of north-south or ingress Avi Vantage VirtualService and Pool objects. OpenShift/Kubernetes Service Configuration in Avi Vantage explains how OpenShift/Kubernetes services map to Avi Vantage’s VirtualService and Pool objects. This article explains how OpenShift routes trigger VirtualService and Pool object creation in Avi Vantage.

OpenShift Routes to Avi Vantage Object Mapping

OpenShift routes can be configured in one of two ways as far as virtual IPs are concerned.

  • Dedicated virtual IP per route: Each OpenShift route is allocated its own VirtualService/Pool objects and a virtual IP (VIP). This provides the best performance, availability and isolation, but consumes an IP Address per route.
  • Shared virtual service across multiple routes: Each OpenShift route is tied to a pre-created parent shared virtual service and will use the shared virtual service’s virtual IP. This conserves IP addresses, but provides variable performance, no isolation, and shares fate with all sibling routes. A parent shared virtual services is pre-created with HTTP/HTTPS application profiles with listeners on ports 80 and 443 for every Service Engine group configured in the cloud. The parent virtual service’s name is of the form parent-vs-SEGroupname-Cloudname.

NB: Support for shared virtual services is available beginning 17.1.9 in the 17.1 release series and 17.2.3 in the 17.2 release series.

The following table maps each type of OpenShift route to Avi Vantage objects.

OpenShift route Avi Vantage Object Comment
Dedicated virtual service per route VirtualService, Pool Every route creates a VirtualService, Pool object. Route annotation avi_proxy: '{"dedicated_route": true}' creates a dedicated virtual service (that uses a VIP) per route in shared VirtualService Projects/Namespaces. Layer4 (TCP, UDP) routes require a dedicated VIP per route
Shared virtual service: Unsecured host/path-based routes Pool HTTP policy is used in parent virtual service to switch host/path to the appropriate pool.
Shared virtual service: Secure routes Child virtual hosted/SNI VirtualService, Pool SNI is used to route traffic to the appropriate child VirtualService/Pool.

Unsupported Features

The following features are unsupported.

  1. Passthrough termination with shared VIPs.
    1. Solution: Use dedicated VIPs instead.
  2. RoutePort in RouteSpec
    1. Solution: Expose a single port in deployment config.

Shared Virtual Service Mode Configuration

shared namespace configuration

  • Field shared_virtualservice_namespace configures the cloud to operate in either dedicated or shared VS mode. Check or uncheck Use Shared Virtual Service in the UI above during cloud configuration. Enabling shared_virtualservice_namespace uses the shared parent virtual service for all routes in all namespaces by default. Disabling shared_virtualservice_namespace creates a dedicated virtual service per route in all namespaces by default.
  • Changing this mode is disruptive and can be time consuming with a large number of routes; hence, it is recommended that this is configured once at initial cloud configuration time.
  • Individual namespaces can be overriden to operate in either shared or dedicated modes using the avi_virtualservice: shared|dedicated namespace annotation.
    • If shared_virtualservice_namespace is enabled for the cloud, namespace annotation avi_virtualservice: dedicated creates dedicated virtual services for all routes in the namespace.
    • On the contrary, if shared_virtualservice_namespace is disabled for the cloud, namespace annotation avi_virtualservice: shared uses the shared virtual service for all routes in the namespace.
  • Every route can be overriden to create a dedicated virtual service for itself, even if the cloud or the namespace is operating in the shared mode. Route annotation avi_proxy: '{"dedicated_route": true}' creates a dedicated virtual service for that route. The most common use case for dedicated virtual service is for non-HTTP(S) layer 4 (TCP, UDP) routes.

Route Configuration Examples

NB: The following examples assume the following cloud configuration.

  • http_container_ports is configured with 80, 8080.
  • use_service_cluster_ip_as_ew_vip is enabled (kube-proxy is disabled cluster wide).
  • default_service_as_east_west_service is enabled (default: on).
  • A Network object is created with an IP address pool for north-south VIPs; a north-south IPAM profile is created and linked to the Network object and the cloud is linked to the north-south IPAM profile.
  • A Network object is created with a IP address pool for east-west VIPs; an east-west IPAM profile is created and linked to the Network object and the cloud is linked to the east-west IPAM profile.

Sample Deployment and Service

The following deployment and service are examples for deploying HTTP pods.

Sample deployment configuration in JSON format.

{
  "kind": "DeploymentConfig",
  "apiVersion": "v1",
  "metadata": {
    "name": "avitest"
  },
  "spec": {
    "template": {
      "metadata": {
        "labels": {
          "name": "avitest"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "avitest",
            "image": "avinetworks/server-os",
            "ports": [
              {
                "name": "http",
                "containerPort": 8080,
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    },
    "replicas": 2,
    "selector": {
      "name": "avitest"
    }
  }
}

Sample service file to create an east-west service in JSON format.

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "avisvc",
    "labels": {
        "svc": "avisvc"
    }
  },
  "spec": {
    "ports": [
    {
      “name”: “http”,
      "port": 80,
      "targetPort": “http”
    }
    ],
    "selector": {
      "name": "avitest"
    }
  }
}

Since default_service_as_east_west_service is enabled in the cloud configuration, this service creates an east-west virtual service.

If HTTPS pods are deployed, containerPort is usually 443 or 8443 and service port is 443 or 8443.

Sample Routes

Dedicated VIP for Unsecured Route

In this case shared_virtualservice_namespace is disabled in the cloud and namespace does not have the avi_virtualservice: shared annotation.

The following sample route file creates a unsecured route with a dedicated VIP for service avisvc.

apiVersion: v1
kind: Route
spec:
  host: aviroute.local
  to:
    kind: Service
    name: avisvc
metadata:
  name: aviroute

This dedicated virtual service route will create its own virtual service and pool with its dedicated VIP.

Shared VIP for Unsecured Route

In this case either shared_virtualservice_namespace is enabled in the cloud or namespace has the avi_virtualservice: shared annotation.

The following sample route file creates a unsecured route associated with parent virtual service parent-vs-Default-Group-Default-Cloud for service avisvc.

apiVersion: v1
kind: Route
spec:
  host: aviroute.local
  to:
    kind: Service
    name: avisvc
metadata:
  name: aviroute

The unsecured application will be created as a pool for the parent virtual service. Pool name will be of the form namespace-routename-aviroute-pool-port-protocol. For example, route name aviroute1 in namespace default will have a pool name default-aviroute1-aviroute-pool-http-tcp. An HTTP policy will provide host/path switching for the pool.

Dedicated VIP for Secured Route with Edge Termination With Certificate Data

In this case shared_virtualservice_namespace is enabled in the cloud but namespace has the avi_virtualservice: dedicated annotation.

The avi_virtualservice: dedicated namespace annotation overrides the shared_virtualservice_namespace cloud configuration and creates dedicated virtual services for all routes in the namespace.

Namespace looks as follows.

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    avi_virtualservice: dedicated
    openshift.io/description: ""
    openshift.io/display-name: ""
    openshift.io/node-selector: region=infra
    openshift.io/requester: system:admin
    openshift.io/sa.scc.mcs: s0:c8,c7
    openshift.io/sa.scc.supplemental-groups: 1000070000/10000
    openshift.io/sa.scc.uid-range: 1000070000/10000
  creationTimestamp: 2017-08-24T23:25:13Z
  labels:
    project1: foo
  name: project1
  resourceVersion: "3845586"
  selfLink: /api/v1/namespacesproject1
  uid: 7b02ef57-8923-11e7-8756-005056b03f90
spec:
  finalizers:
  - openshift.io/origin
  - kubernetes
status:
  phase: Active

The following sample route file creates a secured route with edge termination with a dedicated VIP for service avisvc.

apiVersion: v1
kind: Route
spec:
  host: aviroute.local
  to:
    kind: Service
    name: avisvc
  tls:
    termination: edge
    certificate: |-
      -----BEGIN CERTIFICATE-----
      MIIDsTCCApmgAwIBAgIJANX0VbJbXVUSMA0GCSqGSIb3DQEBCwUAMG8xCzAJBgNV
     AYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRQwEgYDVQQHDAtTYW50
...
      T5vbDSdlZHM1m+xOjk3jz7nQjORxtsX4qsM0He5fNu1j/76dR4qmjAgx2qPWOGfS
      1I+9RjkgA7bOKIloz28GmCAOJDHdv2ecd0UbFsXu26s0WZFE4g==
      -----END CERTIFICATE-----
    key: |-
      -----BEGIN PRIVATE KEY-----
      MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDb6YAxPwee15eI
      KwUauaQcx23yBk1wV0DhN280WHQ86OIwyPmX7MoVicOwzq0fEzw/Zu+1YfV75+GE
...
      1katRly4cgYeryEX+enH4+N0GrZepL4fHTorflL0nuZMG9HA1MLwYCK3UADqLDJ6
      45f3rxRceTmBoh+nQI5QIWk=
      -----END PRIVATE KEY-----
metadata:
  name: aviroute

Dedicated VIP for Unsecured Route

In this case shared_virtualservice_namespace is enabled in the cloud and namespace does not have the <avi_virtualservice: dedicated annotation.

All routes in this namespace create shared virtual services, but the following sample route file creates a unsecured route with a dedicated VIP for service avisvc, because the avi_proxy: '{"dedicated_route": true}' annotation requests a dedicated virtual service.

apiVersion: v1
kind: Route
spec:
  host: aviroute.local
  to:
    kind: Service
    name: avisvc
metadata:
  name: aviroute
  annotations:
    avi_proxy: '{"dedicated_route": true}'

This dedicated virtual service route will create its own virtual service and pool with its dedicated VIP.

Shared VIP for Route With Edge Termination With Certificate Data

In this case shared_virtualservice_namespace is enabled in the cloud and namespace does not have the avi_virtualservice: dedicated annotation.

The following sample route file creates a secured route with edge termination with VIP assigned for parent virtual service parent-vs-Default-Group-Default-Cloud for service avisvc.

apiVersion: v1
kind: Route
spec:
  host: aviroute.local
  to:
    kind: Service
    name: avisvc
  tls:
    termination: edge
    certificate: |-
      -----BEGIN CERTIFICATE-----
      MIIDsTCCApmgAwIBAgIJANX0VbJbXVUSMA0GCSqGSIb3DQEBCwUAMG8xCzAJBgNV
     AYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRQwEgYDVQQHDAtTYW50
...
      T5vbDSdlZHM1m+xOjk3jz7nQjORxtsX4qsM0He5fNu1j/76dR4qmjAgx2qPWOGfS
      1I+9RjkgA7bOKIloz28GmCAOJDHdv2ecd0UbFsXu26s0WZFE4g==
      -----END CERTIFICATE-----
    key: |-
      -----BEGIN PRIVATE KEY-----
      MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDb6YAxPwee15eI
      KwUauaQcx23yBk1wV0DhN280WHQ86OIwyPmX7MoVicOwzq0fEzw/Zu+1YfV75+GE
...
      1katRly4cgYeryEX+enH4+N0GrZepL4fHTorflL0nuZMG9HA1MLwYCK3UADqLDJ6
      45f3rxRceTmBoh+nQI5QIWk=
      -----END PRIVATE KEY-----
metadata:
  name: aviroute

Secure route with a shared VIP will create a SNI-based child VirtualService/Pool for parent virtual service parent-vs-Default-Group-Default-Cloud for service avisvc.

Dedicated VIP Secured Route With Edge Termination, insecureEdgeTerminationPolicy Allowing HTTP Traffic With a Certificate Name

The following sample route file creates a secured route with edge termination with a reference to a certificate name instead of providing certificate data in the route file.

apiVersion: v1
kind: Route
spec:
  host: aviroute.local
  to:
    kind: Service
    name: avisvc
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow
metadata:
  name: aviroute
  annotations:
    avi_proxy: '{"virtualservice": {"ssl_key_and_certificate_refs": ["cert1"]}}'

Path-based Routes (dedicated mode)

In dedicated mode, a single virtual service is created for every host name. If there are multiple routes using the same host name, each route with the same host name corresponds to a pool group(s) and pool(s) for the virtual service. HTTP request policies are automatically configured to match and perform layer 7 URL switching to the appropriate pool group.

Suppose there are 2 routes — aviroute1 and aviroute2 — with host name app.os.acme.com and paths /foo and /bar, respectively. This will result in the following configuration in Avi Vantage.

VirtualService (app.os.acme.com)
  PoolGroup aviroute1-poolgroup-8080-tcp
    Pool aviroute1-pool-8080-tcp
  PoolGroup aviroute2-poolgroup-8080-tcp
    Pool aviroute2-pool-8080-tcp

The following sample route file creates an unsecured route for host app.os.acme.com and path /foo. This maps to a virtual service called app.os.acme.com and its pool group aviroute1-poolgroup-8080-tcp.

apiVersion: v1
kind: Route
spec:
  host: app.os.acme.com
  path: "/foo"
  to:
    kind: Service
    name: avisvc1
metadata:
  name: aviroute1

The following sample route file creates an unsecured route for host app.os.acme.com and path /bar. This maps to a virtual service called app.os.acme.com and its pool group aviroute2-poolgroup-8080-tcp.

apiVersion: v1
kind: Route
spec:
  host: app.os.acme.com
  path: "/bar"
  to:
    kind: Service
    name: avisvc2
metadata:
  name: aviroute2

Automatic Configuration

The following fields in the VirtualServices and Pools objects are automatically derived from OpenShift routes by the cloud connector.


VirtualService Field OpenShift/Kubernetes Field Comments
tenant Project/namespace
name Host name
fqdn Host
ip_address Dedicated VIP: Can be explicitly specified via annotations or auto-allocated from the north-south IPAM object

Shared VIP: If parent virtual service is specified, ip_address isn’t required.

services Port field in the corresponding service
application_profile Layer4 if overridden via annotations or container ports don't match http_container_ports in Cloud Configuration, else Layer7
network_profile Always System-TCP-Proxy
pool_group One for every route with the same host name
pool One for every route with the same host name
network_security_policy A default network security policy is created for every new or child SNI virtual service
http_policies Layer7 policy for URL (host, path, port) switching amongst routes sharing the virtual service
microservice A microservice is automatically created for every new or child SNI virtual service
east_west_placement Always north-south