OpenShift Routes Virtual Service Configuration
In an OpenShift cloud, OpenShift route creation triggers creation of north-south or Ingress Avi Vantage virtual service and pool objects. OpenShift/Kubernetes Service Configuration in Avi Vantage explains how OpenShift/Kubernetes services map to Avi Vantage’s virtual service and pool objects. This article explains how OpenShift routes trigger virtual service and pool object creation in Avi Vantage.
OpenShift Routes to Avi Vantage Object Mapping
OpenShift routes can be configured in one of two ways as far as virtual IPs(VIP)are concerned.
- Dedicated virtual IP per route: Each OpenShift route is allocated its own virtual service/pool objects and a virtual IP (VIP). This provides the best performance, availability and isolation, but consumes an IP Address per route.
- Shared virtual service across multiple routes: Each OpenShift route is tied to a pre-created parent shared virtual service and will use the shared virtual service’s virtual IP. This conserves IP addresses, but provides variable performance, no isolation, and shares fate with all sibling routes. A parent shared virtual services is pre-created with HTTP/HTTPS application profiles with listeners on ports 80 and 443 for every Service Engine group configured in the cloud.The parent virtual service’s name is of the form
parent-vs-SEGroupname-Cloudname
.
Starting with release 18.1.3, the cloud configuration override_service_ports
has been introduced. By default, this knob is disabled. When the knob is not set, virtual service for route will use ports from OpenShift service. When the knob is set, virtual services for Route will use well known ports i.e. 80 for HTTP and 443 for HTTPS.
NB: Support for shared virtual services is available beginning 17.1.9 in the 17.1 release series and 17.2.3 in the 17.2 release series.
The following table maps each type of OpenShift route to Avi Vantage objects.
OpenShift route | Avi Vantage Object | Comment |
Dedicated virtual service per route | Virtual service, Pool | Every route creates a virtual service, pool object. Route annotation avi_proxy: '{"dedicated_route": true}' creates a dedicated virtual service (that uses a VIP) per route in shared virtual service Projects/Namespaces. |
Shared virtual service: Unsecured host/path-based routes | Pool | HTTP policy is used in parent virtual service to switch host/path to the appropriate pool. |
Shared virtual service: Secure routes | Child virtual hosted/SNI virtual service, pool | SNI is used to route traffic to the appropriate child virtual service/Pool. |
Unsupported Features
The following features are unsupported.
- Passthrough termination with shared VIPs.
- Solution: Use dedicated VIPs instead.
- RoutePort in RouteSpec
- Solution: Expose a single port in deployment config.
Shared Virtual Service Mode Configuration
- The Field
shared_virtualservice_namespace
configures the cloud to operate in either dedicated or shared VS mode. Enable Use Shared Virtual Service in the UI, if required. Enablingshared_virtualservice_namespace
uses the shared parent virtual service for all routes in all namespaces by default. Disablingshared_virtualservice_namespace
creates a dedicated virtual service per route in all namespaces by default. - Changing this mode is disruptive and can be time consuming with a large number of routes; hence, it is recommended that this is configured once at initial cloud configuration time.
- Individual namespaces can be overridden to operate in either shared or dedicated modes using the
avi_virtualservice: shared|dedicated
namespace annotation.- If
shared_virtualservice_namespace
is enabled for the cloud, namespace annotationavi_virtualservice: dedicated
creates dedicated virtual services for all routes in the namespace. - On the contrary, if
shared_virtualservice_namespace
is disabled for the cloud, namespace annotationavi_virtualservice: shared
uses the shared virtual service for all routes in the namespace.
- If
- Every route can be overridden to create a dedicated virtual service for itself, even if the cloud or the namespace is operating in the shared mode. Route annotation
avi_proxy: '{"dedicated_route": true}'
creates a dedicated virtual service for that route.
Route Configuration Examples
The following examples assume the following cloud configuration.
http_container_ports
is configured with 80, 8080.use_service_cluster_ip_as_ew_vip
is enabled (kube-proxy is disabled cluster wide).default_service_as_east_west_service
is enabled (default: on).- A network object is created with an IP address pool for north-south VIPs; a north-south IPAM profile is created and linked to the network object and the cloud is linked to the north-south IPAM profile.
- A network object is created with a IP address pool for east-west VIPs; an east-west IPAM profile is created and linked to the network object and the cloud is linked to the east-west IPAM profile.
Sample Deployment and Service
The following deployment and service are examples for deploying HTTP pods.
Sample deployment configuration in JSON format.
{
"kind": "DeploymentConfig",
"apiVersion": "v1",
"metadata": {
"name": "avitest"
},
"spec": {
"template": {
"metadata": {
"labels": {
"name": "avitest"
}
},
"spec": {
"containers": [
{
"name": "avitest",
"image": "avinetworks/server-os",
"ports": [
{
"name": "http",
"containerPort": 8080,
"protocol": "TCP"
}
]
}
]
}
},
"replicas": 2,
"selector": {
"name": "avitest"
}
}
}
Sample service file to create an east-west service in JSON format.
{
"kind": "Service",
"apiVersion": "v1",
"metadata": {
"name": "avisvc",
"labels": {
"svc": "avisvc"
}
},
"spec": {
"ports": [
{
"name": "http",
"port": 80,
"targetPort": "http"
}
],
"selector": {
"name": "avitest"
}
}
}
Since default_service_as_east_west_service
is enabled in the cloud configuration, this service creates an east-west virtual service.
If HTTPS pods are deployed, container port is usually 443 or 8443 and service port is 443 or 8443.
Sample Routes
Dedicated VIP for Unsecured Route
In this case shared_virtualservice_namespace
is disabled in the cloud and namespace does not have the avi_virtualservice: shared
annotation.
The following sample route file creates a unsecured route with a dedicated VIP for service avisvc.
apiVersion: v1
kind: Route
spec:
host: aviroute.local
to:
kind: Service
name: avisvc
metadata:
name: aviroute
This dedicated virtual service route will create its own virtual service and pool with its dedicated VIP.
Shared VIP for Unsecured Route
In this case either shared_virtualservice_namespace
is enabled in the cloud or namespace has the avi_virtualservice: shared
annotation.
The following sample route file creates a unsecured route associated with parent virtual service parent-vs-Default-Group-Default-Cloud
for service avisvc
.
apiVersion: v1
kind: Route
spec:
host: aviroute.local
to:
kind: Service
name: avisvc
metadata:
name: aviroute
The unsecured application will be created as a pool for the parent virtual service. Pool name will be of the form namespace-routename-aviroute-pool-port-protocol
. For example, route name aviroute1
in namespace default
will have a pool name default-aviroute1-aviroute-pool-http-tcp
. An HTTP policy will provide host/path switching for the pool.
Dedicated VIP for Secured Route with Edge Termination With Certificate Data
In this case shared_virtualservice_namespace
is enabled in the cloud but namespace has the avi_virtualservice: dedicated
annotation.
The avi_virtualservice: dedicated
namespace annotation overrides the shared_virtualservice_namespace
cloud configuration and creates dedicated virtual services for all routes in the namespace.
Namespace looks as follows.
apiVersion: v1
kind: Namespace
metadata:
annotations:
avi_virtualservice: dedicated
openshift.io/description: ""
openshift.io/display-name: ""
openshift.io/node-selector: region=infra
openshift.io/requester: system:admin
openshift.io/sa.scc.mcs: s0:c8,c7
openshift.io/sa.scc.supplemental-groups: 1000070000/10000
openshift.io/sa.scc.uid-range: 1000070000/10000
creationTimestamp: 2017-08-24T23:25:13Z
labels:
project1: foo
name: project1
resourceVersion: "3845586"
selfLink: /api/v1/namespacesproject1
uid: 7b02ef57-8923-11e7-8756-005056b03f90
spec:
finalizers:
- openshift.io/origin
- kubernetes
status:
phase: Active
The following sample route file creates a secured route with edge termination with a dedicated VIP for service avisvc
.
apiVersion: v1
kind: Route
spec:
host: aviroute.local
to:
kind: Service
name: avisvc
tls:
termination: edge
certificate: |-
-----BEGIN CERTIFICATE-----
MIIDsTCCApmgAwIBAgIJANX0VbJbXVUSMA0GCSqGSIb3DQEBCwUAMG8xCzAJBgNV
AYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRQwEgYDVQQHDAtTYW50
...
T5vbDSdlZHM1m+xOjk3jz7nQjORxtsX4qsM0He5fNu1j/76dR4qmjAgx2qPWOGfS
1I+9RjkgA7bOKIloz28GmCAOJDHdv2ecd0UbFsXu26s0WZFE4g==
-----END CERTIFICATE-----
key: |-
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDb6YAxPwee15eI
KwUauaQcx23yBk1wV0DhN280WHQ86OIwyPmX7MoVicOwzq0fEzw/Zu+1YfV75+GE
...
1katRly4cgYeryEX+enH4+N0GrZepL4fHTorflL0nuZMG9HA1MLwYCK3UADqLDJ6
45f3rxRceTmBoh+nQI5QIWk=
-----END PRIVATE KEY-----
metadata:
name: aviroute
Dedicated VIP for Unsecured Route
In this case shared_virtualservice_namespace
is enabled in the cloud and namespace does not have the <avi_virtualservice: dedicated
annotation.
All routes in this namespace create shared virtual services, but the following sample route file creates a unsecured route with a dedicated VIP for service avisvc
, because the avi_proxy: '{"dedicated_route": true}'
annotation requests a dedicated virtual service.
apiVersion: v1
kind: Route
spec:
host: aviroute.local
to:
kind: Service
name: avisvc
metadata:
name: aviroute
annotations:
avi_proxy: '{"dedicated_route": true}'
This dedicated virtual service route will create its own virtual service and pool with its dedicated VIP.
Shared VIP for Route With Edge Termination With Certificate Data
In this case shared_virtualservice_namespace
is enabled in the cloud and namespace does not have the avi_virtualservice: dedicated
annotation.
The following sample route file creates a secured route with edge termination with VIP assigned for parent virtual service parent-vs-Default-Group-Default-Cloud
for service avisvc
.
apiVersion: v1
kind: Route
spec:
host: aviroute.local
to:
kind: Service
name: avisvc
tls:
termination: edge
certificate: |-
-----BEGIN CERTIFICATE-----
MIIDsTCCApmgAwIBAgIJANX0VbJbXVUSMA0GCSqGSIb3DQEBCwUAMG8xCzAJBgNV
AYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRQwEgYDVQQHDAtTYW50
...
T5vbDSdlZHM1m+xOjk3jz7nQjORxtsX4qsM0He5fNu1j/76dR4qmjAgx2qPWOGfS
1I+9RjkgA7bOKIloz28GmCAOJDHdv2ecd0UbFsXu26s0WZFE4g==
-----END CERTIFICATE-----
key: |-
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDb6YAxPwee15eI
KwUauaQcx23yBk1wV0DhN280WHQ86OIwyPmX7MoVicOwzq0fEzw/Zu+1YfV75+GE
...
1katRly4cgYeryEX+enH4+N0GrZepL4fHTorflL0nuZMG9HA1MLwYCK3UADqLDJ6
45f3rxRceTmBoh+nQI5QIWk=
-----END PRIVATE KEY-----
metadata:
name: aviroute
Secure route with a shared VIP will create a SNI-based child virtual service/pool for parent virtual service parent-vs-Default-Group-Default-Cloud
for service avisvc
.
Dedicated VIP Secured Route With Edge Termination, insecureEdgeTerminationPolicy Allowing HTTP Traffic With a Certificate Name
The following sample route file creates a secured route with edge termination with a reference to a certificate name instead of providing certificate data in the route file.
apiVersion: v1
kind: Route
spec:
host: aviroute.local
to:
kind: Service
name: avisvc
tls:
termination: edge
insecureEdgeTerminationPolicy: Allow
metadata:
name: aviroute
annotations:
avi_proxy: '{"virtualservice": {"ssl_key_and_certificate_refs": ["cert1"]}}'
Path-based Routes (dedicated mode)
In dedicated mode, a single virtual service is created for every host name. If there are multiple routes using the same host name, each route with the same host name corresponds to a pool group(s) and pool(s) for the virtual service. HTTP request policies are automatically configured to match and perform layer 7 URL switching to the appropriate pool group.
Suppose there are 2 routes — aviroute1
and aviroute2
— with host name app.os.acme.com
and paths /foo
and /bar
, respectively. This will result in the following configuration in Avi Vantage.
VirtualService (app.os.acme.com)
PoolGroup aviroute1-poolgroup-8080-tcp
Pool aviroute1-pool-8080-tcp
PoolGroup aviroute2-poolgroup-8080-tcp
Pool aviroute2-pool-8080-tcp
The following sample route file creates an unsecured route for host app.os.acme.com
and path /foo
. This maps to a virtual service called app.os.acme.com
and its pool group aviroute1-poolgroup-8080-tcp
.
apiVersion: v1
kind: Route
spec:
host: app.os.acme.com
path: "/foo"
to:
kind: Service
name: avisvc1
metadata:
name: aviroute1
The following sample route file creates an unsecured route for host app.os.acme.com
and path /bar
. This maps to a virtual service called app.os.acme.com
and its pool group aviroute2-poolgroup-8080-tcp
.
apiVersion: v1
kind: Route
spec:
host: app.os.acme.com
path: "/bar"
to:
kind: Service
name: avisvc2
metadata:
name: aviroute2
Notes:
Starting with release 18.1.3,
- On assigning a virtual service on the Avi Controller, the IP of the assigned virtual service is reported back to OpenShift.
- If route sync fails, the failure status is reported.
Automatic Configuration
The following fields in the virtual services and pools objects are automatically derived from OpenShift routes by the cloud connector.
VirtualService Field | OpenShift/Kubernetes Field | Comments |
tenant | Project/namespace | |
name | Host name | |
fqdn | Host | |
ip_address | Dedicated VIP: Can be explicitly specified via annotations or auto-allocated from the north-south IPAM object Shared VIP: If parent virtual service is specified, ip_address isn’t required. |
|
services | Port field in the corresponding service | |
application_profile | Layer4 if overridden via annotations or container ports don't match http_container_ports in Cloud Configuration, else Layer7 | |
network_profile | Always System-TCP-Proxy | |
pool_group | One for every route with the same host name | |
pool | One for every route with the same host name | |
network_security_policy | A default network security policy is created for every new or child SNI virtual service | |
http_policies | Layer7 policy for URL (host, path, port) switching amongst routes sharing the virtual service | |
microservice | A microservice is automatically created for every new or child SNI virtual service | |
east_west_placement | Always north-south |
Annotations
Additional configuration for Avi virtual services can be provided via annotations.
Automatic WAF policy
Starting release 17.2.4, the following annotation automatically creates and attaches a Web application firewall (WAF) policy to the VirtualService. After initial creation, the WAF policy can be modified independently of the virtual service. Note that the version field is necessary, since a WAF profile was introduced in the virtual service starting this release.
metadata:
annotations:
avi_proxy: '{"waf_policy": true, "version": "17.2.4"}'
If the cloud uses the shared virtual service model and just a few Routes require WAF service, dedicated virtual services can be created just for these Routes using the dedicated_route flag. The annotation would be as follows in this case.
metadata:
annotations:
avi_proxy: '{"waf_policy": true, "dedicated_route": true, "version": "17.2.4"}'