Using Service Engine Groups with OpenShift/Kubernetes

Introduction

Service Engine groups are resource groups for a set of Avi Service Engines; different Service Engine groups can have different sizes e.g. different CPU cores and memory allocations, different HA models (active/active vs N+M HA). Sometimes it may be desirable to associate different types of virtual service with different SE groups, e.g., production vs test virtual services.

How It Works

Every SE in an OpenShift/Kubernetes cloud is created on a OpenShift/Kubernetes node. Nodes can have associated labels, e.g., type=prod. There can be multiple SE groups on each node. Every SE group can be associated with a node label; all SEs created on nodes with that matching label will be automatically associated with that SE group.

Service Engine Group Configuration

Creating and Configuring a Service Engine Group

Create an SE group prod with host_attribute_key=type and host_attribute_value=prod. Every SE on nodes with label type=prod will be automatically associated with the prod SE group.

Memory allocation for Avi Vantage SE deployments in write access mode is configured via Infrastructure > Cloud > SE Group properties.

SE group editor

There can be a single “default” SE group without any host_attribute_key and host_attribute_value. All SEs created on nodes that don’t find a matching label in any of the other SE groups will be associated with this “default” SE group.

Notes:

  • There can just be a single such “default” SE group.
  • There cannot be more than a single SE group matching the same label key and value.
  • Every node’s labels should just match a single SE group’s label. If a node’s labels match multiple Service Engine groups, the behavior is undefined. The SE can be associated with any of the matching SE groups or none at all.
  • Modification of host_attribute_key and/or host_attribute_value is instantaneously applied and can be disruptive for virtual services. If the modification results in an SE group change for SEs in the SE group, VSs associated with the previous SE group will be moved out to other SEs disruptively. To avoid disruption to virtual services, it is recommended that SE groups be setup during configuration and not modified outside maintenance windows.
  • Creation of a new SE group may be disruptive to existing SEs. Suppose a node is labelled type=test. If there is no SE group that has matching key/values, the SE on this node will be associated with the “default” SE group. When a new SE group with a matching label is created, this SE will be moved to the new SE group and any virtual services associated with this SE will be disruptively moved to other SEs.
  • Similarly, any modification to CPU or memory resource allocations are applied instantaneously, resulting in a re-start of all SEs in the SE group. It is recommended that the modification of CPU or memory resource allocations be undertaken just during maintenance windows.

Deleting a Service Engine Group

An SE group needs to be empty of SEs for deletion. Update the SE group to be deleted with a random host_attribute_key and host_attribute_value. This will migrate all SEs off the SE group. After all SEs are migrated, delete the SE group.

Modification of host_attribute_key and host_attribute_value is applied instantaneously to all SEs in that SE group. This will disrupt all VSs in that SE group, since all SEs in the SE group will be migrated off the SE group.

Associating Virtual Services with SE Groups

Virtual services are associated by default with the “Default-Group” SE Group. VSs can be associated with SE groups other than “Default-Group” in one of two ways.

Method 1: Tagging an OpenShift/K8S namespace with a openshift.io/node-selector annotation like the following matches an SE group prod with host_attribute_key=type and host_attribute_value=prod and all north-south services and routes (VSs) in that namespace are associated with SE group prod.


apiVersion: v1
kind: Namespace
metadata:
  annotations:
   .
   .
   .
    openshift.io/node-selector: type=prod
   .
   .
   .
  name: prod

Method 2:

A route can be explicitly annotated to associate the VS with an SE group. The following annotation associates the VS for route route1 to SE group prod.


metadata:
  name: route1
  annotations:
    avi_proxy: '{"virtualservice": {"se_group_ref": “/api/serviceenginegroup/?name=prod”}}'

Updated: 2018-01-22 07:57:05 +0000