Light Dark Auto

Kubernetes

Kubernetes service discovery

greymatter.io Fabric supports service discovery from Kubernetes. See the greymatter.io Control Kubernetes discovery setup documentation for how to configure this with greymatter.io Control.

Kubernetes Deployments

Kubernetes (k8s) has a number of internal APIs that are used for the complex orchestration of containers. When using k8s as the underlying orchestration platform, greymatter.io Control can utilize some of these APIs to also do easy service announcement and discovery.

The greymatter.io Control server will discover services based on their pod IP and container port, see the kubernetes specifications in the setup docs for how to configure your deployments for service discovery. Also, see the example deployment.

Behavior

As described in the greymatter.io Control service discovery kubernetes docs, the greymatter.io Control server will discover from namespaces specified on startup by the environment variable GM_CONTROL_KUBERNETES_NAMESPACES. In the case that a Kubernetes namespace that Control is configured to discover from is or becomes undiscoverable (see permissions), the Control server follows a specific pattern of behavior.

If a namespace goes down, the Control server will label the connection to the namespace as having a bad state. It will retain the last known state of discovered pods for the namespace and generate clusters for them. If the namespace eventually becomes discoverable again, it will update its pod list accordingly. In the meantime, it will continue to poll the unavailable namespace, and it will be left up to EDS to determine the endpoint health for the last known state of the pods.

Example Deployment

The Kubernetes Deployment below is properly setup (label and port) to be discovered by the greymatter.io Control server.

apiVersion: apps/v1
kind: Deployment
spec:
  selector:
    matchLabels:
      app: example
  replicas: 1
  template:
    metadata:
      labels:
        gm_cluster: example
    spec:
      containers:
      - name: example-service
        image: docker.greymatter.io/internal/example-service:latest
      - name: sidecar
        image: docker.greymatter.io/release/gm-proxy:1.2.1
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 9080
        - name: metrics
          containerPort: 8081
        env:
        - name: PROXY_DYNAMIC
          value: "true"
        - name: XDS_CLUSTER
          value: example
        - name: XDS_HOST
          value: gm-control
        - name: XDS_PORT
          value: "50000"

Kubernetes Discovery

Kubernetes (k8s) has several internal APIs to orchestrate containers. When using k8s as the underlying orchestration platform, gm-control can use these APIs for easy service announcement and discovery.

Pod Label and Named Port

To discover services, you must configure two important bits of information on each deployed pod:

  • the cluster label
  • the port name

Without these two pieces of information, the control plane will ignore the pods.

Cluster Label

The cluster label is a small piece of metadata attached to the pod to determine what specific service is running in this pod. All pods of the same service will be grouped together and load balanced in the mesh. The default metadata key that will be used is gm_cluster=<service_name>, but this is user configurable.

Port Name

The port name determines which named port to expose in the mesh.

Since it's possible for many ports to be open for different purposes, just one must be designated for routing normal traffic. This defaults to the port named http, but is user configurable.

Example Deployment

The Kubernetes Deployment below is properly set up (label and port) to be discovered by the gm-control server.

apiVersion: apps/v1
kind: Deployment
spec:
  selector:
    matchLabels:
      app: example
  replicas: 1
  template:
    metadata:
      labels:
        gm_cluster: example
    spec:
      containers:
      - name: example-service
        image: deciphernow/example-service:latest
      - name: sidecar
        image: docker.production.deciphernow.com/deciphernow/gm-proxy:0.9.1
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 9080
        - name: metrics
          containerPort: 8081
        env:
        - name: PROXY_DYNAMIC
          value: "true"
        - name: XDS_CLUSTER
          value: example
        - name: XDS_HOST
          value: gm-control
        - name: XDS_PORT
          value: "50000"

Service Accounts

To discover services from the internal Kubernetes APIs, the pod running the Control server must have additional permissions granted by an admin. To grant this access, have a cluster admin apply one of the following resources (replacing the greymatter namespace with the namespace of the running control server), and then add the service account to the pod running gm-control.

spec:
  serviceAccountName: control

Single Namespace

This is used when gm-control is only discovering services in the same namespace it's running in.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: control
  namespace: greymatter

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: control-manager
  namespace: greymatter
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]

---

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: control-binding
  namespace: greymatter
subjects:
- kind: ServiceAccount
  name: control
  namespace: greymatter
roleRef:
  kind: Role
  name: control-manager
  apiGroup: rbac.authorization.k8s.io

Multiple Namespaces

This is used when gm-control will be discovering services from multiple namespaces across the cluster.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: control
  namespace: greymatter

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: control-manager
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: control-binding
subjects:
- kind: ServiceAccount
  name: control
  namespace: greymatter
roleRef:
  kind: ClusterRole
  name: control-manager
  apiGroup: rbac.authorization.k8s.io