Light Dark Auto

Local Development

Deploy a Grey Matter service mesh locally

Grey Matter is a tool for distributed systems, but setting up distributed systems is hard. The easiest way to try out Grey Matter is with a local Kubernetes distribution.

Install a Local Kubernetes Distribution

This guide has been tested with Rancher Desktop with the containerd runtime. Other local development distributions of Kubernetes are known to work, including Rancher Desktop with the dockerd runtime, k3s, Kind, and microk8s. But, from now on, we'll assume Rancher Desktop is running.

The minimum resource allocation for Rancher Desktop is 2 GB RAM and 2 vCPUs, but you'll want more than that for this guide. We recommend at least 5 GB RAM and 4 vCPUs.

Launch the Grey Matter Operator

The operator can be launched with the greymatter CLI. If you don't have the CLI, get it here.

Put your Grey Matter account credentials into these environment variables.

export GREYMATTER_REGISTRY_USERNAME='your.name@example.net'
export GREYMATTER_REGISTRY_PASSWORD='my-password'

In the same terminal window, run the following. You may omit the actual kubectl application at the end of the line, if you only want to inspect the manifests that will be applied to set up the operator. Note that this assumes your method of installing Kubernetes has already configured your kubectl, which is common but not universal. Rancher Desktop does configure itself as the "current" Kubernetes environment in kubectl for you.

greymatter k8s operator --registry-username $GREYMATTER_REGISTRY_USERNAME \
                        --registry-password $GREYMATTER_REGISTRY_PASSWORD \
                        --image docker.greymatter.io/release/gm-operator:0.3.3 \
           | kubectl apply -f -

Watch the operator and Spire pods come up with kubectl get pods --all-namespaces -w (which watches pod status) and kubectl logs -n gm-operator -f deployment/gm-operator (which watches the operator logs). Wait for the operator to start up and launch the Spire server and agent pods. At the moment, this can take several minutes on slower machines, or on emulated architectures. The Spire agent especially can take long enough to put it in a crash backoff on emulated machines. Also note that containers may go through intermediate states where they appear not to be functioning properly. This is normal. When Kubernetes reports that the operator, Spire server, and Spire agent are running, and the operator logs say "Starting certificate watcher", "serving webhook server", and "registering webhook.../mutate-workload", then the operator is ready.

Typical final state:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-6c79684f77-vsqlp   1/1     Running     0          12m
kube-system   coredns-5789895cd-9mqkx                   1/1     Running     0          12m
kube-system   helm-install-traefik-crd-h8rnk            0/1     Completed   0          12m
kube-system   helm-install-traefik-b6fkr                0/1     Completed   0          12m
kube-system   svclb-traefik-hjd49                       2/2     Running     0          12m
kube-system   metrics-server-7cd5fcb6b7-qttw6           1/1     Running     0          12m
kube-system   traefik-6bb96f9bd8-mcbpk                  1/1     Running     0          12m
gm-operator   gm-operator-658b44dd5b-244r5              1/1     Running     0          3m57s
spire         server-0                                  2/2     Running     0          2m33s
spire         agent-tsmn5                               1/1     Running     0          2m34s

Typical final operator log lines before mesh installation:

...
{"level":"info","ts":1644961089.9993546,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1644961089.999927,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1644961090.0003026,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-workload"}

Launch a mesh

The operator creates a custom resource definition (CRD) named "Mesh", which can be used to launch and configure a Grey Matter mesh. Check out the Mesh CRD documentation.

The following is a fully functioning example.

apiVersion: greymatter.io/v1alpha1
kind: Mesh
metadata:
  name: mesh-sample
spec:
  release_version: '1.7'
  zone: default-zone
  install_namespace: greymatter
  watch_namespaces:
  - default
---
apiVersion: v1
kind: Namespace
metadata:
  name: greymatter

Save this into a file named mesh.yaml, and run the following:

kubectl apply -f mesh.yaml

This instructs the operator to create a version 1.7 mesh named mesh-sample in the greymatter namespace. This configuration also watches the default namespace for service discovery purposes. All generated configuration for this mesh will be scoped to the (arbitrarily named) default-zone zone.

This will instruct the operator to begin deploying the requested mesh. Wait for less than one galactic rotation, while watching the operator logs. At some point, the mesh components will be deployed into Kubernetes, but the operator will be waiting on Catalog and Control API to be ready. Once they are, the operator will complete the installation by applying its bundle of Grey Matter configurations for all sidecars and Edge.

Typical intermediate state. Note the transient error from the edge pod. This is normal:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   local-path-provisioner-6c79684f77-vsqlp   1/1     Running             0          22m
kube-system   coredns-5789895cd-9mqkx                   1/1     Running             0          22m
kube-system   helm-install-traefik-crd-h8rnk            0/1     Completed           0          22m
kube-system   helm-install-traefik-b6fkr                0/1     Completed           0          22m
kube-system   svclb-traefik-hjd49                       2/2     Running             0          22m
kube-system   metrics-server-7cd5fcb6b7-qttw6           1/1     Running             0          22m
kube-system   traefik-6bb96f9bd8-mcbpk                  1/1     Running             0          22m
gm-operator   gm-operator-658b44dd5b-244r5              1/1     Running             0          13m
spire         server-0                                  2/2     Running             0          12m
spire         agent-tsmn5                               1/1     Running             0          12m
greymatter    svclb-edge-zgn9s                          1/1     Running             0          21s
greymatter    jwt-security-7b4654c4f5-tlpdb             0/2     ContainerCreating   0          18s
greymatter    control-69fb87b98b-bblqv                  0/3     ContainerCreating   0          18s
greymatter    catalog-5d4577bc5b-rmmd7                  0/2     ContainerCreating   0          17s
greymatter    dashboard-6c477d76bd-wlv5r                0/2     ContainerCreating   0          16s
greymatter    gm-redis-0                                0/2     ContainerCreating   0          18s
greymatter    edge-6b95bd8568-mmvrh                     0/1     ImagePullBackOff    0          21s

Once this is done, all pods should be Running, but the configuration is still applying.

The operator, applying Grey Matter configuration:

...
{"level":"info","ts":1644961924.3984356,"logger":"cli","msg":"apply","type":"proxy","key":"catalog","Mesh":"mesh-sample"}
{"level":"info","ts":1644961924.6001592,"logger":"cli","msg":"apply","type":"catalogservice","key":"catalog","Mesh":"mesh-sample"}
{"level":"info","ts":1644961926.7098475,"logger":"cli","msg":"apply","type":"catalogservice","key":"dashboard","Mesh":"mesh-sample"}
{"level":"info","ts":1644961926.7156723,"logger":"cli","msg":"apply","type":"proxy","key":"dashboard","Mesh":"mesh-sample"}
{"level":"info","ts":1644961929.1093829,"logger":"cli","msg":"modify","type":"listener","key":"gm-redis","Mesh":"mesh-sample"}
{"level":"info","ts":1644961931.3026,"logger":"cli","msg":"apply","type":"proxy","key":"control","Mesh":"mesh-sample"}
{"level":"info","ts":1644961931.3145318,"logger":"cli","msg":"apply","type":"catalogservice","key":"control","Mesh":"mesh-sample"}

Once the operator has finished applying mesh configuration, wait for the configuration to propagate. Once your mesh is fully configured, you should be able to reach the Grey Matter dashboard at http://localhost:10808

One convenient way to know what everything has applied and the external port has been exposed is to just put http://localhost:10808 into your browser and wait. Chrome (and some other browsers) automatically retry periodically, so when the port becomes available, the browser will automatically refresh.

Note: Note the mesh is exposed without SSL/TLS. Because the platform is usually under an external load balancer in production, and that load balancer manages the server certificate, the operator installs without securing the edge ingress. This is normal and expected, but you will not be able to access your local mesh installation with https://.

If this is what you see, congratulations. You've just deployed your first Grey Matter service mesh. You're ready to explore how you can configure Grey Matter with GitOps.