registration

When sidecars (gm-proxy) register with the control plane, they must connect to gm-control and announce who/where they are.

Connect

Each Grey Matter sidecar connects to the Grey Matter Control server with a bi-directional gRPC stream. This connection is kept alive as long as both servers are up, and updates to configuration will flow every 30 seconds (default, but configurable) from gm-control to each connected gm-proxy.

To properly set up the connection, gm-proxy is run with the following two environment variables.

PROXY_DYNAMIC=true # To run in dynamic configuration mode
XDS_HOST=<gm-control host>
XDS_PORT=<gm-control port>

Optionally Using TLS

When connecting to the Envoy management server, sidecars may also connect with TLS:

PROXY_DYNAMIC=true # To run in dynamic configuration mode
XDS_SERVER_CA_PATH-<gm-control trust path>
XDS_SERVER_CERT_PATH=<gm-control certificate path>
XDS_SERVER_KEY_PATH=<gm-control certificate key path>

Verifying Connection

Repeated sidecar log messages, of the form below, indicate that the connection to gm-control is failing. Usually this is because the address is incorrect or not addressable. If these logs do not appear, the connection is successful.

A single or intermittent occurrence of this error message, often occurs during startup and is not a concern.

[2019-10-11 15:21:51.635][8][warning][config] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, no healthy upstream
[2019-10-11 15:21:51.635][8][warning][config] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:56] Unable to establish new stream

Announce

When the sidecar connects to control, it also sends an announcement that identifies itself to the control plane. This announcement information is used to isolate nodes into zones, determine which configuration options go to which sidecar instance, etc.

Cluster

The service cluster defines what type of service this sidecar is serving. Examples would be "example-service", "user-service", "data", "catalog", etc. This field is used by the control plane to group together all sidecars that share the same cluster so that they'll be properly routed and load-balanced as instances spin up or down.

Zone

The zone is the logical group that the sidecar is running in. This can correlate to actual geographic regions, different slices of the network, or simply logical groups.

The zone that the sidecar announces must match the --api.zone of the gm-control instance the sidecar connects to. If these two are not in agreement, then the sidecar is considered to be in a different zone and will receive no configuration.

Node ID

The node id is generally a unique identifier for this particular sidecar instance, and can be used to take instance specific actions.

This field is not currently used for any operations in the control plane. By default, each node will get a random ID, so it does not need to be set by the user.

Setting Announcement Info

Using the Grey Matter sidecar, the announcement info can be most easily set through the environment variables:

PROXY_DYNAMIC=true # To run in dynamic configuration mode
XDS_CLUSTER=example-service
XDS_ZONE=us-east-1
XDS_NODE_ID=an58xch3mf78

They can also be set directly at the command line when running the binary directly:

gm-proxy -c ./config.yaml \
--service-cluster=example-service \
--service-zone=us-east-1 \
--service-node=an58xch3mf78

Each flag can also be set directly in the bootstrap config template in the node section like shown below.

node:
cluster: example-service
id: n48xng&9#dsfd9
locality:
zone: us-east-1

Configuration Discovery

After successful registration of the sidecar with the control plan, mesh-wide and service-specific configurations will begin flowing to the sidecar. Though gm-control-api houses all the configuration for the mesh, it's ultimately gm-control that turns these configs into full Envoy configuration objects and sends them to the proxies.‌

The configuration in the Control API is mapped to physical proxies by the name field in the proxy API object. It's very important that this field exactly match the service-cluster identifier that the intended target sidecar used when registering with gm-control.

In the example below, the proxy object, and all other objects linked by their appropriate keys, will be turned into a full Envoy configuration and sent to any proxies that announce as a cluster catalog.‌

Services that announce as:

XDS_CLUSTER=catalog
XDS_REGION=default-zone

The sidecar will receive the config object below, because XDS_CLUSTER==name, and they're both in the same zone.

{
"proxy_key": "catalog-proxy",
"zone_key": "default-zone",
"name": "catalog",
"domain_keys": ["catalog"],
"listener_keys": ["catalog-listener"],
"listeners": null,
"active_proxy_filters": ["gm.metrics"],
"proxy_filters": {
"gm_metrics": {
"metrics_port": 8081,
"metrics_host": "0.0.0.0",
"metrics_dashboard_uri_path": "/metrics",
"metrics_prometheus_uri_path": "/prometheus",
"prometheus_system_metrics_interval_seconds": 15,
"metrics_ring_buffer_size": 4096,
"metrics_key_function": "depth"
}
}
}

Questions

Have a question about the Grey Matter Control Plane? Reach out to us at [email protected] to learn more.