When sidecars (gm-proxy) register with the control plane, they must connect to gm-control and announce who/where they are.
Connect
Each greymatter.io sidecar connects to the greymatter.io Control server with a bi-directional gRPC stream. This connection is kept alive as long as both servers are up, and updates to configuration will flow every 30 seconds (default, but configurable) from gm-control to each connected gm-proxy.
To properly set up the connection, gm-proxy is run with the following two environment variables.
PROXY_DYNAMIC=true # To run in dynamic configuration mode
XDS_HOST=<gm-control host>
XDS_PORT=<gm-control port>
Optionally Using TLS
When connecting to the Envoy management server, sidecars may also connect with TLS:
PROXY_DYNAMIC=true # To run in dynamic configuration mode
XDS_SERVER_CA_PATH-<gm-control trust path>
XDS_SERVER_CERT_PATH=<gm-control certificate path>
XDS_SERVER_KEY_PATH=<gm-control certificate key path>
Verifying Connection
Repeated sidecar log messages, of the form below, indicate that the connection to gm-control is failing. Usually this is because the address is incorrect or not addressable. If these logs do not appear, the connection is successful.
[2019-10-11 15:21:51.635][8][warning][config] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, no healthy upstream
[2019-10-11 15:21:51.635][8][warning][config] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:56] Unable to establish new stream
Announce
When the sidecar connects to control, it also sends an announcement that identifies itself to the control plane. This announcement information is used to isolate nodes into zones, determine which configuration options go to which sidecar instance, etc.
Cluster
The service cluster defines what type of service this sidecar is serving. Examples would be “example-service”, “user-service”, “data”, “catalog”, etc. This field is used by the control plane to group together all sidecars that share the same cluster so that they’ll be properly routed and load-balanced as instances spin up or down.
Zone
The zone is the logical group that the sidecar is running in. This can correlate to actual geographic regions, different slices of the network, or simply logical groups.
Node ID
The node id is generally a unique identifier for this particular sidecar instance, and can be used to take instance specific actions.
Setting Announcement Info
Using the greymatter.io sidecar, the announcement info can be most easily set through the environment variables:
PROXY_DYNAMIC=true # To run in dynamic configuration mode
XDS_CLUSTER=example-service
XDS_ZONE=us-east-1
XDS_NODE_ID=an58xch3mf78
They can also be set directly at the command line when running the binary directly:
gm-proxy -c ./config.yaml \
--service-cluster=example-service \
--service-zone=us-east-1 \
--service-node=an58xch3mf78
Each flag can also be set directly in the bootstrap config template in the node section like shown below.
node:
cluster: example-service
id: n48xng&9#dsfd9
locality:
zone: us-east-1
Configuration Discovery
After successful registration of the sidecar with the control plan, mesh-wide and service-specific configurations will begin flowing to the sidecar. Though gm-control-api houses all the configuration for the mesh, it’s ultimately gm-control that turns these configs into full Envoy configuration objects and sends them to the proxies.
The configuration in the Control API is mapped to physical proxies by the name field in the proxy API object. It’s very important that this field exactly match the service-cluster identifier that the intended target sidecar used when registering with gm-control.
In the example below, the proxy object, and all other objects linked by their appropriate keys, will be turned into a full Envoy configuration and sent to any proxies that announce as a cluster catalog.
Services that announce as:
XDS_CLUSTER=catalog
XDS_REGION=default-zone
The sidecar will receive the config object below, because XDS_CLUSTER==name, and they’re both in the same zone.
{
"proxy_key": "catalog-proxy",
"zone_key": "default-zone",
"name": "catalog",
"domain_keys": ["catalog"],
"listener_keys": ["catalog-listener"],
"listeners": null,
"active_proxy_filters": ["gm.metrics"],
"proxy_filters": {
"gm_metrics": {
"metrics_port": 8081,
"metrics_host": "0.0.0.0",
"metrics_dashboard_uri_path": "/metrics",
"metrics_prometheus_uri_path": "/prometheus",
"prometheus_system_metrics_interval_seconds": 15,
"metrics_ring_buffer_size": 4096,
"metrics_key_function": "depth"
}
}
}