Light Dark Auto

Data Plane

Configuration variables and mini guides for the greymatter.io proxy.

greymatter.io maintains its own proxy with special enterprise featured filters built in at compile time.

Connect to greymatter-control

Each proxy connects to the greymatter-control server with a bi-directional gRPC stream. This connection is kept alive as long as both servers are up, and updates to configuration will flow every 30 seconds (default; but configurable) from Control to each connected Proxy.

Set Up the Connection

To properly set up the connection, run the proxy with the following three environment variables.

PROXY_DYNAMIC=true   # To run in dynamic configuration mode

XDS_HOST=<control host>
XDS_PORT=<control port>

Use TLS (Optional)

When connecting to the control management server, proxies may also connect with TLS:

XDS_SERVER_CA_PATH-<control trust path>
XDS_SERVER_CERT_PATH=<control certificate path>
XDS_SERVER_KEY_PATH=<control certificate key path>

Announce to greymatter-control

When the proxy connects to control, it sends an announcement that identifies itself to the control plane. This announcement information isolates nodes into zones, determining which configuration options go to which proxy instance, etc...

Cluster

The service cluster defines what type of service this proxy is serving. Examples include:

  • example-service
  • user-service
  • data
  • catalog
  • etc.

This field is used by the control plane to group together all proxies that share the same cluster so that they'll be properly routed and load-balanced as instances spin up or down.

Zone

The zone is the logical group that the proxy is running in. This can correlate to actual geographic regions, different slices of the network, or simply logical groups.

Node ID

The node id is generally a unique identifier for this particular proxy instance, and can be used to take instance specific actions.

Set Announcement Info

Using the proxy, you can set the announcement info most easily through the environment variables:

XDS_CLUSTER=example-service
XDS_ZONE=us-east-1
XDS_NODE_ID=an58xch3mf78

You can also set these environment variables directly at the command line when running the binary:

gm-proxy -c ./config.yaml \
    --service-cluster=example-service \
    --service-zone=us-east-1 \
    --service-node=an58xch3mf78

You can also set each flag directly in the bootstrap config template in the node section, as shown below:

node:
  cluster: example-service
  id: n48xng&9#dsfd9
  locality:
    zone: us-east-1

Verify Connection

If you receive repeated proxy log messages in the form below, it means that the connection to gm-control is failing. Usually this is because the address is incorrect or not addressable. If these logs do not appear, the connection is successful.

[2019-10-11 15:21:51.635][8][warning][config] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, no healthy upstream
[2019-10-11 15:21:51.635][8][warning][config] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:56] Unable to establish new stream

Configuration Variables

Environment variables are the primary configuration method for the proxy.

  • ACCEPT_HTTP_10 - Accept HTTP/1.0 connections on the Envoy static listener.
  • ACL_ENABLED - Enables 2-Way SSL impersonation REST filter.
  • ACL_SERVER_LIST - A list of server DNs to be whitelisted (pipe delimited).
  • AWS_ACCESS_KEY_ID - AWS provided access key credential.
  • AWS_CONFIG_FILE - Location of the local AWS config.
  • AWS_PROFILE - A locally defined AWS profile name associated with valid AWS credentials.
  • AWS_REGION - AWS defined region.
  • AWS_SECRET_ACCESS_KEY - AWS provided secret access key credential
  • CW_DIMENSIONS - The dimension names/values that the specified metrics will be stored under.
  • CW_ENABLED - Enable Amazon CloudWatch metrics collection.
  • CW_METRICS_ROUTES - Regular expression describing routes to be recognized
  • CW_METRICS_VALUES - Values reported to Amazon Cloudwatch
  • CW_NAMESPACE - Customize namespace where metrics will be stored
  • DELAY_MEAN - Obfuscation delay mean
  • DELAY_STD - Obfuscation delay std
  • DELTA_XDS - Enable dynamic configuration from greymatter.io xDS using the incremental protocol variant.
  • DRAIN_TIMEOUT - Listener drain timeout.
  • EGRESS_CA_CERT_PATH - Egress trust certificate path.
  • EGRESS_CERT_PATH - Egress certificate path.
  • EGRESS_KEY_PATH - Egress key certificate path.
  • EGRESS_TLS_CERT - Base64 encoded cert written out to ./certs/egress_localhost.crt
  • EGRESS_TLS_KEY - Base64 encoded key written out to ./certs/egress_localhost.key
  • EGRESS_TLS_TRUST - Base64 encoded trust written out to ./certs/egress_intermediate.crt
  • EGRESS_USE_TLS(default: false) - Enable 2-way SSL to the proxied service.
  • ENVOY_ADMIN_HOST(default: 0.0.0.0) - The host the envoy admin server will listen on.
  • ENVOY_ADMIN_LOG_PATH(default: **/dev/null**) - File descriptor envoy writes logs to.
  • ENVOY_ADMIN_PORT(default: **8001**) - The port the envoy admin server will listen on
  • ENVOY_CONFIG - Base64 encoded string of envoy configuration file
  • ENVOY_LOG_LEVEL
  • GM_CONFIG - Base64 encoded string of gm-config.yaml configuration file
  • HOST(default: 0.0.0.0) - Host for Envoy listener
  • IDLE_TIMEOUT - Cluster idle timeout, in seconds; example 3000s
  • INGRESS_CA_CERT_PATH - Ingress trust certificate path
  • INGRESS_CERT_PATH - Ingress certificate path
  • INGRESS_KEY_PATH - Ingress key certificate path
  • INGRESS_TLS_CERT - Base64 encoded cert written out to ./certs/ingress_localhost.crt.
  • INGRESS_TLS_KEY - Base64 encoded key written out to ./certs/ingress_localhost.key.
  • INGRESS_TLS_TRUST - Base64 encoded key written out to ?????
  • INGRESS_USE_TLS - Whether to use TLS for Ingress.
  • INHEADERS_ENABLED(default: false) - Setup Impersonation headers.
  • KAFKA_ENABLED(default: false) - Enable event emission to a Kafka topic.
  • KAFKA_SERVER_CONNECTION(default: localhost:9091) - Comma-separated list of of Kafka brokers.
  • KAFKA_SERVER_NAME - Server name to be used when connecting to Kafka over TLS.
  • KAFKA_TLS_CERT - Certificate to be used when connecting to Kafka over TLS; example: file:///opt/certs/certificate.pem.
  • KAFKA_TLS_KEY - Certificate key to be used when connecting to Kafka over TLS.
  • KAFKA_TLS_TRUSTS - Certificate authorities to be used when connecting to Kafka over TLS (command de-limited).
  • KAFKA_ZK_DISCOVER(default: false) - Discovery of Kafka brokers from ZooKeeper.
  • LISTAUTH_BLACKLIST - List of DNs to be blacklisted (pipe delimited).
  • LISTAUTH_ENABLED - Enable/disable the whitelist/blacklist feature.
  • LISTAUTH_WHITELIST - List of DNs to be whitelisted (pipe delimited).
  • METRICS_FABRIC_PATH(default: **/metrics**) - Route for metrics collection.
  • METRICS_PORT(default: 8081) - Port for metrics listener.
  • METRICS_PROMETHEUS_PATH(default: **/prometheus**) - Route for Prometheus metrics collection.
  • METRICS_USE_TLS(default: **false**) - Metrics listener uses TLS.
  • OAUTH_CLIENT_ID - Client ID issued by the authorization server.
  • OAUTH_CLIENT_SECRET - Client secret issued by the authorization server.
  • OAUTH_DOMAIN - Provider domain.
  • OAUTH_ENABLED - Enable full OAuth 2.0 functionality.
  • OAUTH_SERVER_INSECURE(default: false) - Skip verification for OAuth server.
  • OAUTH_SERVER_NAME - Authorization server name.
  • OAUTH_SESSION_SECRET - OAuth session secret.
  • OBS_ENABLED(default: false) - Enables event emission to various brokers.
  • OBS_FULL_RESPONSE(default: false) - If true, dump full request and response bodies to Observables pipeline.
  • OBS_KAFKA_TOPIC(default: **gm-sidecar-events**) - Kafka topic to send observables on.
  • OBS_TOPIC - Topic for the observable event. Sets eventType in the payload.
  • PORT(default: 8080) - Port for Envoy listener.
  • PROXY_DYNAMIC - Boolean - if false, only static config is used.
  • REDIS_CLUSTER - Envoy cluster name for for proxyied Redis requests with a configured Redis proxy filter.
  • REDIS_HOST(default: redis_server) - The host of a Redis server.
  • REDIS_PORT(default: 3679) - The port of a Redis server.
  • REDIS_SNI - What Server Name Indication (SNI) to assign to the Redis cluster.
  • SERVICE_DNS_TYPE(default: STRICT_DNS) - The type of DNS envoy will use to connect to the static cluster.
  • SERVICE_HOST(default: 0.0.0.0) - Proxied service host.
  • SERVICE_PORT - Proxied service port.
  • SPIRE_CERT_PATH - The path of a Spire agent certificate used to create an mTLS connection.
  • SPIRE_HOST - The host a Spire agent is listening on if connecting over mTLS.
  • SPIRE_KEY_PATH - The path of a Spire agent certificate key used to create an mTLS connection.
  • SPIRE_PATH - The Unix domain socket path Envoy will use to connect to a Spire agent.
  • SPIRE_PORT - The port a Spire agent is listening on if connecting over mTLS.
  • TCP_CLUSTER - Name to assign the cluster that will be used for proxying requests with a configured TCP proxy filter.
  • TCP_HOST - The host of a server that receives TCP connections.
  • TCP_PORT(default: 3000) - The port of a server that receives TCP connections.
  • TCP_SNI - What Server Name Indication (SNI) to assign to the TCP cluster.
  • TIMEOUT - Cluster route timeout.
  • TRACING_ADDRESS - The host of the trace collector server.
  • TRACING_CA_CERT_PATH - Trace server trust certificate path.
  • TRACING_CERT_PATH - Trace server certificate path.
  • TRACING_COLLECTOR_ENDPOINT_VERSION(default: **HTTP_JSON**) - API Version of the tracing collector endpoint.
  • TRACING_COLLECTOR_ENDPOINT(default: **/api/v1/spans**) - Used by Zipkin and Opencensus (only when exporting to Zipkin). Endpoint on the tracing server to send spans..
  • TRACING_DATADOG_SERVICE_NAME(default: **gm-proxy**) - Used by Datadog. A unique identifier to display in the Datadog dashboard.
  • TRACING_DRIVER - One of: "zipkin", "lightstep", "datadog", "opencensus", "instana".
  • TRACING_ENABLED(default: false) - Turn on request tracing using the Zipkin config.
  • TRACING_INSTANA_LIBRARY_PATH(default: /app/instana_sensor.so) - The path of the Instana library file to run when sending spans to Instana.
  • TRACING_KEY_PATH - Trace server key certificate path.
  • TRACING_LIGHTSTEP_ACCESS_TOKEN_PATH - Used by Lightstep. Path to file containing the access token to the LightStep API.
  • TRACING_OPENCENSUS_CONTEXT_HEADER - Header for manually tracking traces across services. Accepts "traceparent", "grpc-trace-bin", "x-cloud-trace-context", "x-b3-*".
  • TRACING_OPENCENSUS_EXPORTER - Receives "ocagent", "stackdriver", "zipkin".
  • TRACING_OPENCENSUS_STACKDRIVER_PROJECT_ID - The cloud project_id to use when exporting to Stackdriver.
  • TRACING_PORT - The port of the trace collector serve.
  • TRACING_USE_TLS - Communicate to the trace server via TLS.
  • USE_HTTP2 - Enable HTTP/2 for static cluster connection (doesn't work with HTTP/1.0).
  • USE_KAFKA_TLS(default: false) - Enable TLS communication with Kafka nodes.
  • XDS_CLUSTER - Envoy xDS proxy cluster identifier.
  • XDS_ENABLE_TLS - Enable TLS when communicating with the xDS server.
  • XDS_HOST - Host of greymatter.io xDS Server.
  • XDS_NODE_ID - Envoy node id per xds configuration.
  • XDS_PORT(default: 18000) - Port of greymatter.io xDS Server.
  • XDS_SERVER_CA_PATH(default: certs/xds_server_ca.crt) - Path to CA file to be used for connecting to xDS.
  • XDS_SERVER_CERT_PATH(default: certs/xds_server_cert.crt) - Path to certificate file to be used for connecting to xDS.
  • XDS_SERVER_KEY_PATH(default: certs/xds_server_key.key) - Path to key file to be used for connecting to xDS.
  • XDS_ZONE - The zone this proxy is a member of.
  • ZK_ADDRS(default: localhost:2181) - comma-separated list of host:port of ZooKeeper nodes.
  • ZK_ANNOUNCE_HOST(default: 0.0.0.0) - Host of the original service.
  • ZK_ANNOUNCE_PATH - Zookeeper discovery path.

Command Line Flags

A few proxy features can be configured with command line flags.

  • --config - Path to config.yaml; Useful for providing static configuration to gm-proxy. An Envoy bootstrap config is generated, if not provided.
  • --service-cluster - Cluster announced to control plane.
  • --service-zone - Zone announced to control plane; must match api.zone configured in gm-control.