Metrics (HTTP)

The HTTP Metrics filter performs several functions:

1. It serves all of Envoy’s statistics, as well as some custom statistics, which are scraped by greymatter’s packaged Prometheus instance and also displayed by the Sense app.
2. It also scrapes itself, and beacons select metrics to Catalog to be used in the health check system (also displayed in the Sense app).
3. Finally, it provides an endpoint that the Sense app can use directly to request the instance metrics from an individual sidecar.

Custom Stats

Greymatter’s Metrics filter produces several families of custom metrics on requests, and breaks them out by HTTP verb and path.

• ...all...: When request statistics are broken out by, e.g., HTTP verb, Greymatter also adds all metrics to roll up those counts.
• status_{response_code} stat per unique request path (to the depth specified by metrics_key_depth - see below) which counts the number of responses with that code. It also rolls them up into 2XX, 3XX, etc.
• latency_ms, latency_max, latency_count, and latency_sum
• in_throughput and out_throughput (in bytes)
• bytes_received and ns_received (nanoseconds it took to receive that many bytes), and similarly bytes_sent and ns_sent
• total_requests, http(s)_requests, and system/start_time. It also tracks errors for 400-and-above status codes.

Configuration

Since the Metrics filter is on by default and already configured to support Sense, you may not need to configure anything. However, there are a few configuration options exposed to users. The Metrics filter is configured using the metrics_options key under a gsl.#HTTPListener, with the following keys.

• metrics_port: This option specifies the port on which the Metrics filter will listen for incoming connections (to serve requests from the Sense app). By convention, this is usually set to 8081.
• metrics_ring_buffer_size: This option specifies the size of the ring buffer that stores metrics data. The default value is 4096, meaning that it will maintain a buffer of 4096 metrics snapshots before dropping the oldest ones. This affects how far back into the past instance metrics can be displayed in Sense.
• prometheus_system_metrics_interval_seconds: This option specifies the interval at which system metrics are queried and made available to Prometheus (though Prometheus retrieves them on its own interval). The default value is 15 seconds.
• metrics_key_depth: How many path segments to track when producing custom metrics (see above). By default, with the depth set to “1”, only differences in the first /-separated path segment will add a new metric to track. Higher values will create new metrics based on differences later in the path, e.g., /story/23/01/25/168213 has five path segments, and the default metrics_key_depth will only keep a special counter for different values of the first one, so a subsequent request to /poll/3234 would start a new counter for /poll, and a subsequent request to /poll/1111 would only increment that /poll counter, not create a new one for the 1111 in the second path segment.

In addition to the configuration under metrics_options, it is possible to configure where health check statistics are sent with the health_options key at the top level of a gsl.#Service or gsl.#Edge. A common use of this is to configure mTLS with Spire to greymatter-datastore. For example:

Edge: gsl.#Edge & {
	context: Edge.#NewContext & globals
	name:              "edge"
  ...
	health_options: {
		spire: gsl.#SpireUpstream & {
			#context: context.SpireContext
			#subjects: ["greymatter-datastore"]
		}
	}
  ...

Example

Here is an example of a gsl.#Service with the Metrics filter configured to use a custom metrics_key_depth, and also to beacon health check information using Spire-based mTLS (demonstrating the use of health_options):

Fakeservice: gsl.#Service & {
	// A context provides global information from globals.cue
	// to your service definitions.
	context: Fakeservice.#NewContext & globals

	// name must follow the pattern namespace/name
  ...
	health_options: {
		spire: gsl.#SpireUpstream & {
			#context: context.SpireContext
			#subjects: ["greymatter-datastore"]
		}
	}
	ingress: {
		(name): {
			gsl.#HTTPListener

      metrics_options: {
        metrics_key_depth: "2"
      }
  ...
	
}