Grey Matter Documentation

The Grey Matter Metrics Filter sets up a local metrics server to gather and report real-time statistics for the sidecar, microservice, and host system.

Prometheus

Optionally, this filter can serve the computed statistics in a form suitable for scraping by Prometheus. The prometheus endpoint will be hosted at {METRICS_HOST}:{METRICS_PORT}{METRICS_PROMETHEUS_URI_PATH}, which can then be scraped directly through the supported Prometheus service discovery mechanisms.

Example Configuration

http_filters:
- name: gm.metrics
  config:
      metrics_dashboard_uri_path: "/metrics"
      metrics_host: "0.0.0.0"
      metrics_key_depth: 1
      metrics_key_function: "depth"
      metrics_port: 8081
      metrics_prometheus_uri_path: "/prometheus"
      use_metrics_tls: false
      metrics_receiver:
        push_interval_seconds: 10
        redis_connection_string: "redis://127.0.0.1:10909"
      metrics_ring_buffer_size: 4096
      prometheus_system_metrics_interval_seconds: 15

Per-Route configuration

There are some parameters that can be configured per route.

{
  "metrics_key_function": <string>,
  "metrics_key_depth": <string>
}

metrics_key_function can be none or depth.

none provide no rollup of the URL meaning the filter will output metrics for each successive path
depth allows a user to specify the rollup level based on the setting of metrics_key_depth

Let's say we have following endpoints:

/apis/my-service/stores/
/apis/my-service/users/37
/apis/another-service/featured/2020/09
/apis/another-service/home.html

With metrics_key_depth of 1, the average response time for the above routes get rolled up to one key:

/apis

If you chose metrics_key_depth of 2, the same URLs get rolled up to two:

/apis/my-service
/apis/another-service

This would likely give you an idea of the average response time for each micro service. If URLs are structured as something like https://[domain]/[service]/ in your environment, you can get the same granularity of the information for metrics_key_depth of 1 (i.e. key="/my-service" and key="/another-service").

If you chose metrics_key_depth of 3, the URLs in the example would get rolled up to:

/apis/my-service/stores/
/apis/my-service/users/
/apis/another-service/featured/
/apis/another-service/home.html

The greater metrics_key_depth, the finer-grained metrics you will end up with for analysis. However, there are some tradeoffs to consider.

Every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality, such as user IDs, email addresses, or other unbounded sets of values.

Many high-cardinality data series can cause the disk and memory requirements of Prometheus to explode.