Light Dark Auto

Audit Pipeline

Out of the box, greymatter collects information from user-initiated requests flowing through the service mesh. Each user-initiated request contains data that can be used to determine who is making the request, which services they're accessing, and when the request is made and from where (if properly configured). This data is aggregated for historical analysis, from the mesh level down to individual services, providing capabilities such as:

  • Determining user experience
  • Measuring user activity
  • Following an audit trail of a given user's activity

Audit Pipeline

The greymatter audit pipeline is composed of four main parts.

  1. edge proxy and service sidecars configured to emit audit events to logs
  2. an audit agent watching logs for audit events
  3. the audit agent syncing audit events to ElasticSearch
  4. an audit dashboard for searching and visualizing audit events in ElasticSearch

Configuration

The audit pipeline is enabled by default when deploying greymatter. In https://github.com/<your-org>/greymatter-core/blob/main/inputs.cue, there are two blocks of configuration for the audit pipeline.

A toggle that enables the audit pipeline for a mesh, enabled by default.

enable_audits: bool | *true @tag(enable_audits,type=bool)

A block of configuration that controls the behavior of the audit pipeline.

audits: {
  // index determines the index ID in Elasticsearch. The default naming convention
  // will generate a new index each month. The index configuration can be changed
  // to create more or less indexes depending on your storage and performance requirements.
  index: "gm-audits-%Y-%m"
  // elasticsearch_host can be an IP address or DNS hostname to your Elasticsearch instance.
  elasticsearch_host: ""
  // elasticsearch_port is the port of your Elasticsearch instance.
  elasticsearch_port: 443
  // mounted_certs determines whether the audit dashboard uses certificates
  // mounted at /etc/proxy/tls/sidecar in the audit dashboard's sidecar. 
  mounted_certs: bool | *false
  // elasticsearch_endpoint is the full endpoint containing protocol, host, and
  // port of your Elasticsearch instance. This is used to sync audit data
  // with Elasticsearch.
  elasticsearch_endpoint: "https://\(elasticsearch_host):\(elasticsearch_port)"
}

Once the audit pipeline is correctly configured, the audit dashboard will provide visual analysis of audit data.

Reference

Further capabilities can be built around the greymatter audit pipeline, beyond the out-of-the-box features. Below is a reference containing detailed explanations of each field present in an audit event.

Audit Event Fields

FieldDescriptionValue
id

A GUID, unique for each event.

eventChain

An optional chain of GUIDs.

schemaVersion

The version of the Observable schema in use structure.

originatorTokens

Identifiers (e.g. USER_DN) for originators of this event.

topic

Human readable string to identify/indicate what this event is about.
E.g. service-a-prod, dev-west, greymatter-core.

timestamp

Unix timestamp of when the observable was produced according to the host system.

xForwardedForIp

IP address of the original requester.

systemIp

IP address of the system originating the event.

action

Action being performed on the service.

create, read, update, delete, undelete

payload

Request/Response information (see below)

payload.isSuccessful

true if service responded with a 2xx code.

payload.request

Information about the initial request.

payload.requestEndpoint

The URI path that was invoked. E.g. /admin, /services/catalog/1.0/, /apps

payload.request.headers

all headers sent along with the request

payload.request.body

(optional) The full body sent with the request.

payload.response

Information about the returned response.

payload.response.code

returned HTTP code.

payload.response.headers

all headers sent back to the user.

payload.response.body

(optional) The full body sent back to the user

Example Payload

Below is an example payload that is emitted for every audit event across the edge proxies and sidecars. In addition to the metadata below, the request/response bodies can also be emitted for each transaction. However, due to the high data volume this can incur on the supporting infrastructure, this should be turned on only for systems that can support it or for sidecars/routes in which the data volumes are acceptable.

{
  "_index": "gm-audits-2022-10",
  "_id": "SmcD_YEBBQgviFIwE4dh",
  "_score": 1,
  "_ignored": [
    "payload.request.headers.access_token.keyword",
    "payload.request.headers.cookie.keyword"
  ],
  "_source": {
    "action": "GET",
    "eventChain": [
      "5a723688-037d-11ed-9164-76f9f2bba2dd",
      "5a720eff-037d-11ed-9ea7-128ea9dbcf18"
    ],
    "eventId": "5a720eff-037d-11ed-9ea7-128ea9dbcf18",
    "eventType": "catalog",
    "originatorToken": [
      "James Blonde"
    ],
    "payload": {
      "isSuccessful": true,
      "request": {
        "endpoint": "/summary",
        "headers": {
          ":authority": "demo.greymatter.io:10808",
          ":method": "GET",
          ":path": "/summary",
          "accept": "application/json, text/plain, */*",
          "accept-encoding": "gzip, deflate, br",
          "accept-language": "en-US,en;q=0.9",
          "access_token": "",
          "content-length": "0",
          "cookie": "",
          "gm-observable-chain": "5a723688-037d-11ed-9164-76f9f2bba2dd",
          "referer": "https://demo.greymatter.io:10808/",
          "sec-fetch-dest": "empty",
          "sec-fetch-mode": "cors",
          "sec-fetch-site": "same-origin",
          "sec-gpc": "1",
          "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36",
          "user_dn": "James Blonde",
          "x-envoy-original-path": "/services/catalog/summary",
          "x-forwarded-proto": "https",
          "x-gm-domain": "*:10808",
          "x-gm-route": "catalog",
          "x-gm-rule": "DEFAULT",
          "x-gm-shared-rules": "DEFAULT",
          "x-request-id": "b0f6c3f4-9993-4ed0-8e98-ad8e2d2d1f4b"
        },
        "kafka": {}
      },
      "response": {
        "code": 200,
        "headers": {
          ":status": "200",
          "content-type": "application/json",
          "date": "Thu, 14 Jul 2022 14:00:44 GMT",
          "transfer-encoding": "chunked",
          "user_dn": "b0f6c3f4-9993-4ed0-8e98-ad8e2d2d1f4b",
          "x-envoy-upstream-service-time": "0",
          "x-gm-route": "catalog_local",
          "x-gm-rule": "DEFAULT",
          "x-gm-shared-rules": "DEFAULT",
          "x-request-id": "b0f6c3f4-9993-4ed0-8e98-ad8e2d2d1f4b"
        },
        "kafka": {}
      }
    },
    "schemaVersion": "1.0",
    "systemIp": "10.244.2.218",
    "timestamp": "2022-07-14T14:00:44Z"
  }
}