Light Dark Auto

Observables

HTTP Telemetry Filter

The Observables Filter configures the Proxy to emit a JSON payload with every request made to the microservice. This JSON payload contains a variety of different information about the request being made, as well as the user/system issuing the request. These Observables can then be aggregated to perform analysis like: audits, user-experience tracking, etc.

Observable publishing defaults to stdout but can also be published to a Kafka topic or location on disk.

Filter Configuration Options

  • emitFullResponse(Boolean, default: false) - *Show response body in the observable object.
  • topic(String, default: "") - Sets the eventType field of the observable; used to sort and group messages by service/region/environment/etc in later analysis..
  • useKafka(Boolean, default: false) - .Publish observable message to a Kafka topic
  • enforceAudit(Boolean, default: false) - Block requests until an observable has been successfully published to Kafka (Only applies if useKafke=true).
  • encryptionAlgorithm(String, default: "") - Type of encryption. Must be ‘aes’ or blank.
  • encryptionKey(String, default: "") - Must be blank or base 64 encoded string of 16, 24, or 32 bytes. We recommend 32..
  • encryptionKeyID(uint32, default: 0) - User supplied number to identify the key used in encryption.
  • eventTopic(String, default: "") - The Kafka topic that will hold the published observable messages.
  • kafkaZKDiscover(Boolean, default: false) - Kafka will be discovered through a zookeeper node.
  • kafkaServerConnection(String, default: "") - Comma delimited list of Kafka addresses, or if kafkaZKDiscover is true, a list of ZooKeeper addresses.
  • useKafkaTLS(Boolean, default: false) - Enable TLS communication to the supplied kafka brokers.
  • kafkaCAs(String, default: "") - List of file URLs that point to trusts to be used when connecting to kafka.
  • kafkaCertificate(String, default: "") - File URL pointing to certificate to use when connecting to kafka over TLS.
  • kafkaCertificateKey(String, default: "") - .File URL pointing to certificate key to use when connecting to kafka
  • kafkaServerName(String, default: "") - Certificate server name to use when connecting to kafka.

Payloads

Turning on full response payloads (emitFullResponse=true) can cause significant amounts of data to be written to the payload. The observables filter will attempt to write the entire response, even if it’s a gigabyte payload, so take care.

Encryption

Payload encryption can be enabled in the filter config. Each encryption key should be assigned a unique ID.

http_filters:
- name: gm.observables
  config:
    emitFullResponse: true
    useKafka: false
    enforceAudit: false
    logLevel: debug
    encryptionAlgorithm: aes
    encryptionKey: kvTujluRwliCWBWQvvvIxQr2Fxw3tY4cNCfkdlEobtQ=
    encryptionKeyID: 1

Frames

When published in a file (or stdout), the (possibly encrypted) JSON payload is packaged in Frames. This allows deliniating each emitted payload.

VersionSizeKey IDPayload
1 byte7 bytes8 bytes(Size) bytes

Example

  • Version = 1
  • Size = 0x00033e = 830
  • Key ID = 0x0000002a = 42
hexdump -C example_events example.json
00000000  01 00 03 3e 00 00 00 2a  4e 7a 7e ea 06 0d 4a 37  |...>...*Nz~...J7|
00000010  81 b4 fa 37 f6 54 23 cb  39 1a 9b d7 5a c3 9d b6  |...7.T#.9...Z...|
00000020  b6 bd d2 c3 fa ff 07 b2  2a 0f 5b 92 2f bb 58 2d  |........*.[./.X-|

{
  "eventId": "01fad8f2-eb80-11e9-9e69-0a580a82033a",
  "eventChain": [
    "01fad8f2-eb80-11e9-9e69-0a580a82033a"
  ],
  "schemaVersion": "1.0",
  "originatorToken": null,
  "eventType": "example-service",
  "timestamp": 1570727063,
  "systemIp": "10.130.3.58",
  "action": "GET",
  "payload": {
    "isSuccessful": true,
    "request": {
      "endpoint": "/user/me",
      "headers": {
        ":authority": "localhost:8443",
        ":method": "GET",
        ":path": "/user/me",
        "content-length": "0",
        "user-agent": "Wget",
        "x-envoy-original-path": "/services/example-service/1.0/user/me",
        "x-forwarded-proto": "http",
        "x-request-id": "e63a82a7-e3b8-4ead-a8c5-af45d5a65482"
      }
    },
    "response": {
      "code": 200,
      "headers": {
        ":status": "200",
        "content-length": "4",
        "content-type": "text/plain; charset=utf-8",
        "date": "Thu, 10 Oct 2019 17:04:23 GMT",
        "x-envoy-upstream-service-time": "1"
      },
      "body": "{user: sample-user}\n"
    }
  },
  "event_mapping": {
    "type": "EventAccess",
    "action": "ACCESS"
  },
  "time_audited": "20191010T170423.317408"
}

Per-Route configuration 

{
  "emitFullResponse": <bool>
}