Observables

The Observables Filter configures the Proxy to emit a JSON payload with every request made to the microservice. This JSON payload contains a variety of different information about the request being made, as well as the user/system issuing the request. These Observables can then be aggregated to perform analysis like: audits, user-experience tracking, etc.
Observable publishing defaults to stdout but can also be published to a Kafka topic or location on disk.
Filter Configuration Options
Name
Type
Default
Description
logLevel
String
Info
Verbosity of logging: Warn, Info, Debug
fileName
String
blank
Name of file to write events to. (stdout if blank)
emitFullResponse
Boolean
false
*Show response body in the observable object
topic
String
""
Sets the eventType field of the observable; used to sort and group messages by service/region/environment/etc in later analysis.
useKafka
Boolean
false
Publish observable message to a Kafka topic
enforceAudit
Boolean
false
Block requests until an observable has been successfully published to Kafka (Only applies if useKafke=true)
encryptionAlgorithm
String
""
Type of encryption. Must be 'aes' or blank
encryptionKey
String
""
Must be blank or base 64 encoded string of 16, 24, or 32 bytes. We recommend 32.
encryptionKeyID
uint32
0
User supplied number to identify the key used in encryption
eventTopic
String
""
The Kafka topic that will hold the published observable messages
kafkaZKDiscover
Boolean
false
Kafka will be discovered through a zookeeper node
kafkaServerConnection
String
""
Comma delimited list of Kafka addresses, or if kafkaZKDiscover is true, a list of ZooKeeper addresses
useKafkaTLS
Boolean
false
Enable TLS communication to the supplied kafka brokers
kafkaCAs
String
""
List of file URLs that point to trusts to be used when connecting to kafka
kafkaCertificate
String
""
File URL pointing to certificate to use when connecting to kafka over TLS
kafkaCertificateKey
String
""
File URL pointing to certificate key to use when connecting to kafka
kafkaServerName
String
""
Certificate server name to use when connecting to kafka
Payloads
*Note on Full Response Payloads
Turning on full response payloads (emitFullResponse=true) can cause significant amounts of data to be written to the observable.
The observables filter will attempt to write the entire response, whether it's a:
Large data file
Full-frame JPEG
Even a multi gigabyte MP4 video
This can use significant storage resources. Make sure your provisioned infrastructure can support this filter.
Key ID and Dynamic Key Rollover
Users can roll over the encryption key dynamically by changing the Observables configuration in the Proxy.
To enable convenient decryption, each key should be assigned a unique key ID.
Example Configuration
http_filters:
- name: gm.observables
  config:
    emitFullResponse: true
    useKafka: false
    enforceAudit: false
    logLevel: debug
    encryptionAlgorithm: aes
    encryptionKey: kvTujluRwliCWBWQvvvIxQr2Fxw3tY4cNCfkdlEobtQ=
    encryptionKeyID: 1
Frames
About Frames
When published in a file (or stdout), the (possibly encrypted) JSON payload is packaged in Frames.
Version
Size
Key ID
Payload
1 byte
7 bytes
8 bytes
(Size) bytes
Example
Version = 1
Size = 0x00033e = 830
Key ID = 0x0000002a = 42
hexdump -C example_events example.json
00000000  01 00 03 3e 00 00 00 2a  4e 7a 7e ea 06 0d 4a 37  |...>...*Nz~...J7|
00000010  81 b4 fa 37 f6 54 23 cb  39 1a 9b d7 5a c3 9d b6  |...7.T#.9...Z...|
00000020  b6 bd d2 c3 fa ff 07 b2  2a 0f 5b 92 2f bb 58 2d  |........*.[./.X-|
Kafka Headers
We normally write the key id to Kafka Record Headers. Such headers are only available after Kafka Version 0.11.
Have an older version of Kafka? Avoid errors by using a key id of zero. This means you cannot roll over keys dynamically.
Example Payload
{
  "eventId": "01fad8f2-eb80-11e9-9e69-0a580a82033a",
  "eventChain": [
    "01fad8f2-eb80-11e9-9e69-0a580a82033a"
  ],
  "schemaVersion": "1.0",
  "originatorToken": null,
  "eventType": "example-service",
  "timestamp": 1570727063,
  "systemIp": "10.130.3.58",
  "action": "GET",
  "payload": {
    "isSuccessful": true,
    "request": {
      "endpoint": "/user/me",
      "headers": {
        ":authority": "localhost:8443",
        ":method": "GET",
        ":path": "/user/me",
        "content-length": "0",
        "user-agent": "Wget",
        "x-envoy-original-path": "/services/example-service/1.0/user/me",
        "x-forwarded-proto": "http",
        "x-request-id": "e63a82a7-e3b8-4ead-a8c5-af45d5a65482"
      }
    },
    "response": {
      "code": 200,
      "headers": {
        ":status": "200",
        "content-length": "4",
        "content-type": "text/plain; charset=utf-8",
        "date": "Thu, 10 Oct 2019 17:04:23 GMT",
        "x-envoy-upstream-service-time": "1"
      },
      "body": "{user: sample-user}\n"
    }
  },
  "event_mapping": {
    "type": "EventAccess",
    "action": "ACCESS"
  },
  "time_audited": "20191010T170423.317408"
}
Per-Route configuration
{
  "emitFullResponse": <bool>
}
See Routing.
Questions
Need help configuring your observables filter? Contact our team at: Grey Matter Support.

OAuth

OIDC Authentication

Last updated 1 month ago

Name	Type	Default	Description
`logLevel`	String	Info	Verbosity of logging: Warn, Info, Debug
`fileName`	String	blank	Name of file to write events to. (stdout if blank)
`emitFullResponse`	Boolean	false	*Show response body in the observable object
`topic`	String	""	Sets the `eventType` field of the observable; used to sort and group messages by service/region/environment/etc in later analysis.
`useKafka`	Boolean	false	Publish observable message to a Kafka topic
`enforceAudit`	Boolean	false	Block requests until an observable has been successfully published to Kafka (Only applies if `useKafke=true`)
`encryptionAlgorithm`	String	""	Type of encryption. Must be 'aes' or blank
`encryptionKey`	String	""	Must be blank or base 64 encoded string of 16, 24, or 32 bytes. We recommend 32.
`encryptionKeyID`	uint32	0	User supplied number to identify the key used in encryption
`eventTopic`	String	""	The Kafka topic that will hold the published observable messages
`kafkaZKDiscover`	Boolean	false	Kafka will be discovered through a zookeeper node
`kafkaServerConnection`	String	""	Comma delimited list of Kafka addresses, or if `kafkaZKDiscover` is `true`, a list of ZooKeeper addresses
`useKafkaTLS`	Boolean	false	Enable TLS communication to the supplied kafka brokers
`kafkaCAs`	String	""	List of file URLs that point to trusts to be used when connecting to kafka
`kafkaCertificate`	String	""	File URL pointing to certificate to use when connecting to kafka over TLS
`kafkaCertificateKey`	String	""	File URL pointing to certificate key to use when connecting to kafka
`kafkaServerName`	String	""	Certificate server name to use when connecting to kafka