The Observables Filter configures the Proxy to emit a JSON payload with every request made to the microservice. This JSON payload contains a variety of different information about the request being made, as well as the user/system issuing the request. These Observables can then be aggregated to perform analysis like: audits, user-experience tracking, etc.
Observable publishing defaults to stdout
but can also be published to a Kafka topic or location on disk.
Filter Configuration Options
emitFullResponse
(Boolean, default: false)
- *Show response body in the observable object.topic
(String, default: "")
- Sets the eventType field of the observable; used to sort and group messages by service/region/environment/etc in later analysis..useKafka
(Boolean, default: false)
- .Publish observable message to a Kafka topicenforceAudit
(Boolean, default: false)
- Block requests until an observable has been successfully published to Kafka (Only applies if useKafke=true).encryptionAlgorithm
(String, default: "")
- Type of encryption. Must be ‘aes’ or blank.encryptionKey
(String, default: "")
- Must be blank or base 64 encoded string of 16, 24, or 32 bytes. We recommend 32..encryptionKeyID
(uint32, default: 0)
- User supplied number to identify the key used in encryption.eventTopic
(String, default: "")
- The Kafka topic that will hold the published observable messages.kafkaZKDiscover
(Boolean, default: false)
- Kafka will be discovered through a zookeeper node.kafkaServerConnection
(String, default: "")
- Comma delimited list of Kafka addresses, or if kafkaZKDiscover is true, a list of ZooKeeper addresses.useKafkaTLS
(Boolean, default: false)
- Enable TLS communication to the supplied kafka brokers.kafkaCAs
(String, default: "")
- List of file URLs that point to trusts to be used when connecting to kafka.kafkaCertificate
(String, default: "")
- File URL pointing to certificate to use when connecting to kafka over TLS.kafkaCertificateKey
(String, default: "")
- .File URL pointing to certificate key to use when connecting to kafkakafkaServerName
(String, default: "")
- Certificate server name to use when connecting to kafka.
Payloads
Turning on full response payloads (emitFullResponse=true
) can cause significant amounts of data to be written to the payload. The observables filter will attempt to write the entire response, even if it’s a gigabyte payload, so take care.
Encryption
Payload encryption can be enabled in the filter config. Each encryption key should be assigned a unique ID.
http_filters:
- name: gm.observables
config:
emitFullResponse: true
useKafka: false
enforceAudit: false
logLevel: debug
encryptionAlgorithm: aes
encryptionKey: kvTujluRwliCWBWQvvvIxQr2Fxw3tY4cNCfkdlEobtQ=
encryptionKeyID: 1
Frames
When published in a file (or stdout
), the (possibly encrypted) JSON payload is packaged in Frames
. This allows deliniating each emitted payload.
Version | Size | Key ID | Payload |
---|---|---|---|
1 byte | 7 bytes | 8 bytes | (Size) bytes |
Example
- Version = 1
- Size = 0x00033e = 830
- Key ID = 0x0000002a = 42
hexdump -C example_events example.json
00000000 01 00 03 3e 00 00 00 2a 4e 7a 7e ea 06 0d 4a 37 |...>...*Nz~...J7|
00000010 81 b4 fa 37 f6 54 23 cb 39 1a 9b d7 5a c3 9d b6 |...7.T#.9...Z...|
00000020 b6 bd d2 c3 fa ff 07 b2 2a 0f 5b 92 2f bb 58 2d |........*.[./.X-|
{
"eventId": "01fad8f2-eb80-11e9-9e69-0a580a82033a",
"eventChain": [
"01fad8f2-eb80-11e9-9e69-0a580a82033a"
],
"schemaVersion": "1.0",
"originatorToken": null,
"eventType": "example-service",
"timestamp": 1570727063,
"systemIp": "10.130.3.58",
"action": "GET",
"payload": {
"isSuccessful": true,
"request": {
"endpoint": "/user/me",
"headers": {
":authority": "localhost:8443",
":method": "GET",
":path": "/user/me",
"content-length": "0",
"user-agent": "Wget",
"x-envoy-original-path": "/services/example-service/1.0/user/me",
"x-forwarded-proto": "http",
"x-request-id": "e63a82a7-e3b8-4ead-a8c5-af45d5a65482"
}
},
"response": {
"code": 200,
"headers": {
":status": "200",
"content-length": "4",
"content-type": "text/plain; charset=utf-8",
"date": "Thu, 10 Oct 2019 17:04:23 GMT",
"x-envoy-upstream-service-time": "1"
},
"body": "{user: sample-user}\n"
}
},
"event_mapping": {
"type": "EventAccess",
"action": "ACCESS"
},
"time_audited": "20191010T170423.317408"
}
Per-Route configuration
{
"emitFullResponse": <bool>
}