Observables

The Observables Filter configures the Proxy to emit a JSON payload with every request made to the microservice. This JSON payload contains a variety of different information about the request being made, as well as the user/system issuing the request. These Observables can then be aggregated to perform analysis like: audits, user-experience tracking, etc.

Observable publishing defaults to stdout but can also be published to a Kafka topic or location on disk.

Filter Configuration Options

Name

Type

Default

Description

logLevel

String

Info

Verbosity of logging: Warn, Info, Debug

fileName

String

blank

Name of file to write events to. (stdout if blank)

emitFullResponse

Boolean

false

*Show response body in the observable object

topic

String

""

Sets the eventType field of the observable; used to sort and group messages by service/region/environment/etc in later analysis.

useKafka

Boolean

false

Publish observable message to a Kafka topic

enforceAudit

Boolean

false

Block requests until an observable has been successfully published to Kafka (Only applies if useKafka=true)

encryptionAlgorithm

String

""

Type of encryption. Must be 'aes' or blank

encryptionKey

String

""

Must be blank or base 64 encoded string of 16, 24, or 32 bytes. We recommend 32.

encryptionKeyID

uint32

0

User supplied number to identify the key used in encryption

eventTopic

String

""

The Kafka topic that will hold the published observable messages

kafkaZKDiscover

Boolean

false

Kafka will be discovered through a zookeeper node

kafkaServerConnection

String

""

Comma delimited list of Kafka addresses, or if kafkaZKDiscover is true, a list of ZooKeeper addresses

useKafkaTLS

Boolean

false

Enable TLS communication to the supplied kafka brokers

kafkaCAs

String

""

List of file URLs that point to trusts to be used when connecting to kafka

kafkaCertificate

String

""

File URL pointing to certificate to use when connecting to kafka over TLS

kafkaCertificateKey

String

""

File URL pointing to certificate key to use when connecting to kafka

kafkaServerName

String

""

Certificate server name to use when connecting to kafka

Payloads

*Note on Full Response Payloads

Turning on full response payloads (emitFullResponse=true) can cause significant amounts of data to be written to the observable.

The observables filter will attempt to write the entire response, whether it's a:

  • Large data file

  • Full-frame JPEG

  • Even a multi gigabyte MP4 video

This can use significant storage resources. Make sure your provisioned infrastructure can support this filter.

Key ID and Dynamic Key Rollover

Users can roll over the encryption key dynamically by changing the Observables configuration in the Proxy.

To enable convenient decryption, each key should be assigned a unique key ID.

Example Configuration

http_filters:
- name: gm.observables
config:
emitFullResponse: true
useKafka: false
enforceAudit: false
logLevel: debug
encryptionAlgorithm: aes
encryptionKey: kvTujluRwliCWBWQvvvIxQr2Fxw3tY4cNCfkdlEobtQ=
encryptionKeyID: 1

Frames

About Frames

When published in a file (or stdout), the (possibly encrypted) JSON payload is packaged in Frames.

Version

Size

Key ID

Payload

1 byte

7 bytes

8 bytes

(Size) bytes

Example

  • Version = 1

  • Size = 0x00033e = 830

  • Key ID = 0x0000002a = 42

hexdump -C example_events example.json
00000000 01 00 03 3e 00 00 00 2a 4e 7a 7e ea 06 0d 4a 37 |...>...*Nz~...J7|
00000010 81 b4 fa 37 f6 54 23 cb 39 1a 9b d7 5a c3 9d b6 |...7.T#.9...Z...|
00000020 b6 bd d2 c3 fa ff 07 b2 2a 0f 5b 92 2f bb 58 2d |........*.[./.X-|

Kafka Headers

We normally write the key id to Kafka Record Headers. Such headers are only available after Kafka Version 0.11.

Have an older version of Kafka? Avoid errors by using a key id of zero. This means you cannot roll over keys dynamically.

Example Payload

{
"eventId": "01fad8f2-eb80-11e9-9e69-0a580a82033a",
"eventChain": [
"01fad8f2-eb80-11e9-9e69-0a580a82033a"
],
"schemaVersion": "1.0",
"originatorToken": null,
"eventType": "example-service",
"timestamp": 1570727063,
"systemIp": "10.130.3.58",
"action": "GET",
"payload": {
"isSuccessful": true,
"request": {
"endpoint": "/user/me",
"headers": {
":authority": "localhost:8443",
":method": "GET",
":path": "/user/me",
"content-length": "0",
"user-agent": "Wget",
"x-envoy-original-path": "/services/example-service/1.0/user/me",
"x-forwarded-proto": "http",
"x-request-id": "e63a82a7-e3b8-4ead-a8c5-af45d5a65482"
}
},
"response": {
"code": 200,
"headers": {
":status": "200",
"content-length": "4",
"content-type": "text/plain; charset=utf-8",
"date": "Thu, 10 Oct 2019 17:04:23 GMT",
"x-envoy-upstream-service-time": "1"
},
"body": "{user: sample-user}\n"
}
},
"event_mapping": {
"type": "EventAccess",
"action": "ACCESS"
},
"time_audited": "20191010T170423.317408"
}

Questions

Need help configuring your observables filter? Contact our team at: Decipher Support.