Circuit Breakers

Search…
Circuit Breakers
Circuit breaking options allow the enforcement of limits on the number of connections and requests that a cluster of upstream hosts can receive. Circuit breakers allow a quick and early detection of failures and prevents wasting resources waiting for timed out requests or retrying requests endlessly on a failing system. Setting these limits can protect clusters against sudden surges of traffic. Grey Matter supports these limits through Envoy's circuit breaking options set on the cluster object.
Configuration Options
The circuit breaking configuration options are:
Circuit breaker
Description
Type
Default
max_connections
Maximum number of connections that will be established to all instances in a cluster
int
512
max_pending_requests
Maximum number of pending requests that Envoy will allow to the upstream cluster
int
512
max_requests
Maximum number of parallel requests that Envoy will make to the upstream cluster
int
512
max_retries
Maximum number of parallel retries that Envoy will allow to the upstream cluster
int
2
max_connection_pools
Maximum number of connection pools per cluster that Envoy will concurrently support at once
int
512
track_remaining
If true, publishes stats that expose the number of resources remaining until the circuit breakers open
bool
false
Circuit breakers are always on for all clusters, and use the default values shown above when not configured explicitly. Circuit breakers cannot be entirely turned off, but can be effectively disabled by setting extremely large thresholds. E.g.
1
  "circuit_breakers": {
2
    "max_connections": 1000000000,
3
    "max_pending_requests": 1000000000,
4
    "max_retries": 1000000000,
5
    "max_requests": 1000000000,
6
  }
Copied!
Grey Matter Configuration
There is an option to configure up to two sets of circuit breaking thresholds on each cluster. The first threshold is the default threshold, which will automatically be configured for normal traffic to the cluster. The second is the option to configure a set of thresholds for routes with "high" routing priority.
Default
To configure a single set of circuit breaker options for the cluster, the circuit_breakers object set on the cluster should look like the below:
1
{
2
  "circuit_breakers": {
3
    "max_connections": 1,
4
    "max_pending_requests": 1,
5
    "max_retries": 1,
6
    "max_requests": 1,
7
    "max_connection_pools": 1,
8
    "track_remaining": true,
9
    "high": null
10
  }
11
}
Copied!
High priority
Grey Matter also provides the option to tailor circuit breakers for high priority traffic. To enable this feature, the high field in the circuit_breakers object must be configured. If left at the default ("high": null) all traffic will use the default circuit breaker. A properly configured circuit breaker for high priority traffic will look something like the following:
1
{
2
  "circuit_breakers": {
3
    "max_connections": 1,
4
    "max_pending_requests": 1,
5
    "max_retries": 1,
6
    "max_requests": 1,
7
    "max_connection_pools": 1,
8
    "track_remaining": true,
9
    "high": {
10
      "max_connections": 5,
11
      "max_pending_requests": 5,
12
      "max_retries": 5,
13
      "max_requests": 5,
14
      "max_connection_pools": 1,
15
      "track_remaining": true
16
    }
17
  }
18
}
Copied!
With this configuration of a set of high priority circuit breakers, two different circuit breaking thresholds will be created. The first will apply to traffic with default priority, and the second will only apply to priority: HIGH traffic. To specify which routes to the cluster should be using the default threshold or the high priority threshold, specify "high_priority": true on the Grey Matter Route object.
Resilience
Health Checks
Last modified 1yr ago
Export as PDF
Copy link
Contents
Configuration Options
Grey Matter Configuration
Default
High priority