greymatter.io Documentation | Configure Circuit Breakers

Circuit breaking options allow the enforcement of limits on the number of connections and requests that a cluster of upstream hosts can receive. Circuit breakers allow a quick and early detection of failures and prevents wasting resources waiting for timed out requests or retrying requests endlessly on a failing system. Setting these limits can protect clusters against sudden surges of traffic. greymatter.io supports these limits through Envoy’s circuit breaking options set on the cluster object.

Configuration Options

The circuit breaking configuration options are:

Circuit Breaker	Type	Default	Description
`max_connections`	`int`	`512`	Maximum number of connections that will be established to all instances in a cluster.
`max_pending_requests`	`int`	`512`	Maximum number of pending requests that Envoy will allow to the upstream cluster.
`max_requests`	`int`	`512`	Maximum number of parallel requests that Envoy will make to the upstream cluster.
`max_retries`	`int`	`2`	Maximum number of parallel retries that Envoy will allow to the upstream cluster.
`max_connection_pools`	`int`	`512`	Maximum number of connection pools per cluster that Envoy will concurrently support at once.
`track_remaining`	`bool`	`false`	If true, publishes stats that expose the number of resources remaining until the circuit breakers open.

Circuit breakers are always on for all clusters, and use the default values shown above when not configured explicitly. Circuit breakers cannot be entirely turned off, but can be effectively disabled by setting extremely large thresholds. E.g.

  "circuit_breakers": {
    "max_connections": 1000000000,
    "max_pending_requests": 1000000000,
    "max_retries": 1000000000,
    "max_requests": 1000000000,
  }

greymatter.io Configuration

There is an option to configure up to two sets of circuit breaking thresholds on each cluster. The first threshold is the default threshold, which will automatically be configured for normal traffic to the cluster. The second is the option to configure a set of thresholds for routes with “high” routing priority.

Default

To configure a single set of circuit breaker options for the cluster, the circuit_breakers object set on the cluster should look like the below:

{
  "circuit_breakers": {
    "max_connections": 1,
    "max_pending_requests": 1,
    "max_retries": 1,
    "max_requests": 1,
    "max_connection_pools": 1,
    "track_remaining": true,
    "high": null
  }
}

High priority

greymatter.io also provides the option to tailor circuit breakers for high priority traffic. To enable this feature, the high field in the circuit_breakers object must be configured. If left at the default ("high": null) all traffic will use the default circuit breaker. A properly configured circuit breaker for high priority traffic will look something like the following:

{
  "circuit_breakers": {
    "max_connections": 1,
    "max_pending_requests": 1,
    "max_retries": 1,
    "max_requests": 1,
    "max_connection_pools": 1,
    "track_remaining": true,
    "high": {
      "max_connections": 5,
      "max_pending_requests": 5,
      "max_retries": 5,
      "max_requests": 5,
      "max_connection_pools": 1,
      "track_remaining": true
    }
  }
}

With this configuration of a set of high priority circuit breakers, two different circuit breaking thresholds will be created. The first will apply to traffic with default priority, and the second will only apply to priority: HIGH traffic. To specify which routes to the cluster should be using the default threshold or the high priority threshold, specify "high_priority": true on the greymatter.io Route object.