Light Dark Auto

Circuit Breakers

Circuit breaking options allow the enforcement of limits on the number of connections and requests that a cluster of upstream hosts can receive. Circuit breakers allow a quick and early detection of failures and prevents wasting resources waiting for timed out requests or retrying requests endlessly on a failing system. Setting these limits can protect clusters against sudden surges of traffic. greymatter.io supports these limits through Envoy's circuit breaking options set on the cluster object.

Configuration Options

The circuit breaking configuration options are:

Circuit BreakerTypeDefaultDescription
max_connections

int

512

Maximum number of connections that will be established to all instances in a cluster.

max_pending_requests

int

512

Maximum number of pending requests that Envoy will allow to the upstream cluster.

max_requests

int

512

Maximum number of parallel requests that Envoy will make to the upstream cluster.

max_retries

int

2

Maximum number of parallel retries that Envoy will allow to the upstream cluster.

max_connection_pools

int

512

Maximum number of connection pools per cluster that Envoy will concurrently support at once.

track_remaining

bool

false

If true, publishes stats that expose the number of resources remaining until the circuit breakers open.

greymatter.io Configuration

There is an option to configure up to two sets of circuit breaking thresholds on each cluster. The first threshold is the default threshold, which will automatically be configured for normal traffic to the cluster. The second is the option to configure a set of thresholds for routes with "high" routing priority.

Default

To configure a single set of circuit breaker options for the cluster, the circuit_breakers object set on the cluster should look like the below:

{
  "circuit_breakers": {
    "max_connections": 1,
    "max_pending_requests": 1,
    "max_retries": 1,
    "max_requests": 1,
    "max_connection_pools": 1,
    "track_remaining": true,
    "high": null
  }
}

High priority

greymatter.io also provides the option to tailor circuit breakers for high priority traffic. To enable this feature, the high field in the circuit_breakers object must be configured. If left at the default ("high": null) all traffic will use the default circuit breaker. A properly configured circuit breaker for high priority traffic will look something like the following:

{
  "circuit_breakers": {
    "max_connections": 1,
    "max_pending_requests": 1,
    "max_retries": 1,
    "max_requests": 1,
    "max_connection_pools": 1,
    "track_remaining": true,
    "high": {
      "max_connections": 5,
      "max_pending_requests": 5,
      "max_retries": 5,
      "max_requests": 5,
      "max_connection_pools": 1,
      "track_remaining": true
    }
  }
}

With this configuration of a set of high priority circuit breakers, two different circuit breaking thresholds will be created. The first will apply to traffic with default priority, and the second will only apply to priority: HIGH traffic. To specify which routes to the cluster should be using the default threshold or the high priority threshold, specify "high_priority": true on the greymatter.io Route object.