Fault Injection

Grey Matter Sidecar's can be configured with fault injection in order to test resiliency and failures. For reference on the fault injection filter, see the usage docs.

Note: This guide will use the fibonacci service deployment from the deploy a service guide as an example. If you have a different existing service in the mesh that you wish to try fault injection on, be sure that it is properly configured and has an endpoint that can be reached, and replace the listener object modifications in step 1 with the corresponding listener for your service.

Prerequisites

  1. An existing Grey Matter deployment running on Kubernetes (tutorial)

  2. An existing instance of the fibonacci service from the deploy a service guide

  3. kubectl or oc setup with access to the cluster

  4. greymatter cli setup with access to the deployment

Steps

1. Enable the Fault Injection Filter

The example fault injection configuration we will use is:

{
"abort": {
"percentage": {
"numerator": 3,
"denominator": "HUNDRED"
},
"http_status": 418
}
}

For this example, we will configure the sidecar to abort on 30% of requests, and the aborts will send back an http status code 418.

To apply this filter, run:

greymatter edit listener fibonacci-listener
  1. In the active_http_filters field and add "envoy.fault" to the list.

  2. In the http_filters field, add the configuration above for "envoy_fault".

The listener object should look like:

...
"active_http_filters": [
"envoy.fault"
],
...
"http_filters": {
"envoy_fault": {
"abort": {
"percentage": {
"numerator": 30,
"denominator": "HUNDRED"
},
"http_status": 418
}
}
},
...

Save this. Check the logs for the fibonacci sidecar and verify that there are no configuration failures.

2. Test

To test the implementation of this filter, try making repeated request to the endpoint for your service in a browser. For the fibonacci example, this is https://{edge-dns}/services/fibonacci/.

The 200 response to this endpoint is Alive. Continue to refresh this endpoint, you will periodically see:

fault filter abort

Indicating fault injection!

For a more precise test of what's happening, save and run the following script from the same terminal where you have the Grey Matter CLI configured with environment variables:

#!/bin/sh
echo "Sending requests to the fibonacci service, using Grey Matter variables from env"
path="/services/fibonacci/"
x=0; count=0; count2=0
while [ $x -lt 100 ]
do
resp=$(curl --write-out '%{http_code}' --output /dev/null --silent https://$GREYMATTER_API_HOST$path -k --cert $GREYMATTER_API_SSLCERT --key $GREYMATTER_API_SSLKEY)
if [ $resp == "418" ]
then
count=$[$count+1]
elif [ $resp == "200" ]
then
count2=$[$count2+1]
fi
x=$[$x+1]
done
echo "Sent 100 requests to https://$GREYMATTER_API_HOST$path"
echo "Number of 200 responses: $count2"
echo "Number of 418 responses: $count"
echo "The percentage of fault injections: $count%"

The output should be very close to:

Sent 100 requests to https://localhost:30000/services/fibonacci/
Number of 200 responses: 70
Number of 418 responses: 30
The percentage of fault injections: 30%

If you set the fault injection filter on a different service, change the path variable in the script to your services path.

For more information on fault injection and configuration options, see the usage docs.