Catalog

Server Setup for Grey Matter Catalog

Grey Matter Catalog provides additional metadata about services to the Grey Matter Dashboard. It connects to Grey Matter Control) to retrieve discovered services and combines them with user driven configuration to create summarized metadata about services running within a service mesh. Although it was designed to power the Grey Matter Intelligence 360 Application, this service can be leveraged as an API to develop further visualizations of a service mesh.

See our guide on how to use Grey Matter Catalog for how to interact with this service when it's up and running.

Server Configuration Options

Environment Variables

Name

Type

Default

Description

HOST

String

"0.0.0.0"

Hostname or IP address.

PORT

Number

8080

TCP port to listen on.

AUTHORIZED_USERS

String

""

List of PKI Distinguished Names (DNs) approved to access metrics views in the dashboard. An empty string permits all access.

INSTANCE_POLLING_INTERVAL

String

"5s"

The interval at which health checks are made to service instances.

INSTANCE_MAX_SILENCE

String

"15s"

The wait time before a non-responsive service instance is considered down.

METRICS_MAX_RETRIES

Number

3

The maximum number of retries made before stopping requests to a service instance's metrics endpoint.

METRICS_RETRY_DELAY

String

"10s"

The wait period between attempts to connect to a service instance's metrics endpoint.

METRICS_REQUEST_TIMEOUT

String

"15s"

The wait time to indicate a request timeout from a service instance's metrics endpoint.

Control

The Catalog service discovers service instances from Grey Matter Control and up to ten instances of Control may be configured. Each environment variable for a specific control instance should have its own index number by replacing <N>. For example, CONTROL_SERVER_0_ADDRESS, CONTROL_SERVER_1_ADDRESS, etc.

Name

Type

Default

Description

CONTROL_SERVER_<N>_ADDRESS

String

"localhost:50000"

Address for a number of Control servers.

CONTROL_SERVER_<N>_ZONE_NAME

String

"region-1"

Federated zone name used for tracking, retrieving, and updating a mesh in the Catalog. Each mesh represents a collection of service clusters.

CONTROL_SERVER_<N>_REQUEST_CLUSTER_NAME

String

"edge"

The cluster used in the request made to Control to retrieve discovered service instances (i.e. the edge node proxy).

CONTROL_SERVER_RETRY_DELAY

String

"5s"

For each Control server, a "retry delay" can be specified. If an error occurs during creation of a client connection, establishing a stream, or sending a DiscoveryRequest, Catalog will restart the connection process. This occurs indefinitely and can only be stopped by sending a DELETE request to Control's /zones endpoint with the configured server's specified zoneName.

TLS

In Grey Matter service mesh deployments, Transport Layer Security (TLS) is typically handled by a Grey Matter Sidecar sidecar in front of the Catalog Service. In other words, the Catalog Service is treated like any other service in the service mesh, whereby security is offloaded to the sidecar proxy. However, this service can run standalone, and in these cases the following environment variables can be set to configure TLS.

Name

Type

Default

Description

USE_TLS

Boolean

false

Enables TLS

CA_CERT_PATH

String

""

Path to the certificate authority file.

SERVER_CERT_PATH

String

""

Path to the server certificate.

SERVER_KEY_PATH

String

""

Path to the server certificate key.

Persistence

Name

Type

Default

Description

CONFIG_SOURCE

String

"env"

Configures where to store and read service configuration from. Set value to gmdata to store in Grey Matter Data. Set value to env to use environment variables.

CONFIG_POLLING_INTERVAL

String

"3s"

Interval for checking the store for configuration changes. Only applicable when CONFIG_SOURCE=gmdata.

As indicated above, there are two options for configuring services known by the Catalog service - gmdata and env. Think about your deployment model before choosing either of these options, which are highlighted below.

Configure Persistence Grey Matter Data

When the Grey Matter service mesh is deployed via our Helm charts, Catalog will already be configured to persist service configuration in an internal instance of Grey Matter Data. This is our recommended approach for production deployments because of the added security and change history provided by Grey Matter Data. Changes to services stored in Grey Matter Data are made possible via the Catalog RESTful API.

Below is a list of environment variables required to use Grey Matter Data as the persistent store. Note, that these are automatically set when deploying with our Helm charts.

Name

Type

Default

Description

CLIENT_ADDRESS

String

""

Hostname or IP address of Grey Matter Data.

CLIENT_PORT

String

""

TCP port exposed by Grey Matter Data.

CLIENT_PREFIX

String

""

Grey Matter Data RESTful API prefix.

CLIENT_IDENTITY

String

""

Distinguished Name (DN) of JWT user associated with Catalog, see Helm Charts under internal-jwt-users for details.

CLIENT_EMAIL

String

""

Email address used as the directory name where services JSON will be persisted.

CLIENT_USE_TLS

Boolean

false

Enable TLS.

CLIENT_CERT

String

""

base64 encoded certificate for TLS.

CLIENT_KEY

String

""

base64 encoded key for TLS.

CLIENT_TRUST

String

""

base64 encoded certificate for TLS.

INSECURE_TLS

Boolean

false

MONGO_USE_TLS

Boolean

false

GMDATA_STARTUP_DELAY

String

"5s"

Delay at startup to allow all instances to write to Grey Matter Data.

GMDATA_ROOT_EVENT_NAME

String

"world"

Name of the top level folder in Grey Matter Data where services.json will be persisted.

GMDATA_MAX_EVENTS

Number

1000

GMDATA_POLLING_INTERVAL

String

"2s"

The interval at which Catalog will poll Grey Matter Data for updated services.json.

GMDATA_MAX_RETRIES

Number

10

GMDATA_RETRY_DELAY

String

"1s"

The delay for retry attempts for the initial Grey Matter Data connection.

GMDATA_OBJECT_POLICY_TEMPLATE

String

(if (contains email %s)(yield-all)(yield R X)))

Grey Matter Data object policy template to enforce permissions on servics.json folder. Default policy allows read and execute for the Catalog service.

GMDATA_POLICY_LABEL

String

"localuser owned"

GMDATA_TOP_LEVEL_FOLDER_SECURITY

String

{"label":"PUBLIC//EVERYBODY","foreground":"#FFFFFF","background":"#008800"}

GMDATA_SERVICES_FILE_SECURITY

String

{"label":"PUBLIC//EVERYBODY","foreground":"#FFFFFF","background":"#008800"

Configure Persistence with Environment Variables

The second configuration option is to statically define the metadata associated with each service via environment variables. This is a quick way to get Catalog running without having to setup Grey Matter Data but there are some pitfalls with this approach.

Pitfalls of Configuring Persistence with Environment Variables

If you switch to using Grey Matter Data after using environment variables then the metadata will be stored in Grey Matter Data and subsequent changes to the environment variables won't take effect. You can end up creating a disconnect between configurations if you switch back and forth.

If you use the Catalog RESTful API to make configuration changes and then switch back to using environment variables, you will need to keep environment variables in sync manually with those changes.

The following environment variables define the additional metadata associated with each service known by Catalog. Each environment variable for a specific service instance should have its own index number by replacing <N>. For example, SERVICE_0_CLUSTER_NAME, SERVICE_1_CLUSTER_NAME, etc.

Name

Type

Default

Description

SERVICE_<N>_CLUSTER_NAME

String

""

The name of the cluster (logical group name of service instances), provided by gm-control.

SERVICE_<N>_ZONE_NAME

String

""

The name of the zone associated with the cluster, provided by gm-control. This must match the value of CONTROL_SERVER_<N>_ZONE_NAME defined in Control's environment variables.

SERVICE_<N>_NAME

String

""

The name of the service to be displayed in the Intelligence 360 Dashboard (make this as human friendly as possible).

SERVICE_<N>_VERSION

String

""

The semver version of the service.

SERVICE_<N>_OWNER

String

""

The name of the service owner like "Decipher" or "Cool Customer". Used to sort by Owner in the Intelligence 360 Dashboard.

SERVICE_<N>_CAPABILITY

String

""

The name of the service capability like "Storage" or "Security". Used to sort by Capability in the Intelligence 360 Dashboard.

SERVICE_<N>_DOCUMENTATION

String

""

A URL path to documentation relative to the root URL of the service or a full path depending on your needs.

SERVICE_<N>_PROMETHEUSJOB

String

""

The name of the Prometheus Job used to store and query time series data associated with this service. This must match the cluster name.

SERVICE_<N>_MIN_INSTANCES

Number

1

The minimum number of instances this service should scale to. If below this number, the service will be in a warning state in the Intelligence 360 Dashboard.

SERVICE_<N>_MAX_INSTANCES

Number

1

The maximum number of instances this service should scale to. If above this number, the service will be in a warning state in the Intelligence 360 Dashboard.

SERVICE_<N>_ENABLE_INSTANCE_METRICS

Boolean

true

Enable the instance metrics view in the Intelligence 360 Dashboard.

SERVICE_<N>_ENABLE_HISTORICAL_METRICS

Boolean

false

Enable the historical metrics view in the Intelligence 360 Dashboard.

SERVICE_<N>_METRICS_TEMPLATE

String

""

URL template for constructing the service's instance metrics endpoint e.g. http://{{host}}:{{port}}/metrics.

SERVICE_<N>_METRICS_PORT

Number

8081

TCP port serving up service instance metrics.

Configuration File

Note: the following is not a typical deployment model.

If you're running the Catalog executable binary manually you can also provide a configuration file with key/value pairs.**

./gm-catalog --config=settings.toml

If there are conflicting configurations between this file and environment variables, then the environment variables will take precedence.

Questions

Need help configuring Grey Matter Catalog? Contact us at https://support.greymatter.io/support/home.