sync
greymatter sync and its child commands allow the CLI users to setup "GitOps" i.e. all the mesh object definitions are stored in a version controlled storage such as GitHub. By running greymatter sync apply, CLI applies the stored configurations to destination if:
  1. 1.
    Resources' definitions in the cache storage differ from the last time greymatter sync apply was executed
  2. 2.
    Destination objects differ from the last time greymatter sync apply was executed
You can review what actions will be taken if you run apply by running greymatter sync plan.
Configuring sync functionality is complex and not needed for regular CLI use (unless you choose to do so). If sync.source.command is empty in ~/.greymatter.yaml, CLI will assume that you are not intending to use this functionality, hence no further check for the validity of the configuration for sync command would be done. For more information about configuring sync functionality, please see the Configuration section below.

Child Commands

Command
Description
plan
Plans the synchronization of the source and destination without applying
apply
Applies the source to the destination
utils
Utility commands for sync

plan

plan allows us to review what actions will be taken when we synchronize the source and the destination. The output of greymatter sync plan looks like the following:
1
Action: NOOP, Key: domain-prometheus-egress, Kind: DOMAIN
2
Action: NOOP, Key: domain-dashboard-egress, Kind: DOMAIN
3
Action: CREATE, Key: listener-control-api-egress, Kind: LISTENER
4
Action: UPDATE, Key: cluster-jwt-security, Kind: CLUSTER
5
Action: UPDATE, Key: edge-to-catalog-cluster, Kind: CLUSTER
6
Action: DELETE, Key: edge-prometheus-route, Kind: ROUTE
Copied!
  • NOOP No operation
  • CREATE Source repository contains an object that does not exist in the destination
  • UPDATE Source object and destination object differs
  • DELETE Destination contains an object that does not exist in the source

apply

apply will actually synchronize the source and the destination mesh objects.

Utils

cipher

cipher is a utility tool to encrypt/decrypt a value which will be used like this in the mesh object configurations:
1
"{{ decrypt "AgA+jVHymttkItPqpm1ukHecZ9lnPdtdHkqPvVdar0qk3HytmB3V8ZLVpK80CQzHVW7SEwPObZXwN+GbR0UJ3AHM/agbMWsaEXPnp76K27PliGJLLV5TU/CLpk0XLAp5CE8/zz4ORmLxwkd96YNCmPDkYCuAejRtKWg0Y1w2WE4iSO2cP46Eyba5HIzHNkf1l1TaOtJoHG+3ieyWFs4QrHCDHBHU4Tke+MZEsDHfDR1Mq7Q+S8Rjq3UVEoTQngSA9kxmGNjtgEp8F1tzIRf3GgB7YxKnCY8v8y+bqHAYFTUvJzQAbEZilWyfpcnT0bdqGpbgX2JLU96LkFo1XC87w3QdiATv/KRQgQsfopj5fagdRrdFe2pBvN0+GjvgWnCxy6EYZDEbPMEgVMHjGcI20/PaEndez28JecidybaZdF1gLg2y4osQQu5wdPoJMEWCs92bw20g8360UtbrmX4UC7q6nV4ddTcmbPcrcWDHCpkPKLJQ9NQohCw6/CB9UdD9IHB3D6JYVel5QBcxJP238dG2yzbLlE04Su7xLi1spg8CwV+H91RN5QpfulIKSSrSXhDL8obWIMhaz7yJ1VO/iJvJUN9c2SX7j10wY91YvTxi2bzbjtf0ZuJzsGDnQdAXQt7ZRWfUgagZHlab+TYV5ToawTAvkLT8cyKBwS0TbrrmCpTQ0FyYMgAxrahI+fLNlTZW7EwtICk=" }}"
Copied!
The outer double quotes will surround the decrypted value to indicate it is a string
This is the simplest method to avoid storing sensitive information in the resource storage. However, since the configuration file itself has the actual encrypted value, you cannot use the same repository for multiple environments. For more flexible approach, please use secret below.
For example, using template.json files with the following in it:
1
{
2
"hello": "{{ decrypt "AgADd7RVzRL8O8qeFdTfWkBP+RV/PdW4leL6Bj+LzJKJfd8Zm8gJNcD24Lhmqeig6kj4gxjsSaxvIpkhTLL0YHKAxYWKsYAooOBhhX3BGFXsXQmNWKJrUohHZLhZgxeAXCQifzni13DJcYRt7Gz7x/AQ8HSFfXQoWNVJLAbYFEYwJPGPZ6S7jclm54fhcxolVFsOEtN6Rh7IaorXvizMYUHi2SG/kE5KWQzcBHxHTZK4Jbt30DSspoKdXGmWcEA0VB5jVDFE3LexPN0v9leeo8mKEZjxeK8UxT5GMuueS9k3nychabwpWFWVH1hVXHqfjVDjY3Z9dGu/HqhbRJcFPWNa4qtydIZX4jKUaD2jfXT7fOYRgqhq1gOYCne4N/V5lTmuVaZvPRKr3xLXNgoDokft71oBgg8x06Lt0GTA0bAms3+1SOhr8svP/pxZ4VVUNmNYWwlrIeoxbqm3C+tmiOCf/Br22XOAtZHwe4Qr7YND15ZM29U4xlSk0aUFbDgx0PqLbcn1yAHJxy3WZOt1V6wGRoVb453EgIUtfunSvDkbmuC9BazYJh6WvAjGADz9KvFnMg5bRMPK1q7zGAKgfsIY24/AdXIfUc5+yaP1Mm2N71EKeUp2vFe0UgQg+0C0cYpZY8u6B/C6otXr6BQ07x4FEnwOtbRrX5Jb8P7bv8LYOTJ0X93XAFiizqDTBSv5q8GOI0te4w==" }}"
3
}
Copied!
The output of greymatter sync utils render will look as follows:
1
$ greymatter sync utils render < template.json
2
{
3
"hello": "world"
4
}
Copied!
encrypt
encrypt will read from stdin and encrypt that value.
decrypt
decrypt will read from stdin and decrypt that value. It is also used during the rendering process.
Example:
1
$ echo hello-world | greymatter sync utils cipher encrypt > output.txt
2
$ greymatter sync utils cipher decrypt < output.txt
3
hello-world
Copied!

secret

secret manages key-value pairs used to replace {{ secret "key" }} in templates with an actual value. This is similar to variable below, but allows it to come from somewhere else with different access restrictions (i.e., Secret).
It has following child commands to manage the stored secrets:
1
clear Clear the secret store
2
delete Delete a value from the secret store
3
get Get a value from the secret store
4
list List the keys in the secret store
5
put Put a value in the secret store
Copied!

variable

variable manages key-value pairs used to replace {{ variable "key" }} in templates with an actual value.
Variable is intended to be used for environment specific things (i.e., in Kubernetes I can mount a Configmap to back the variables plugin).
1
clear Clear the variable store
2
delete Delete a value from the variable store
3
get Get a value from the variable store
4
list List the keys in the variable store
5
put Put a value in the variable store
Copied!

render

render allows us to review what the rendered templates look like by reading the template from stdin:
1
greymatter sync utils render < template.json
Copied!

Configuration

The common configuration in .greymatter.yaml looks like below:
1
sync:
2
source:
3
command:
4
- greymatter
5
- sync
6
- plugin
7
- remote
8
- git
9
configuration:
10
url: [email protected]:user/repo.git
11
reference: branch:master
12
credentials:
13
privateKey: rsa_private.pem
14
folders:
15
zone: zones
16
proxy: proxies
17
listener: listeners
18
domain: domains
19
route: routes
20
shared_rules: rules
21
cluster: clusters
22
catalog-service: catalog-services
23
catalog-mesh: catalog-meshes
24
25
destination:
26
command:
27
- greymatter
28
- sync
29
- plugin
30
- remote
31
- api
32
33
cache:
34
command:
35
- greymatter
36
- sync
37
- plugin
38
- store
39
- badger
40
configuration:
41
path: .cache/
42
43
variable:
44
command:
45
- greymatter
46
- sync
47
- plugin
48
- store
49
- badger
50
configuration:
51
path: .variable/
52
53
secret:
54
command:
55
- greymatter
56
- sync
57
- plugin
58
- store
59
- badger
60
configuration:
61
path: .secret/
62
63
cipher:
64
command:
65
- greymatter
66
- sync
67
- plugin
68
- cipher
69
- rsa
70
configuration:
71
privateKey: rsa_private.pem
72
publicKey: rsa_public.pem
Copied!
Grey Matter CLI uses hashicorp/go-plugin behind the scene, and command section in above example indicates which plugin to use. For example, if you were to simply run:
1
greymatter sync plugin store badger
Copied!
You will get an output like this:
1
This binary is a plugin. These are not meant to be executed directly.
2
Please execute the program that consumes these plugins, which will
3
load any plugins automatically
Copied!
That is why plugin commands are hidden from help, but they exist.
The configuration: section under a plugin's command in the configuration file (e.g. .greymatter.yaml) will contain necessary information for the particular plugin.

Plugins

cipher plugin

cipher plugin is used by greymatter sync utils cipher
  • rsa: this is currently the only cipher plugin Configuration:
    • privateKey: the PKCS1 private key for the cipher which can be created like so:
      1
      openssl genrsa -out rsa_private.pem 4096
      Copied!
    • publicKey: the public key for the cipher which can be created like so:
      1
      openssl rsa -in rsa_private.pem -pubout -out rsa_public.pem
      Copied!

remote

remote plugin is used by greymatter sync plan and greymatter sync apply
  • api: gm-control-api as a remote storage Configuration:
    • delete: whether resources that exist in the destination but not in the source should be deleted
  • git: GitHub as a remote storage Configuration:
    • url: the URL of the repository. The URL must be a valid URL, and not ssh shorthand (ssh://[email protected]/user/repo instead of [email protected]:user/repo)
    • reference: the branch, tag, or commit to clone, in the format branch:<branch_name>, tag:<tag>, or commit:<commit>. Leave empty for HEAD.
    • credentials: the credentials for authenticating with the repository
      • username: the username for authentication (HTTP or SSH). Leave empty to parse the username from the url.
      • password: the password for basic authentication (HTTP) or the private key (SSH).
      • privateKey: the path to the private key file (SSH). If no Private Key is passed, the ssh-agent will be used.
    • folders: a collection of folder names for folders that contain object definition files to compare with resources in the destination.

renderer

renderer plugin is used by greymatter sync utils render
  • contextual: the typical renderer Configuration: None
  • noop: a renderer that does not do any variable resolutions Configuration: None

store

store plugin is used as a key-value storage and used by greymatter sync utils variable, greymatter sync utils secret, as well as keeping track of what was applied and observed in order to determine what action to take when greymatter sync plan is executed.
  • badger: dgraph-io/badger as a key-value storage Configuration:
    • path: the path to the database directory
  • file: a file as a key-value storage Configuration:
    • path: the path to the store file
  • memory: this is used only for testing purposes. There is no persistence once the command exists. Configuration: None

Example setup for a Git source and Grey Matter Control API destination

Source

The Git repository should store JSON object definition files of a given type (YAML support will come in future) in folders that match the names specified in sync.source.configuration.folders:
1
folders:
2
zone: zones
3
proxy: proxies
4
listener: listeners
5
domain: domains
6
route: routes
7
shared_rules: rules
8
cluster: clusters
9
catalog-service: catalog-services
10
catalog-mesh: catalog-meshes
Copied!
Given a folder name for an object definition type, all folders in the Git repository that match that folder name will be searched for JSON object definition files. For example, if the domain object definition type has the folder name domain_configs, then JSON object definition files can be found in the following folders:
  • /domain_configs/
  • /zone-a/domain_configs/
  • /zone-b/domain_configs/
  • /other-configs/zone-c/service-1/domain_configs/
  • etc.
If an object definition type has no folder name configured, the default folder name specified in this example is used.
Source and destination configuration should look similar to below:
1
source:
2
command:
3
- greymatter
4
- sync
5
- plugin
6
- remote
7
- git
8
configuration:
9
url: [email protected]:user/repo.git
10
reference: branch:master
11
credentials:
12
privateKey: rsa_private.pem
13
folders:
14
zone: zones
15
proxy: proxies
16
listener: listeners
17
domain: domains
18
route: routes
19
shared_rules: rules
20
cluster: clusters
21
catalog-service: catalog-services
22
catalog-mesh: catalog-meshes
23
24
destination:
25
command:
26
- greymatter
27
- sync
28
- plugin
29
- remote
30
- api
Copied!
For more detail, please refer to remote section.

Overlap between greymatter generate and greymatter sync

greymatter generate only renders templates. greymatter sync is a GitOps tool that applies mesh configurations from a Git repository. They solve two different problems, generate is to quickly generate configurations on disk based upon templates controlled in a repository. sync is to synchronize configuration in a repository with the api as well as rendering templates. It allows the configurations to be templates and it will substitute values every time it tries to apply them. This is to:
a) prevent secrets from being checked into git un-encrypted and
b) allow for the same configurations to be applied to multiple meshes (by separate sync processes).
Last modified 1yr ago