2.7. Configuring Secure Communications with etcd #

This section describes how to configure secure communications between the etcd store and Shardman services and tools.

etcd is a critical component for a Shardman cluster. If an intruder gets access to the etcd store, it gains full control over the whole cluster, including DBMS access with DBA privileges. To protect your cluster, it is recommended that you configure TLS authentication between etcd daemons and Shardman services.

To this end, you can use HTTPS transport with certificates signed by your local certification authority (CA) to encrypt traffic between the etcd cluster and Shardman services and restrict etcd access. To do this, perform the steps described in the next sections.

2.7.1. Generating SSL Certificates #

To generate SSL certificates, perform the following steps:

  1. If the CA does not exist, generate a self-signed root certificate. Generate all certificates on one trusted host. Here certificates that expire in 10000 days are generated (you can choose a more suitable interval):

    # openssl genrsa -out rootCA.key 4096
    # openssl req -x509 -new -key rootCA.key -days 10000 -out rootCA.crt 
    

  2. Prepare the following openssl configuration file for each etcd host:

    [ req ]
    default_bits       = 4096
    distinguished_name = req_distinguished_name
    req_extensions     = req_ext
    [ req_distinguished_name ]
    countryName                 = Country Name (2 letter code)
    stateOrProvinceName         = State or Province Name (full name)
    localityName               = Locality Name (eg, city)
    organizationName           = Organization Name (eg, company)
    commonName                 = Common Name (e.g. server FQDN or YOUR name)
    [ req_ext ]
    subjectAltName = @alt_names
    [alt_names]
    DNS.1   = n1
    IP.1   = 192.168.1.1
    IP.2   = 127.0.0.1
    

    Under [alt_names], specify alternative subject names for the etcd host. These names must include the etcd server hostname, IP address and local IP. Including the local IP is convenient rather than required.

    Save the file. For example, the names of configuration files for nodes n1n3 can be n1.san.confn3.san.conf.

  3. Using the configuration files prepared, generate private keys and certificate requests for etcd hosts:

    # openssl genrsa -out n1.etcd.key 4096
    # openssl req -config n1.san.conf -new -key n1.etcd.key -out n1.etcd.csr -subj "/C=RU/ST=Moscow Region/L=Moscow/O=Test/CN=n1"
    

    Here /C=RU/ST=Moscow Region/L=Moscow/O=Test/CN=n1 means that the certificate request is generated with the country name RU, state Moscow Region, locality Moscow, organization Test and common name n1. The common name must match the DNS name of your etcd server.

  4. Sign the certification request:

    # openssl x509 -extfile n1.san.conf -extensions req_ext -req -in n1.etcd.csr  -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out n1.etcd.crt -days 10000
    

  5. Check the certificates to ensure they contain correct X509v3 Subject Alternative Name fields. The fields must contain the list of DNS names and IP addresses that you added to the openssl configuration file:

    # openssl x509 -in n1.etcd.crt -noout -text
    

  6. Generate client certificates for Shardman services and client tools. These certificates do not need to contain the subjectAltName header, and CN is not important in the example below. It generates one common certificate-key pair for services and one — for tools:

    # openssl x509 genrsa -out shardman_services.key 4096
    # openssl req -new -key shardman_services.key -out shardman_services.csr -subj "/C=RU/ST=Moscow Region/L=Moscow/O=Test/CN=shardman_services"
    # openssl x509 -req -in shardman_services.csr  -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out shardman_services.crt -days 10000
    # openssl x509 genrsa -out shardman_tools.key 4096
    # openssl req -new -key shardman_tools.key -out shardman_tools.csr -subj "/C=RU/ST=Moscow Region/L=Moscow/O=Test/CN=shardman_tools"
    # openssl x509 -req -in shardman_tools.csr  -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out shardman_tools.crt -days 10000
    

2.7.2. Configuring etcd and shardmand Services #

Now configure services ( etcd and shardmand ) to use the generated certificates. To do this, perform the following steps:

  1. On each etcd node, put rootCA.crt, nX.etcd.crt and nX.etcd.key in the location accessible to the etcd daemon (for example, create /etc/etcd directory and put files there). Ensure that the nX.etcd.key file is only accessible to the etcd daemon user.

  2. Specify the following configuration for etcd daemons in /etc/default/etcd:

    # unqualified first name
    ETCD_NAME=n1
    # where we actually listen for peers
    ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
    # where we actually listen for clients
    ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
    # advertise where this machine is listening for clients
    ETCD_ADVERTISE_CLIENT_URLS=https://n1:2379
    
    # --initial flags are used during bootstrapping and ignored afterwards, so it is
    # ok to specify them always
    # advertise where this machine is listening for peer
    ETCD_INITIAL_ADVERTISE_PEER_URLS=https://n1:2380
    ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
    # ansible_nodename is fqdn
    ETCD_INITIAL_CLUSTER=n1=https://n1:2380,n2=https://n2:2380,n3=https://n3:2380
    ETCD_INITIAL_CLUSTER_STATE=new
    
    ETCD_DATA_DIR=/var/lib/etcd/default/member
    ETCD_AUTO_COMPACTION_RETENTION=1
    
    ETCD_KEY_FILE=/etc/etcd/n1.etcd.key
    ETCD_CERT_FILE=/etc/etcd/n1.etcd.crt
    ETCD_TRUSTED_CA_FILE=/etc/etcd/rootCA.crt
    ETCD_CLIENT_CERT_AUTH=true
    
    ETCD_PEER_CERT_FILE=/etc/etcd/n1.etcd.crt
    ETCD_PEER_KEY_FILE=/etc/etcd/n1.etcd.key
    ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/rootCA.crt
    ETCD_PEER_CLIENT_CERT_AUTH=true
    

    Replace n1 here with the appropriate node name.

  3. Restart etcd services on all etcd cluster nodes:

    # systemctl restart etcd
    

  4. To check the new configuration, use the following command:

    # etcdctl  --endpoints=https://n1:2379,https://n2:2379,https://n3:2379 --cacert /etc/etcd/rootCA.crt --cert /etc/etcd/n1.etcd.crt --key /etc/etcd/n1.etcd.key member list -w table
    
    +------------------+---------+------+-----------------+-----------------+------------+
    |        ID        | STATUS  | NAME |    PEER ADDRS   |   CLIENT ADDRS  | IS LEARNER |
    +------------------+---------+------+-----------------+-----------------+------------+
    | 66ebe06d7302c3f0 | started |   n2 | https://n2:2380 | https://n2:2379 |      false |
    | b1080bf5ff059980 | started |   n1 | https://n1:2380 | https://n1:2379 |      false |
    | d98323257249fefb | started |   n3 | https://n3:2380 | https://n3:2379 |      false |
    +------------------+---------+------+-----------------+-----------------+------------+
    
    

  5. On each Shardman cluster node, put rootCA.crt, shardman_services.crt and shardman_services.key in a location accessible to the postgres user (for example, create the /etc/shardman directory and put files there). Ensure that the shardman_services.key file is only accessible to the postgres user.

  6. Edit the shardmand configuration file /etc/shardman/shardmand-cluster0.env as follows:

    SDM_STORE_ENDPOINTS=https://n1:2379,https://n2:2379,https://n3:2379
    SDM_STORE_CA_FILE=/etc/shardman/rootCA.crt
    SDM_STORE_CERT_FILE=/etc/shardman/shardman_services.crt
    SDM_STORE_KEY=/etc/shardman/shardman_services.key
    

  7. Restart shardmand@cluster0 services on all Shardman nodes:

    # systemctl restart shardmand@cluster0
    

2.7.3. Using Shardman Tools #

Before using Shardman tools, copy rootCA.crt, shardman_tools.crt and shardman_tools.key to some location on the Shardman management node where they are accessible to the management user. Here, any node with installed Shardman utilities that is used to manage the Shardman cluster is meant by management node. This can also be one of the Shardman cluster nodes (or all of them). By management user, a user is meant who runs shardmanctl tool. It is assumed that the certificates and key are located in the /etc/shardman directory.

When using Shardman tools, be sure to add --store-ca-file, --store-cert-file and --store-key options to shardmanctl command. For example, the following command gets the cluster status:

# shardmanctl --store-ca-file /etc/shardman/rootCA.crt --store-cert-file /etc/shardman/shardman_tools.crt --store-key /etc/shardman/shardman_tools.key --store-endpoints https://n1:2379,https://n2:2379,https://n3:2379 status

pdf