2.7. Configuring Secure Communications with etcd #
This section describes how to configure secure communications between the etcd store and Shardman services and tools.
etcd is a critical component for a Shardman cluster. If an intruder gets access to the etcd store, it gains full control over the whole cluster, including DBMS access with DBA privileges. To protect your cluster, it is recommended that you configure TLS authentication between etcd daemons and Shardman services.
To this end, you can use HTTPS transport with certificates signed by your local certification authority (CA) to encrypt traffic between the etcd cluster and Shardman services and restrict etcd access. To do this, perform the steps described in the next sections.
2.7.1. Generating SSL Certificates #
To generate SSL certificates, perform the following steps:
If the CA does not exist, generate a self-signed root certificate. Generate all certificates on one trusted host. Here certificates that expire in 10000 days are generated (you can choose a more suitable interval):
#
openssl genrsa -out rootCA.key 4096
#
openssl req -x509 -new -key rootCA.key -days 10000 -out rootCA.crt
Prepare the following openssl configuration file for each etcd host:
[ req ] default_bits = 4096 distinguished_name = req_distinguished_name req_extensions = req_ext [ req_distinguished_name ] countryName = Country Name (2 letter code) stateOrProvinceName = State or Province Name (full name) localityName = Locality Name (eg, city) organizationName = Organization Name (eg, company) commonName = Common Name (e.g. server FQDN or YOUR name) [ req_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = n1 IP.1 = 192.168.1.1 IP.2 = 127.0.0.1
Under
[alt_names]
, specify alternative subject names for the etcd host. These names must include the etcd server hostname, IP address and local IP. Including the local IP is convenient rather than required.Save the file. For example, the names of configuration files for nodes
n1
—n3
can ben1.san.conf
—n3.san.conf
.Using the configuration files prepared, generate private keys and certificate requests for etcd hosts:
#
openssl genrsa -out n1.etcd.key 4096
#
openssl req -config n1.san.conf -new -key n1.etcd.key -out n1.etcd.csr -subj "/C=RU/ST=Moscow Region/L=Moscow/O=Test/CN=n1"
Here “
/C=RU/ST=Moscow Region/L=Moscow/O=Test/CN=n1
” means that the certificate request is generated with the country nameRU
, stateMoscow Region
, localityMoscow
, organizationTest
and common namen1
. The common name must match the DNS name of your etcd server.Sign the certification request:
#
openssl x509 -extfile n1.san.conf -extensions req_ext -req -in n1.etcd.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out n1.etcd.crt -days 10000
Check the certificates to ensure they contain correct
X509v3 Subject Alternative Name
fields. The fields must contain the list of DNS names and IP addresses that you added to the openssl configuration file:#
openssl x509 -in n1.etcd.crt -noout -text
Generate client certificates for Shardman services and client tools. These certificates do not need to contain the
subjectAltName
header, andCN
is not important in the example below. It generates one common certificate-key pair for services and one — for tools:#
openssl x509 genrsa -out shardman_services.key 4096
#
openssl req -new -key shardman_services.key -out shardman_services.csr -subj "/C=RU/ST=Moscow Region/L=Moscow/O=Test/CN=shardman_services"
#
openssl x509 -req -in shardman_services.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out shardman_services.crt -days 10000
#
openssl x509 genrsa -out shardman_tools.key 4096
#
openssl req -new -key shardman_tools.key -out shardman_tools.csr -subj "/C=RU/ST=Moscow Region/L=Moscow/O=Test/CN=shardman_tools"
#
openssl x509 -req -in shardman_tools.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out shardman_tools.crt -days 10000
2.7.2. Configuring etcd and shardmand Services #
Now configure services ( etcd and shardmand ) to use the generated certificates. To do this, perform the following steps:
On each etcd node, put
rootCA.crt
,nX.etcd.crt
andnX.etcd.key
in the location accessible to the etcd daemon (for example, create/etc/etcd
directory and put files there). Ensure that thenX.etcd.key
file is only accessible to the etcd daemon user.Specify the following configuration for etcd daemons in
/etc/default/etcd
:# unqualified first name ETCD_NAME=n1 # where we actually listen for peers ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380 # where we actually listen for clients ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379 # advertise where this machine is listening for clients ETCD_ADVERTISE_CLIENT_URLS=https://n1:2379 # --initial flags are used during bootstrapping and ignored afterwards, so it is # ok to specify them always # advertise where this machine is listening for peer ETCD_INITIAL_ADVERTISE_PEER_URLS=https://n1:2380 ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster # ansible_nodename is fqdn ETCD_INITIAL_CLUSTER=n1=https://n1:2380,n2=https://n2:2380,n3=https://n3:2380 ETCD_INITIAL_CLUSTER_STATE=new ETCD_DATA_DIR=/var/lib/etcd/default/member ETCD_AUTO_COMPACTION_RETENTION=1 ETCD_KEY_FILE=/etc/etcd/n1.etcd.key ETCD_CERT_FILE=/etc/etcd/n1.etcd.crt ETCD_TRUSTED_CA_FILE=/etc/etcd/rootCA.crt ETCD_CLIENT_CERT_AUTH=true ETCD_PEER_CERT_FILE=/etc/etcd/n1.etcd.crt ETCD_PEER_KEY_FILE=/etc/etcd/n1.etcd.key ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/rootCA.crt ETCD_PEER_CLIENT_CERT_AUTH=true
Replace
n1
here with the appropriate node name.Restart etcd services on all etcd cluster nodes:
#
systemctl restart etcd
To check the new configuration, use the following command:
#
etcdctl --endpoints=https://n1:2379,https://n2:2379,https://n3:2379 --cacert /etc/etcd/rootCA.crt --cert /etc/etcd/n1.etcd.crt --key /etc/etcd/n1.etcd.key member list -w table
+------------------+---------+------+-----------------+-----------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+------+-----------------+-----------------+------------+ | 66ebe06d7302c3f0 | started | n2 | https://n2:2380 | https://n2:2379 | false | | b1080bf5ff059980 | started | n1 | https://n1:2380 | https://n1:2379 | false | | d98323257249fefb | started | n3 | https://n3:2380 | https://n3:2379 | false | +------------------+---------+------+-----------------+-----------------+------------+
On each Shardman cluster node, put
rootCA.crt
,shardman_services.crt
andshardman_services.key
in a location accessible to thepostgres
user (for example, create the/etc/shardman
directory and put files there). Ensure that theshardman_services.key
file is only accessible to thepostgres
user.Edit the shardmand configuration file
/etc/shardman/shardmand-cluster0.env
as follows:SDM_STORE_ENDPOINTS=https://n1:2379,https://n2:2379,https://n3:2379 SDM_STORE_CA_FILE=/etc/shardman/rootCA.crt SDM_STORE_CERT_FILE=/etc/shardman/shardman_services.crt SDM_STORE_KEY=/etc/shardman/shardman_services.key
Restart
shardmand@cluster0
services on all Shardman nodes:#
systemctl restart shardmand@cluster0
2.7.3. Using Shardman Tools #
Before using Shardman tools, copy rootCA.crt
, shardman_tools.crt
and shardman_tools.key
to some location on the Shardman management node where they are accessible to the management user. Here, any node with installed Shardman utilities that is used to manage the Shardman cluster is meant by management node. This can also be one of the Shardman cluster nodes (or all of them). By management user, a user is meant who runs shardmanctl tool. It is assumed that the certificates and key are located in the /etc/shardman
directory.
When using Shardman tools, be sure to add --store-ca-file
, --store-cert-file
and --store-key
options to shardmanctl
command. For example, the following command gets the cluster status:
#
shardmanctl --store-ca-file /etc/shardman/rootCA.crt --store-cert-file /etc/shardman/shardman_tools.crt --store-key /etc/shardman/shardman_tools.key --store-endpoints https://n1:2379,https://n2:2379,https://n3:2379 status