Navigation :

Setting up Upstream TLS

You can configure Gloo Gateway to use TLS or mTLS when connecting to upstream services.

For certificates that are issued by a trusted certificate authority (CA), the upstream automatically uses TLS when the useTls: true setting is included in the static upstream spec, or the port is set to 443 (and useTls is not explicitly set to false). If the TLS certificate is not trusted or you want to verify the certificate, continue with the following sections.

Prepare sample environment

Let’s deploy a sample application and configure a route to it. We will expect the route to return errors because the sample application is serving HTTPS, not HTTP.

kubectl apply -n default -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: example-tls-server
  name: example-tls-server
spec:
  selector:
    matchLabels:
      app: example-tls-server
  replicas: 1
  template:
    metadata:
      labels:
        app: example-tls-server
    spec:
      containers:
      - image: docker.io/soloio/example-tls-server:latest
        imagePullPolicy: Always
        name: example-tls-server
        ports:
        - containerPort: 8080
          name: http
---
apiVersion: v1
kind: Service
metadata:
  name: example-tls-server
  labels:
    service: example-tls-server
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    app: example-tls-server
EOF

If we query the Gloo Gateway upstream generated by Discovery, we should see it:

glooctl get upstream default-example-tls-server-8080

+---------------------------------|------------|----------|--------------------------------+
|            UPSTREAM             |    TYPE    |  STATUS  |            DETAILS             |
+---------------------------------|------------|----------|--------------------------------+
| default-example-tls-server-8080 | Kubernetes | Accepted | svc name:                      |
|                                 |            |          | example-tls-server             |
|                                 |            |          | svc namespace: default         |
|                                 |            |          | port:          8080            |
|                                 |            |          |                                |
+---------------------------------|------------|----------|--------------------------------+

Now let’s create a route to the example tls server like we did in the hello world tutorial

glooctl add route \
    --path-exact /hello \
    --dest-name default-example-tls-server-8080

One-way TLS

Envoy will connect to an upstream server over HTTPS.

In this case, the Upstream CRs require a sslConfig block to create an HTTPS request. Otherwise, Envoy will try plain text HTTP.

Untrusted mode

By default, Envoy will not verify the upstream server certificate. But you still have to provide Gloo with a TLS secret to enable HTTPS (you need the two default fields named tls.key and tls.crt).

The certificates used for this service are hard-coded for testing purposes. Let’s create local copies of them:

# create the certificate
cat > cert.pem <<EOF
-----BEGIN CERTIFICATE-----
MIIDeDCCAmACCQDigH3yyOvEADANBgkqhkiG9w0BAQsFADB+MQswCQYDVQQGEwJV
UzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNTW91bnRhaW4gVmlldzEa
MBgGA1UECgwRWW91ciBPcmdhbml6YXRpb24xEjAQBgNVBAsMCVlvdXIgVW5pdDES
MBAGA1UEAwwJbG9jYWxob3N0MB4XDTE5MTAwOTEzMjkwOVoXDTI5MTAwNjEzMjkw
OVowfjELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcM
DU1vdW50YWluIFZpZXcxGjAYBgNVBAoMEVlvdXIgT3JnYW5pemF0aW9uMRIwEAYD
VQQLDAlZb3VyIFVuaXQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAOFNQLb4iFwM8z1llBmgBtgYdXcJyatgKcNr2OdU
P5q1GijCblcc20+6RaOY0i1ibkFwlxDNn22TxvANSQlm4eMZpE/N9/isVuIA9s6A
7DlQxjcxOOWj0308ukmSzSNv+SGGxDh9yeOxh3O7tQRb6y9FTEpe32qTMRkDMXNs
1pSP+JKRF3huTgt+kdxSXm2I9+ZMQURxky3cFO4eQ/cI05q1gcJViqa0rT8saDNz
kqPOm9OCWWVP9rBE04Ilyj4CKFvzlToKJOopRd6owGL8tSv+bdy+D8EFiqvwSXE6
EpaduMp/HfWVaOfrJDsfNn/3imu4fWtlCN6jndQHcG77y60CAwEAATANBgkqhkiG
9w0BAQsFAAOCAQEAWRn+YUOhltQAtnMZeDMfjMw0UJRjCrczfOmwAFnVqbhhtaev
F5XMbTYInPE0xIHlRVN8RxAvdLxMrRFhOBdKBM7efMD0XVMo5TeQzL90M0Nozgly
IOSuMlBCL8tyoSLPpMy+jmY2ALYooxoXOqC6fdLbvPAIDoTuRlRB3zq8OX58yt+J
lYvOIZmr5ImoKQvLn8fsUZtY93e6bo+l+iFBgbxo5UovZp1IjehWGprViC1+INZr
PRzwa8qrDxnNsVdUFQhgKb6s+uFmjtN8fqF00t2xvdKblJr5WN3TnFiQgLX+DFyB
p80GBupbMkI8FUpM3QwCPRxkIE74WY4dCgbkng==
-----END CERTIFICATE-----
EOF

# create the key
cat > key.pem <<EOF
-----BEGIN PRIVATE KEY-----
MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDhTUC2+IhcDPM9
ZZQZoAbYGHV3CcmrYCnDa9jnVD+atRoowm5XHNtPukWjmNItYm5BcJcQzZ9tk8bw
DUkJZuHjGaRPzff4rFbiAPbOgOw5UMY3MTjlo9N9PLpJks0jb/khhsQ4fcnjsYdz
u7UEW+svRUxKXt9qkzEZAzFzbNaUj/iSkRd4bk4LfpHcUl5tiPfmTEFEcZMt3BTu
HkP3CNOatYHCVYqmtK0/LGgzc5KjzpvTglllT/awRNOCJco+Aihb85U6CiTqKUXe
qMBi/LUr/m3cvg/BBYqr8ElxOhKWnbjKfx31lWjn6yQ7HzZ/94pruH1rZQjeo53U
B3Bu+8utAgMBAAECggEAPH6eusJe8sBza2/j5UGHtOxUVgMlyENI03UYx3xim6q2
/GzAbdmMtYqhejzlalQ8oIuXtGZRwX1ldD1M+B5M1sqiyN7YD0hPB94UZvxM8VLT
9ivcSCTF+6Gbr3egZzyAm1TxSO3VkLKxWQz0nNgFfSrRQkLZIGenTj0CQSjfMQI6
NPbJbp3mD9AQsKjDSvwo4rwwSe8vedXyYikuwMABHZ0bydIeuB+So8LvUZipToM5
jf08GejdeL+vuNoCueRsJ2hvMrGrqdfKx3+FcyB26SKv9nuQmv7verxlAn+t+Afi
pH/1BOHrCoJQgP5lGH0H+UmGwf9HO8ciJqu3dWd7rQKBgQD7RTdLGBWQS451ULxI
Utj62iHv/TzB3vge2ZGbTb56SefhPkcj1o1NXDqBIYOpusYddcr38LeBHs3QBSF9
WMvRa2NojV7R5CUXvEzioH7GlMOaL7sIjmoBYjE1QRsfuKUAcKDXw3VJVYuoHRgK
HswFSvhtELEihID6UZcAvWpvDwKBgQDliuc2ipMRjNCMune5uoElYDyamRM/4v8x
GYRAWoRTCVcsAKGtUso9/CXsw9+Ld2Q7XqaEsD4YsyfWQEED2gIY6OcEURAT/o+o
IQQdJnO4A0EJks2mhLzBwN3has/Q0IgJXOZp2ulZ+bsbh3GeifXnd6NF2p9IM1/Q
VaXl6IuZgwKBgQDbxxftE/zQgHXzgRGexPBKwf8LNdotzQQDn9PvHlosBnbOmjWJ
UEG516DIj/Lkw5xD6mME6UToqHPmroYzaDamTyLdMUItnjsffrFVTIJ22WoZdARJ
IJ/x49wcs3yxC0UvlFPrRWhSI4QLIJ+FQpi7TG7snrwA8BsMV88Xc5Yj2wKBgQDK
MRB5epcRXnhVfer4LtCTm7HGfA/4tnsTROa5yQHGIvQmTmgbxFFhSDof1GmU8BXa
NgV328bW+vicQP0D54TxbDYSF1WSRylDb9Gv268S58riI+4CP+oEwV6wsOVdilJJ
7QsJM0tZdiDanvP2Mo/o0/l+DpU/hAFiAg+f9LcDAQKBgQCTPi8SrDqwQGXI+S+P
6j8thYveTF5qRSpbIxA0sXXXFF6Ufa3F+uhbiEAEohgtMkyw1pjjixtqIAwspI0R
UkZt4Iyw6JHANf03UDGCDunBaDoHxef0e3k9/kC38dbbq2vaKUvN5kLqhDa0t798
kXRTzTu3F02+HoyBwn0QC0tWWg==
-----END PRIVATE KEY-----
EOF

Now we should create the Kubernetes secret to hold these certs:

kubectl create secret tls upstream-tls --key key.pem \
   --cert cert.pem --namespace default

secret/upstream-tls created

Now we’ve got to configure the default-example-tls-server-8080 upstream to reference this secret in its sslConfig.

This can be done using either glooctl to modify the Upstream directly, or adding an annotation to the the service:

glooctl
kubectl


glooctl edit upstream \
    --name default-example-tls-server-8080 \
    --namespace gloo-system \
    --ssl-secret-name upstream-tls \
    --ssl-secret-namespace default


kubectl annotate service -n default example-tls-server gloo.solo.io/sslService.secret=upstream-tls

See the guide on using Service annotations to configure SSL for the full set of options when using Service annotations to configure upstream SSL.

Now if we get the default-example-tls-server-8080 Upstream, we should see the new SSL configuration:

kubectl get upstream -n gloo-system \
    default-example-tls-server-8080 -o yaml

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  labels:
    discovered_by: kubernetesplugin
    service: example-tls-server
  name: default-example-tls-server-8080
  namespace: gloo-system
spec:
  discoveryMetadata: {}
  kube:
    selector:
      app: example-tls-server
    serviceName: example-tls-server
    serviceNamespace: default
    servicePort: 8080
  sslConfig:
    secretRef:
      name: upstream-tls
      namespace: default
status:
  reportedBy: gloo
  state: 1

Try the request again

curl $(glooctl proxy url)/hello

Now you should see the following response:

Hello, world!

Great! Now we’ve seen how Gloo Gateway can be configured to encrypt traffic sent to a backend service which is configured to serve TLS.

Trusted mode

If you want Envoy to verify the upstream server certificate, there are two ways:

using a standard TLS secret, as shown above, with an additional field named ca.crt. Only this ca.crt field will be transmitted from Gloo to Envoy, as part of trusted CAs.
using an Opaque secret created with the following command:

glooctl create secret tls --rootca=...

Two-way TLS (mTLS)

During the TLS handshake, the upstream server will ask Envoy to present a client certificate.

Client certificate without a trust store

There are two ways:

using a standard TLS secret with the tls.key and tls.crt fields
using an Opaque secret created with the following command:

glooctl create secret tls --privatekey ... --certchain ...

Client certificate with a trust store

There are two ways:

using a standard TLS secret with the tls.key, tls.crt and also ca.crt fields
using an Opaque secret created with the following command:

glooctl create secret tls --rootca=...

Do not use a generic secret with these three keys (tls.crt, tls.key, ca.crt) because Gloo does not recognize generic secrets in this case. Instead, you must create a tls secret from an existing .PEM encoded public and private key pair.

Gloo Gateway supports client-side TLS where the proxy (Envoy) presents a certificate to upstream servers when initiating a connection on behalf of a downstream client, encrypting all traffic between the proxy and the upstream.

Troubleshooting

Secret not found

Sometimes, the Gloo control plane might not process and send the sslConfig to the Envoy data plane. To check if a sync issue happened, review the Gloo pod logs.

kubectl -n gloo-system logs -l gloo=gloo

Example of an issue:

{"level":"warn","ts":1631524528.8111176,"logger":"gloo-ee.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"syncer/envoy_translator_syncer.go:140","msg":"proxy gloo-system.gateway-proxy was rejected due to invalid config: 2 errors occurred:\n\t* invalid resource gloo-system.default-server-mtls\n\t* SSL secret not found: list did not find secret default.client-mtls\n\n\nAttempting to update only EDS information","version":"1.17.4"}

To see which sslConfig is actually used by Envoy, run an Envoy config dump.

Mismatched Ciphers

There have been cases where users have experienced issues with valid Upstream TLS configurations failing to even establish a connection with the upstream service. In some cases, this is due to a reduction in the set of ciphers presented by default to upstream services. This change occurred in Envoy 1.17.0 (January 2021) when RSA key transport and SHA-1 cipher suites were removed from client-side defaults. This impacts Gloo Gateway versions 1.6+. If the default cipher set offered by Envoy does not match any of the ciphers in the upstream service’s cipher suite, then requests will fail as described earlier.

How to Detect Mismatched Ciphers

The current set of default ciphers offered by Envoy clients is documented here. This is the current complete list:

[ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-CHACHA20-POLY1305]
[ECDHE-RSA-AES128-GCM-SHA256|ECDHE-RSA-CHACHA20-POLY1305]
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384

If your upstream server does not support at least one of these ciphers in its TLS config, then requests will fail. To determine the contents of your server’s cipher suite, we recommend using this bash script:

#!/usr/bin/env bash

# OpenSSL requires the port number. Example: httpbin.org:443
SERVER=$1
DELAY=1
ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')

echo Obtaining cipher list from $(openssl version).

for cipher in ${ciphers[@]}
do
echo -n Testing $cipher...
result=$(echo -n | openssl s_client -cipher "$cipher" -connect $SERVER 2>&1)
if [[ "$result" =~ ":error:" ]] ; then
  error=$(echo -n $result | cut -d':' -f6)
  echo NO \($error\)
else
  if [[ "$result" =~ "Cipher is ${cipher}" || "$result" =~ "Cipher    :" ]] ; then
    echo YES
  else
    echo UNKNOWN RESPONSE
    echo $result
  fi
fi
sleep $DELAY
done

The script will evaluate a list of known ciphers against your upstream service. The output will look something like this:

% ./cipher-test.sh httpbin.org:443
Obtaining cipher list from LibreSSL 2.8.3.
Testing ECDHE-RSA-AES256-GCM-SHA384...YES
Testing ECDHE-ECDSA-AES256-GCM-SHA384...NO (sslv3 alert handshake failure)
Testing ECDHE-RSA-AES256-SHA384...YES
Testing ECDHE-ECDSA-AES256-SHA384...NO (sslv3 alert handshake failure)

Remediating Mismatched Ciphers

Are there any matches between the ciphers in Envoy’s default list and the cipher suite offered by your upstream service, as enumerated by the YES entries from the cipher list bash script? If not, then you will need to modify your Gloo Gateway Upstream configuration to support a custom cipher list that does match at least one of the offered ciphers. The example below adds the cipher ECDHE-RSA-AES256-SHA to the default Envoy client list.

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: my-upstream
  namespace: gloo-system
spec:
  sslConfig:
    secretRef:
      name: my-upstream-tls
      namespace: gloo-system
    parameters:
      cipherSuites:
        - "[ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-CHACHA20-POLY1305]"
        - "[ECDHE-RSA-AES128-GCM-SHA256|ECDHE-RSA-CHACHA20-POLY1305]"
        - "ECDHE-ECDSA-AES256-GCM-SHA384"
        - "ECDHE-RSA-AES256-GCM-SHA384"
        - "ECDHE-RSA-AES256-SHA"
  static:
    hosts:
      - addr: my-upstream.example.com
        port: 443
    useTls: true

Once the custom cipher suite list is deployed in your Envoy configuration, then the mismatched cipher problem should disappear.