2 posts tagged with "certificates"

In an earlier post, I installed cert-manager to automatically manage my SSL certificates for TLS for my home-lab services. Given it was a set of services within my internal network, I was comfortable with issuing a single wildcard certificate *.local.kasj.live.

The problem​

I've since realised a bit of an issue. The wildcard certificates that are issued are issued as a kubernetes secret resource which is specific to a namespace. This meant that in order for me to use the wildcard cert for all my services, I needed to deploy all my services into the same namespace (not ideal).

The research​

Turns out there were many solutions to this, many of which I don't really understand but I tried anyway. This included:

• Installing traefik again with K3s out of the box and getting it to issue certs through Let's Encrypt without cert-manager. This didn't seem to change the behaviour I was experiencing before.
• Storing the wildcard secret in a persistent volume for cert-manager to pick up as a "default" certificate when nothing else was provided. Unfortunately I couldn't get the persistent volume to work either

The solution​

I landed with a solution outline here and it still doesn't work how I want it to but its definitely a step forward.

The first few steps are exactly as I'd performed in my earlier post:

1. Install Traefik
2. Install Cert-Manager
3. Set up Let's Encrypt as an Issuer

This is where I learnt something new:

4. Issue a wilcard certificate in the same namespace as Traefik in my case this was the kube-system namespace - I also issued myself a new one here to keep things fresh *.home.kasj.live

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-home-kasj-live
  namespace: kube-system
spec:
  secretName: wildcard-home-kasj-live-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  # commonName: "*.home.kasj.live"
  dnsNames:
    - "home.kasj.live"
    - "*.home.kasj.live"

Traefik by default normally uses its own self-signed certificate for each ingress service that you define. What I needed to configure was something to tell Traefik to serve the new wildcard certifate I'd created instead. This can be done through a kubernetes resource called TLSStore.

5. Create a TLSStore resource with the name default. According to the article above, it needed to be called default to be picked up by Traefik by default:

---
apiVersion: traefik.containo.us/v1alpha1
kind: TLSStore
metadata:
  name: default
  namespace: kube-system
spec:
  defaultCertificate:
    secretName: wildcard-home-kasj-live-tls"

6. Restart Traefik deployment so that it knows to pick up the new cert by default

Testing the new solution​

To test if Traefik was issuing my new wildcard certificate by default, I created a simple nginx server and exposed it using the following manifest on test.home.kasj.live:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  namespace: nginx
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/redirect-entry-point: https
spec:
  rules:
    - host: test.home.kasj.live
      http:
        paths:
          - backend:
              service:
                name: nginx
                port:
                  number: 80
            path: /
            pathType: Prefix

Note: I've moved away from the IngressRoute resource to Ingress

Success!

Now earlier, I mentioned it still wasn't working as I wanted it to and that's because there is still an issue with some services that already expose their services on SSL/HTTPS (Port 443) by default like nextcloud. Stay tuned for a future post on how I tackle that one but for now I'm going to enjoy this win.

Automatic Certificate Management Environment (ACME) Certificates can are usually provided through issuers. LetsEncrypt is a nonprofit Certificate Authority that provides free TLS certificates to millions of websites all around the world. This is was good enough for me!

Adding cloudflare token to cert-manager​

First I needed a domain name which I purchased through CloudFlare but can be from anywhere really. You guessed it - mine is kasj.live. From there I needed to obtain an cloudflare token which was a personal access token to manage my DNS records in my cloudflare account. I needed this as I needed to provide it to cert-manager, which will be brokering the certificates between letsencrypt and my domain.

Providing cert-manager my cloudflare token could be done with a simple manifest:

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  cloudflare-token: <redacted>

To apply the manifest run:

kubectl apply -f secret-cf-token.yaml

Adding Let's Encrypt as an Issuer to cert-manager​

I now need to let cert-manager know that I'll be using Let's Encrypt as my certificate issuer of choice through another manifest:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: kasunj@gmail.com
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
      - dns01:
          cloudflare:
            email: kasunj@gmail.com
            apiTokenSecretRef:
              name: cloudflare-token-secret
              key: cloudflare-token
        selector:
          dnsZones:
            - "kasj.live"

and execute using:

kubectl apply -f letsencrypt-production.yaml

Issuing certificates​

With the issuer now configured, all I need to do is request for a certificate. I will be hosting all my internal applications under the subdomain local.kasj.live so i will request for a wildcard certicate that covers *.local.kasj.live

The certificate is issued with the following manifest:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: local-kasj-live
  namespace: default
spec:
  secretName: local-kasj-live-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  commonName: "*.local.kasj.live"
  dnsNames:
  - "local.kasj.live"
  - "*.local.kasj.live"

and execute using:

kubectl apply -f local-kasj-live.yaml

Issuing and validating the certificates takes time (20 minutes minimum). To check how things are progressing run:

kubectl get challenges

Testing the issued certificate​

Once the kubectl get challenges command produces nothing, that's when you know the process is complete. To use a certificate, you need to ensure a couple of things:

• The certificate needs to be made available in multiple namespaces. The certificate only works if it is deployed in the same namespaces as the service you are using it for. With a bit of googling I've been using the following solution for this.

• We use Traefik to specify and ingressRoute which essentionally provides traefik with the instructions on where to route traffic hitting the reverse proxy. We can also specify here that a certificate must be used.

To test above, I deployed the Traefik dashboard (with the help of their documentation and TechnoTim) with the following steps:

Create and deploy a middleware manifest that forces https:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: traefik-dashboard-basicauth
  namespace: traefik
spec:
  basicAuth:
    secret: traefik-dashboard-auth

Generate a credential whichi is mandatory for the dashboard:

# Generate a credential / password that’s base64 encoded
htpasswd -nb kas <redacted> | openssl base64

Create and apply a manifest to deploy the dashboard. Note you need to use the output from command above for the password:

---
apiVersion: v1
kind: Secret
metadata:
  name: traefik-dashboard-auth
  namespace: traefik
type: Opaque
data:
  users: <redacted hased password which is output from above>

Finally I create a manifest for an ingressRoute which will route traffic from traefik.local.kasj.live to my dashboard using TLS certificate I just created:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
  annotations:
    kubernetes.io/ingress.class: traefik-external
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`traefik.local.kasj.live`)
      kind: Rule
      middlewares:
        - name: traefik-dashboard-basicauth
          namespace: traefik
      services:
        - name: api@internal
          kind: TraefikService
  tls:
    secretName: local-kasj-live-tls

And the results​

So now if I navigate to https://traefik.local.kasj.live I can not see the traefik dashboard

And more importantly with a certificate issued from Let's Encrypt!