Tailscale is a VPN service that makes it easy to create secure networks and connections between devices. The main reason I want to use Tailscale is that I need a secure and private way to access services that should not be publicly accessible. Tailscale allows me to create a virtual private network (VPN) where my devices and services can communicate securely over the internet, without exposing them to the public.

How to create your own Tailscale account with OpenId Connect (OIDC)?

This use case has already been described in my previous blog post How to set up a Tailscale account with OIDC?.

How to deploy Tailscale Operator in Kubernetes?

I decided to deploy the Tailscale Operator which manages the Tailscale Custom Resource Definitions (CRDs) in my Kubernetes cluster. For the deployment, I used the Helm chart provided by Tailscale. However, I created it as a rainbow chart to include my custom values like syncing the credentials from my 1Password vault. Here is an example of my values.yaml file:

Chart.yaml

1
2
3
4
5
6
7
8
9

apiVersion: v2
name: tailscale
version: 1.0.0

dependencies:
  - name: tailscale-operator
    version: 1.92.5
    repository: https://pkgs.tailscale.com/helmcharts
    alias: tso

values.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

tailscaleOperator:
  oauth:
    onePasswordItemPath: "vaults/<my-vault-uuid>/items/<my-item-uuid>"
    restartOnChange: true
tso:
  operatorConfig:
    image:
      repository: ghcr.io/tailscale/k8s-operator
      # tag: "1.92.5" # will be set by the dependency
      pullPolicy: IfNotPresent
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 64Mi

Update your access control settings in Tailscale

To allow the Tailscale Operator to join your Tailscale network, you need to update your access control settings in Tailscale. Navigate to the Tailscale Admin Console - Tags section. Add a new tag called tag:k8s-operator and allow devices with this tag to join your network.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  ...
  "tagOwners": {
    "tag:k8s-operator": [],
    "tag:k8s": [
      "tag:k8s-operator"
    ]
  },
  ...
}

How to get the OAuth credentials?

Open Tailscale and navigate to the “Settings” page. Under the “Trust credentials” section, create new credentials. Choose “OAuth” as the type and enter “Kubernetes Operator” as the description.

As you can see, I set the onePasswordItemPath to sync the OAuth credentials from my 1Password vault. This way, I can securely manage my Tailscale OAuth credentials without hardcoding them in the Helm values. Therefore, I want that the operator reads the credentials from a secret which is synced from 1Password. My secret looks like this:

Consider that the secret must be called operator-oauth and must be created in the same namespace where the Tailscale. Also, the keys must be named client_id and client_secret. Read more

1
2
3
4
5
6
7
8
9

apiVersion: v1
data:
  client_id: <base-64-encoded-client-id>
  client_secret: <base-64-encoded-client-secret>
kind: Secret
metadata:
  name: operator-oauth
  namespace: tailscale
type: Opaque

Granting additional permissions to the Tailscale Operator

One small note, you need to give the Tailscale operator a bit more permissions to create the necessary resources. Therefore, I also added a namespace resource in my templates folder of the rainbow chart:

I use argo-cd hooks to ensure that the namespace is created before any other resources are applied. If you are using helm, you can use helm hooks instead.

1
2
3
4
5
6
7
8

apiVersion: v1
kind: Namespace
metadata:
  name: tailscale
  annotations:
    argocd.argoproj.io/hook: PreSync # ensures the namespace is created before other resources are applied
  labels:
    pod-security.kubernetes.io/enforce: privileged # adds the privileged policy to the namespace

After creating the rainbow chart, I deployed it to my Kubernetes cluster using Argo CD.

How to access Kubernetes applications through Tailscale?

To access Kubernetes applications through Tailscale, I used the Tailscale Ingress Controller. This controller allows me to expose my Kubernetes services to the Tailscale network securely. Here is an example of how I configured the Ingress Controller:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  defaultBackend:
    service:
      name: my-service
      port:
        number: 8000
  ingressClassName: tailscale # ensures that the Tailscale Ingress Controller handles this ingress
  rules:
    - host: myservice.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  number: 8000
  tls:
    - hosts:
        - myservice.example.com
      secretName: my-service-tls

Accessing the application through Tailscale

To access my Kubernetes application through Tailscale, I first need to ensure that my Tailscale client is running on my client. Then, I have to create a CNAME DNS record that points to the Tailscale IP address of my Kubernetes cluster. This way, when I access myservice.example.com, the request is routed through Tailscale to my Kubernetes cluster. Get the Tailscale DNS name of your Kubernetes cluster from the Tailscale Admin Console under Machines. Open the entry for your service and search for Full domain. Copy the value and create the CNAME record in your DNS provider:

1
2
3
4

CNAME myservice.example.com -> <tailscale-dns-name-of-k8s-cluster>

# Example:
CNAME myservice.example.com -> my-service.tail12345.ts.net

Consider that DNS propagation might take some time depending on your DNS provider. You can use tools like dig to check if the DNS record has propagated:

1

dig +short myservice.example.com

And that’s it! Now I can access my Kubernetes applications securely through Tailscale.

Conclusion

In this blog post, I explained how I set up Tailscale in my Kubernetes cluster using the Tailscale Operator and how I access my Kubernetes applications through Tailscale using the Tailscale Ingress Controller. Tailscale provides a secure and private way to connect to services without exposing them to the public internet. I hope this guide helps you set up Tailscale in your own Kubernetes cluster! Thanks to Josh Noll for his great guide on this topic.

Additional resources

• Josh Nolls’ Guide

Tailscale is a VPN service that makes it easy to create secure networks and connections between devices. The main reason I want to use Tailscale is that I need a secure and private way to access services that should not be publicly accessible. Tailscale allows me to create a virtual private network (VPN) where my devices and services can communicate securely over the internet, without exposing them to the public.

Why I want to use ZITADEL as an OIDC provider

ZITADEL is an IdP that provides identity and access management. It is already my main IdP for other services, so it makes sense to use it for Tailscale as well. By using ZITADEL as my OIDC provider, I can reuse my existing user accounts and authentication mechanisms, which simplifies the overall management of my Tailscale network.

Why I created this guide

Even though setting up Tailscale with an OIDC provider is quite well documented, I still faced some difficulties during the configuration process. Therefore, I decided to create this guide to help others who might face similar challenges when setting up Tailscale with ZITADEL as their OIDC provider.

First, some background on how this works: Tailscale loads the OIDC configuration using WebFinger, based on the domain of the email address you use to log in. For example, if I want to use my own email account name@example.com to authenticate with Tailscale, Tailscale will look for the OIDC configuration at:

https://example.com/.well-known/webfinger

This endpoint must return the following JSON:

1
2
3
4
5
6
7
8
9

{
  "subject": "acct:name@example.com",
  "links": [
    {
      "rel": "http://openid.net/specs/connect/1.0/issuer",
      "href": "https://account.example.com"
    }
  ]
}

The href field points to the issuer URL of the OIDC provider. You can find this value in the OpenID configuration endpoint:

https://account.example.com/.well-known/openid-configuration

Also note that the rel field is a fixed value defined by the OpenID Connect specification.

Using the WebFinger of my main domain

Since I already have a website running at example.com, but did not want to add the WebFinger file to my main website, I tried to set up a dedicated WebFinger server. However, there are only a few WebFinger server implementations available, and none of them worked well for my use case. Most of them are no longer actively maintained, or cannot easily be deployed on Kubernetes.

For example, I tried https://github.com/peeley/carpal, which seemed to be one of the more popular and maintained projects. However, it defines the WebFinger subject (such as acct:name@example.com) directly in the filename. Kubernetes does not allow mounting files with special characters like : and @, which made this approach unusable for me. I also did not want to introduce an external database just for running a WebFinger server.

In the end, I decided to use a static Caddy server and simply serve a static JSON file.

Caddyfile - ConfigMap

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: v1
kind: ConfigMap
metadata:
  name: caddy-config
data:
  Caddyfile: |
    :{{ .Values.service.port | default 8080 }} {
      root * /srv
      try_files {path} /.well-known/webfinger
      file_server

      header {
        Content-Type application/jrd+json
      }
    }

webfinger.json - ConfigMap

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: v1
kind: ConfigMap
metadata:
  name: webfinger-resource
data:
  webfinger.json: |
    {
      "subject": "acct:name@example.com",
      "links": [
        {
          "rel": "http://openid.net/specs/connect/1.0/issuer",
          "href": "https://account.example.com"
        }
      ]
    }

Deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webfinger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webfinger
  template:
    metadata:
      labels:
        app: webfinger
    spec:
      containers:
        - name: caddy
          image: caddy:alpine
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 15m
              memory: 10Mi
            limits:
              memory: 16Mi
          volumeMounts:
            # Caddy config
            - name: caddy-config
              mountPath: /etc/caddy/Caddyfile
              subPath: Caddyfile

            # WebFinger JSON served as a FILE
            - name: webfinger-resource
              mountPath: /srv/.well-known/webfinger
              subPath: webfinger.json

      volumes:
        - name: caddy-config
          configMap:
            name: caddy-config

        - name: webfinger-resource
          configMap:
            name: webfinger-resource

Service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: Service
metadata:
  name: webfinger-service
spec:
  selector:
    app: webfinger
  ports:
    - name: http
      port: 8080
      targetPort: 8080

Ingress

You can keep your other ingress for the main website and just add a new path for the webfinger service. However, both need to be accessible on the same host.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "webfinger-path"
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/use-regex: "true" # required to mach the guest path
    kubernetes.io/ingress.class: 'nginx'
    cert-manager.io/cluster-issuer: 'letsencrypt-prod'
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /.well-known
            pathType: ImplementationSpecific
            backend:
              service:
                name: webfinger-service
                port:
                  number: 8080
  tls:
    - hosts:
        - example.com
      secretName: webfinger-service-tls

Configure ZITADEL

Create a new ZITADEL application for Tailscale and make sure that the user you want to use for login has access to this application. It is important that the email address or username matches the subject you configured in the WebFinger setup.

Create a Web Application of type Code and add the following Redirect URI:

https://login.tailscale.com/a/oauth_response

Configure Tailscale

In Tailscale, go to the sign-up page and choose OIDC as the authentication method. Enter the email address associated with your ZITADEL account (for example, name@example.com). Tailscale will use WebFinger to discover the OIDC provider and then redirect you to ZITADEL for authentication.

After a successful login, you will be prompted to enter the clientId and clientSecret of the ZITADEL application you created earlier. Once these credentials are entered, you should be redirected back to Tailscale and logged in successfully.

How to verify that everything works

To verify that everything is working correctly, you can open an incognito or private browsing window in your web browser and try to log in to Tailscale using the same email address.

Keep in mind that you must first enter your email address on the Tailscale sign-in page. After that, you will be redirected to ZITADEL for authentication.

Conclusion

Setting up Tailscale with ZITADEL as a OIDC provider needs some tweaks, especially when it comes to the WebFinger server. However, this only lets you create a basic Tailscale account. If you want to protect your services with Tailscale, you will need to add Tailscale in front of your services, which is a topic for another guide.

Additional resources

• Tailscale OIDC documentation

Harbor is an open-source container image registry that can also function as a chart or image proxy. This is especially useful when you want to cache images from Docker Hub or other registries. My main reason for using Harbor was to bypass the rate limits imposed by Docker Hub.

What is Zitadel?

Zitadel is an open-source identity and access management (IAM) solution that provides authentication and authorization services. It supports multiple authentication methods, including OAuth2, OpenID Connect and so on. Zitadel allows you to manage user identities, roles, and permissions for your applications and services.

How to Protect Harbor with Zitadel

To secure Harbor with Zitadel, you need to configure Harbor to use Zitadel as an external authentication provider. My goal was to achieve this by updating the Harbor configuration file and setting up Zitadel as an OAuth2 provider. However, a few additional steps were required to make it work.

Prerequisites

• A running Harbor instance (I used the official Helm chart)

  • If you want to use PKCE, make sure to use Harbor v2.13.0 or higher.

• A running Zitadel instance (I used the official Helm chart).

Step 1: Configure Zitadel

1. Log in to your Zitadel instance and create a new PKCE application.

   • The redirect URL should be https://<your-harbor-domain>/c/oidc/callback.

2. In the Token Settings tab, enable User roles inside ID Token.

3. Create a role called harbor_administrators (or any name you prefer) and assign it to users who should have admin access in Harbor.

4. Go to the Action tab and create a new action:

   • Name: flatRoles (the name must match the function name)

   • Code:

     1 2 3 4 5 6 7 8 9 10 11 12 13 14

     function flatRoles(ctx, api) { if (ctx.v1.user.grants == undefined || ctx.v1.user.grants.count == 0) { return; } let grants = []; ctx.v1.user.grants.grants.forEach(claim => { claim.roles.forEach(role => { grants.push(role) }) }); api.v1.claims.setClaim('groups', grants) }

   • Timeout: 10 seconds

   • Enable Allowed To Fail

5. Go to the Flow section and select Complement Token as the Flow Type.

6. Add triggers: Pre Userinfo creation and Pre access token creation, selecting the flatRoles action for both triggers.

7. Update your Harbor configuration with the following userSettings:

   Make sure the JSON is valid. If you get errors, check for issues like comments (//) which aren’t valid in JSON.

   1 2 3 4 5 6 7 8 9

   { "auth_mode": "oidc_auth", // must be set to oidc_auth to enable OIDC authentication "oidc_name": "Zitadel", // display name for the OIDC provider "oidc_endpoint": "https://zitadel.domain.com", // issuer of your Zitadel instance "oidc_client_id": "335927275759403852", // client ID of your PKCE application we created earlier "oidc_scope": "openid,profile,email,offline_access", // scopes to request "oidc_groups_claim": "groups", // claim in the ID token that contains the user's groups "oidc_admin_group": "harbor_administrators" // group (which we created earlier) that will be mapped to Harbor administrators }

8. Restart Harbor to apply the changes.

9. Log out of Harbor; you should see a new login button with the name specified in oidc_name.

10. Log in with a user assigned to the harbor_administrators role to gain admin access.

That’s it! Your Harbor instance is now using Zitadel as IdP, allowing you to manage user access and permissions centrally.

Troubleshooting

To inspect your JWT token, you can use Auth-Playground. To check the group claim, temporarily add https://auth-playground.makefermion.com/oauth2/ as a redirect URL in your PKCE application on Zitadel.

Settings example:

• Authorize URL: https://zitadel.domain.com/oauth/v2/authorize
• Redirect URL: https://auth-playground.makefermion.com/oauth2
• Client-ID: <your-client-id-of-the-pkce-application>
• Client Secret: leave empty (not needed for PKCE)
• Response type: code
• Code Challenge: <your-code-challange> (ignore the placeholder on the site)
• Code Challenge Method: S256
• Scopes: openid profile email offline_access
• Custom attributes: state=<random-string>

After logging in, you’ll be redirected back to Auth-Playground. Copy the code shown in the pop-up and exchange it for a token:

1
2
3
4
5
6
7
8

curl --request POST \
  --url 'https://zitadel.domain.com/oauth/v2/token' \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'client_id=<your-client-id>' \
  --data-urlencode 'code_verifier=<original-code-verifier>' \
  --data-urlencode 'code=<code-from-popup>' \
  --data-urlencode 'grant_type=authorization_code' \
  --data-urlencode 'redirect_uri=https://auth-playground.makefermion.com/oauth2'

Generate Code Challenge and State

Generate a code verifier and challenge with:

1
2
3
4
5
6
7
8

CODE_VERIFIER=$(openssl rand -base64 64 | tr -d '=+/' | tr -d '\n')
echo "CODE_VERIFIER: $CODE_VERIFIER"

CODE_CHALLENGE=$(echo -n $CODE_VERIFIER | openssl dgst -binary -sha256 | openssl base64 | tr -d '=+/')
echo "CODE_CHALLENGE: $CODE_CHALLENGE"

STATE=$(openssl rand -base64 32 | tr -d '=+/' | tr -d '\n')
echo "STATE: $STATE"

References

• Harbor - Configure User Settings
• Helpful comment on Zitadel Action

Setup

I already had a Zitadel instance running, along with an Ingress controller and Cert-Manager.

The first service I wanted to secure was the Kubernetes Dashboard. I deployed OAuth2-Proxy and configured it. At first, I mistakenly set up an internal upstream. This worked fine for the dashboard itself, but not for other apps - since OAuth2-Proxy only supports a single upstream.

That’s when I realized I had configured OAuth2-Proxy as a reverse proxy, instead of just as an authentication layer. I removed the upstream setting and instead defined allowed domains.

One important detail: when setting domain values, you must include the leading dot (e.g., .domain.tld). Without it, subdomains won’t be allowed through OAuth2-Proxy.

Here’s my final configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

provider = "oidc"
oidc_issuer_url = "https://zitadel.domain-a.tld"
scope = "openid email profile"
code_challenge_method = "S256" # Required for OIDC providers that use PKCE
pass_access_token = true
skip_provider_button = false
ssl_insecure_skip_verify = false
proxy_prefix = "/oauth2"

email_domains = [".domain-a.tld", ".domain-b.tld", "domain-a.tld", "domain-b.tld"]
whitelist_domains = [".domain-a.tld", ".domain-b.tld", "domain-a.tld", "domain-b.tld"]
cookie_domains = [".domain-a.tld", ".domain-b.tld", "domain-a.tld", "domain-b.tld"]
cookie_name = "_oauth2_proxy"
cookie_expire = "168h"
cookie_secure = true
cookie_httponly = true
cookie_samesite = "lax"
cookie_csrf_per_request = true
cookie_csrf_expire = "5m"
set_xauthrequest = true
pass_user_headers = true
pass_authorization_header = true
pass_basic_auth = false
reverse_proxy = false
show_debug_on_error = false

You also need to provide the following environment variables. While you could set them directly in the config file, I prefer to keep them in 1Password and sync them into my namespace as Kubernetes secrets — for better security and easier management:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

env:
  - name: OAUTH2_PROXY_CLIENT_ID
    valueFrom:
      secretKeyRef:
        name: oauth2-proxy-credentials
        key: client-id
  - name: OAUTH2_PROXY_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: oauth2-proxy-credentials
        key: client-secret
  - name: OAUTH2_PROXY_COOKIE_SECRET
    valueFrom:
      secretKeyRef:
        name: oauth2-proxy-credentials
        key: cookie-secret

Note: If you’re using Let’s Encrypt staging certificates, you’ll need to set ssl_insecure_skip_verify = true. Once you switch to production certificates, you can (and should) set it back to false.

Adding OAuth2-Proxy to Ingress

Once OAuth2-Proxy was properly configured and I could log in with my Zitadel account, I integrated it with my Ingress. After a bit of trial and error, I ended up with this setup, which now works reliably across all my services:

1
2
3
4
5
6
7

kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/auth-url: "https://oauth-proxy.domain.tld/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: >-
  https://oauth-proxy.domain.tld/oauth2/start?rd=$scheme://$host$request_uri

Depending on the service, I sometimes add these extra annotations:

1
2

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" # or "HTTP" if the service doesn’t use TLS
nginx.ingress.kubernetes.io/proxy-buffer-size: "8k"

This setup is based on the official Ingress NGINX OAuth2-Proxy guide, with a few tweaks to fit my use case.

In advance, I want to say that I’m still quite new to the whole Kubernetes world, so please be understanding if there are better ways to do things or if I did something wrong. :)

In autumn 2024, I heard about a new managed Kubernetes service here in Switzerland by my preferred hosting company, Infomaniak. The service was not released yet, but I already started planning the migration of my whole infrastructure. At that time, it was distributed across several VPSs, a Kubernetes cluster at DigitalOcean, and some services running at home. I started searching for Helm charts or wrote my own if none were available.

After Infomaniak finally released their managed Kubernetes service, I could breathe a sigh of relief, since the pricing was more or less what I expected. My main fear was that it would be as expensive as other providers and therefore not really worth it for a private person to run a cluster. Fortunately, Infomaniak released a free control plane, and you only pay for the worker nodes via their public cloud pricing.

I opened their calculator and added two worker nodes with 4 CPUs and 8 GB of memory each. The price was reasonable, and I could easily upgrade later to the next node size with 4 CPUs and 16 GB of memory. I also added a Load Balancer and a public IPv4 address to my shopping cart and started setting up my new Kubernetes cluster.

• Infomaniak’s Kubernetes service
• Infomaniak’s Public Cloud

Security Notes

At this point, it is important to mention that this article describes a personal Kubernetes setup and intentionally omits some production-grade hardening steps.

⚠️ Important considerations:

• The Kubernetes Dashboard should never be exposed publicly.
• cluster-admin roles are used here for demonstration purposes only.
• Kubernetes Secrets are base64-encoded, not encrypted.
• All credentials shown are placeholders.
• Real secrets are managed via the 1Password Kubernetes Operator.

If you plan to use similar patterns in production, additional measures such as RBAC minimization, NetworkPolicies, secret encryption at rest, and private cluster access are strongly recommended.

Services

As mentioned earlier, I had several different VPSs and other pieces of infrastructure. For example, I was running InvoicePlane, Shlink, and more (services to be added later). My goal was to migrate all of them to the new Kubernetes cluster.

For services like Shlink, where the UI is not protected by default, I wanted to add an authentication layer in front of them. Because of that, I decided to use Zitadel. I had already used it in other projects and was always very satisfied with it.

I then started writing my own Helm charts where no official or community-maintained ones existed. The first service I deployed on the cluster was the NGINX Ingress Controller, which was quite simple (see the example below). After that, I installed cert-manager, which was also easy to deploy. Then I created a custom resource to define two ClusterIssuers for issuing TLS certificates.

values.yaml for the ingress-nginx Helm chart

1
2
3
4
5
6
7
8

controller:
  admissionWebhooks:
    enabled: true
    patch:
      enabled: true

  service:
    externalTrafficPolicy: Local

values.yaml for the cert-manager Helm chart

1
2
3
4
5
6
7

cert-manager:
  enabled: true
  crds:
    enabled: true

issuer:
  email: hi@mydomain.com

ClusterIssuer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: hi@mydomain.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: hi@mydomain.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx

After having those two issuers in place, I wanted to deploy the Kubernetes Dashboard. Since I had already deployed it on my old DigitalOcean cluster, I remembered it as a single pod (or at least it was back then). This time, however, I ended up with several pods, proxies, and additional components.

I initially tried to simplify the deployment because I wanted to keep it as minimal as possible. After spending several hours on this, I decided to use the default setup instead—which, in most cases, is the better approach anyway. And who would have thought: it worked perfectly.

I opened my browser, navigated to the Kubernetes Dashboard, typed thisisunsafe to bypass Chrome’s SSL warning, and there it was—my dashboard. I then noticed that I could no longer log in using my kubeconfig, as I was able to do with the old dashboard. Instead, I created a service account and issued a token for authentication.

Typing thisisunsafe should only be done on trusted networks, as it bypasses important security warnings. In a production environment, you should always issue a valid TLS certificate for the dashboard to avoid these warnings entirely.

Apply the following YAML to create a service account with the cluster-admin role:

Warning: Granting cluster-admin permissions to a service account gives it full access to the entire cluster. Make sure you understand the security implications before proceeding. In production environments, more restrictive roles are strongly recommended.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-ds-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-ds-admin-role-binding
subjects:
  - kind: ServiceAccount
    name: kube-ds-admin
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Then generate a token for the service account:

Warning: The token generated below is valid for 1 hour. Adjust the --duration flag as needed for your use case. Avoid long-lived tokens in production environments due to increased security risks.

If you generate a long-lived token and want to revoke it later, you can do so by deleting the service account or the associated secret.

1

kubectl create token kube-ds-admin -n kube-system --duration=1h

Some time ago, I noticed that 1Password provides its own Kubernetes operator, which allows syncing specific 1Password items directly into a Kubernetes cluster. My curiosity and the security benefits convinced me to try it out.

During the setup, I ran into a very annoying issue. Everything seemed to be configured correctly, but the secret simply would not sync. After spending much more time on this than I want to admit, I finally realized that I had missed setting the operator.token.value in the Helm command. I had only attached the credentials.json file. Even though this is clearly documented in the 1Password documentation, I still managed to miss it.

Once this was fixed, everything worked as expected. I was able to sync credentials into Kubernetes. I created a dummy application that displayed the synced secret, and when I changed the credential in 1Password, it was automatically synced to Kubernetes and the pod restarted.

Before discovering the 1Password operator, my plan was to encrypt secrets using Sealed Secrets. That way, I could store the encrypted files directly in my repository.

Next, I had to figure out how to deploy probably the most important services of all: databases. Unfortunately, I need several of them—MySQL, PostgreSQL, and MongoDB.

I decided to start with PostgreSQL because I need it for Zitadel. My first choice was the Zalando Postgres Operator, but after installing it, I ran into issues with backups and was generally not convinced by its behavior. I therefore decided to move on and try the CloudNativePG (CNPG) operator.

At first, I was very happy with it—it worked well and felt clean. However, later on, I experienced some issues with the initDB and backup functionality. My main problem with initDB was that I did not really want it. I prefer to store the database custom resource inside the service directory. For example, the database for Zitadel should live in the Zitadel folder.

The reason for this is that I want to deploy everything using ArgoCD in the future. In my opinion, it makes the most sense to keep related resources in one directory. Unfortunately, I did not find a way to fully disable the initDB behavior.

At some point, I decided that it might be acceptable to leave the default behavior in place. Still, it felt a bit unclear to generate a secret and just leave it there without a proper backup—especially since the database would probably stay empty forever.

As a workaround, I added a 1Password custom resource to the database directory and synced the default initDB credentials from 1Password.

Note: Do not store secrets in your Git repository in plain text. Since the values are only base64-encoded, they can be easily decoded. Use a secret management solution like 1Password, HashiCorp Vault, or similar tools to manage secrets securely. Another option is to use Sealed Secrets to encrypt secrets before committing them to your repository.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# This is only the generated secret format expected by CNPG.
# In my setup, the secret is synced from 1Password to Kubernetes.

apiVersion: v1
kind: Secret
metadata:
  name: postgres-initdb-credentials
  namespace: my-database-cluster
data:
  username: bXktZGItdXNlcm5hbWU= # base64-encoded: `my-db-username`
  password: bXktc3VwZXItc2VjdXJlLWRiLXBhc3N3b3k= # base64-encoded: `my-super-secure-db-password`

• write about MongoDB and MySQL
• generalize and summarize the service deployment
• details about Zitadel will be covered in a separate article

Unfortunately, Pingvin Share is no longer maintained. Because of that, I’ve stopped using it and am now exploring other tools. Read post

I often find myself in situations where I want to share files with others, but I want something simpler than services like Google Drive or kDrive. So, I spent quite a bit of time looking for a solution and eventually discovered Pingvin Share.

I wanted to run it on my Kubernetes cluster, so I made my first “bigger” open-source contribution by adding S3 support. A bit later, I also added the option to configure it using a file. After that, I took a break for a while, since I had a lot of other things going on.

When I wanted to continue working on making Pingvin Share more Kubernetes-friendly, I saw that someone else had already added a key feature: Postgres support—since it had only used SQLite before. With that, all the pieces were in place to deploy it on Kubernetes. Unfortunately, the project lost its maintenance before this merge request was merged.

Services

Shlink

I use Shlink as a URL shortener. Occasionally, I need to share long links, and shortening them makes things easier. The Web UI is also protected by OAuth2-Proxy.

Zitadel

This is my preferred identity provider. I use it both for SaaS solutions and to secure my own services. It’s also the provider used for OAuth2-Proxy authentication.

My Website

Just a simple Nuxt-based website. You can visit it at mattiamueggler.ch.

My Blog

A small blog where I occasionally write posts about development, DevOps, or other tech topics.

My SaaS Solutions

Some applications I’ve built are also deployed on my Kubernetes cluster. These were actually the main reason I started setting up the cluster in the first place.

Homer Dashboards

Homer provides a dashboard that’s configured via YAML. I use it to list all my services with quick links. I’ve also built a custom image so I can include my own SVG icons.

WordPress Site

A small WordPress website I host for my sister.

Uptime-Kuma

Uptime-Kuma helps me monitor the availability of my services. I’ve added all my services to it and created dashboards to quickly check their status.

Fluent Bit

After one of my nodes crashed due to disk pressure, I looked for a better logging solution. I decided to use Fluent Bit to send logs to an S3 bucket and clean them up from the nodes to save space. I also added two new nodes with more disk capacity.

Kube-Prometheus Stack

To monitor not just the services, but also the nodes and the cluster itself, I set up the kube-prometheus-stack and added some dashboards. All services are only available internally within the cluster, and the Grafana dashboard is protected by OAuth2-Proxy.

Databases

Since many services require a database, I’ve deployed several kinds—MongoDB, PostgreSQL, and MariaDB. To save resources, I run each database as part of a shared cluster, rather than setting up one per service. I isolate access by creating individual users for each database.

The only exception is MariaDB: since I can’t easily add more databases via CRs, I created my own Helm chart on top of the operator. This allows me to manage everything as dependencies, with the correct configuration out of the box.

Additional Services

I also run a few additional services that I only need occasionally.

Additional Infrastructure

Kubernetes Dashboard

To interact with my cluster, I’ve installed the Kubernetes Dashboard and exposed it behind OAuth2-Proxy.

OAuth2-Proxy

This is a key part of my security setup. Any service that doesn’t need to be publicly accessible is protected with OAuth2-Proxy. This gives me secure access to internal tools. I might replace it with Tailscale at some point, since that would be even more secure.

NGINX Ingress Controller

This is the ingress controller I use to expose my services.

Cert Manager

I use Cert Manager so I don’t have to manually manage TLS certificates. I’ve set up two cluster issuers with Let’s Encrypt—one for staging and one for production—because Let’s Encrypt limits the number of requests to the production API.

ArgoCD

I use ArgoCD for GitOps. All my services are defined in a Git repository, and ArgoCD ensures that the cluster state is always in sync with the repository.

1Password Operator

This operator is great for syncing secrets, credentials, and passwords directly to my cluster. That way, I don’t have to encrypt them (e.g., with Sealed Secrets) to store them in GitHub—they’re never stored there at all.

CNPG Operator

I use this operator to manage PostgreSQL instances, including user and database provisioning.

MariaDB Operator

This one handles my MariaDB setup, including backups.

MongoDB Operator

Used to deploy and manage MongoDB.