Tailscale is a VPN service that makes it easy to create secure networks and connections between devices. The main reason I want to use Tailscale is that I need a secure and private way to access services that should not be publicly accessible. Tailscale allows me to create a virtual private network (VPN) where my devices and services can communicate securely over the internet, without exposing them to the public.

How to create your own Tailscale account with OpenId Connect (OIDC)?

This use case has already been described in my previous blog post How to set up a Tailscale account with OIDC?.

How to deploy Tailscale Operator in Kubernetes?

I decided to deploy the Tailscale Operator which manages the Tailscale Custom Resource Definitions (CRDs) in my Kubernetes cluster. For the deployment, I used the Helm chart provided by Tailscale. However, I created it as a rainbow chart to include my custom values like syncing the credentials from my 1Password vault. Here is an example of my values.yaml file:

Chart.yaml

1
2
3
4
5
6
7
8
9

apiVersion: v2
name: tailscale
version: 1.0.0

dependencies:
  - name: tailscale-operator
    version: 1.92.5
    repository: https://pkgs.tailscale.com/helmcharts
    alias: tso

values.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

tailscaleOperator:
  oauth:
    onePasswordItemPath: "vaults/<my-vault-uuid>/items/<my-item-uuid>"
    restartOnChange: true
tso:
  operatorConfig:
    image:
      repository: ghcr.io/tailscale/k8s-operator
      # tag: "1.92.5" # will be set by the dependency
      pullPolicy: IfNotPresent
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
      requests:
        cpu: 50m
        memory: 64Mi

Update your access control settings in Tailscale

To allow the Tailscale Operator to join your Tailscale network, you need to update your access control settings in Tailscale. Navigate to the Tailscale Admin Console - Tags section. Add a new tag called tag:k8s-operator and allow devices with this tag to join your network.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  ...
  "tagOwners": {
    "tag:k8s-operator": [],
    "tag:k8s": [
      "tag:k8s-operator"
    ]
  },
  ...
}

How to get the OAuth credentials?

Open Tailscale and navigate to the “Settings” page. Under the “Trust credentials” section, create new credentials. Choose “OAuth” as the type and enter “Kubernetes Operator” as the description.

As you can see, I set the onePasswordItemPath to sync the OAuth credentials from my 1Password vault. This way, I can securely manage my Tailscale OAuth credentials without hardcoding them in the Helm values. Therefore, I want that the operator reads the credentials from a secret which is synced from 1Password. My secret looks like this:

Consider that the secret must be called operator-oauth and must be created in the same namespace where the Tailscale. Also, the keys must be named client_id and client_secret. Read more

1
2
3
4
5
6
7
8
9

apiVersion: v1
data:
  client_id: <base-64-encoded-client-id>
  client_secret: <base-64-encoded-client-secret>
kind: Secret
metadata:
  name: operator-oauth
  namespace: tailscale
type: Opaque

Granting additional permissions to the Tailscale Operator

One small note, you need to give the Tailscale operator a bit more permissions to create the necessary resources. Therefore, I also added a namespace resource in my templates folder of the rainbow chart:

I use argo-cd hooks to ensure that the namespace is created before any other resources are applied. If you are using helm, you can use helm hooks instead.

1
2
3
4
5
6
7
8

apiVersion: v1
kind: Namespace
metadata:
  name: tailscale
  annotations:
    argocd.argoproj.io/hook: PreSync # ensures the namespace is created before other resources are applied
  labels:
    pod-security.kubernetes.io/enforce: privileged # adds the privileged policy to the namespace

After creating the rainbow chart, I deployed it to my Kubernetes cluster using Argo CD.

How to access Kubernetes applications through Tailscale?

To access Kubernetes applications through Tailscale, I used the Tailscale Ingress Controller. This controller allows me to expose my Kubernetes services to the Tailscale network securely. Here is an example of how I configured the Ingress Controller:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  defaultBackend:
    service:
      name: my-service
      port:
        number: 8000
  ingressClassName: tailscale # ensures that the Tailscale Ingress Controller handles this ingress
  rules:
    - host: myservice.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  number: 8000
  tls:
    - hosts:
        - myservice.example.com
      secretName: my-service-tls

Accessing the application through Tailscale

To access my Kubernetes application through Tailscale, I first need to ensure that my Tailscale client is running on my client. Then, I have to create a CNAME DNS record that points to the Tailscale IP address of my Kubernetes cluster. This way, when I access myservice.example.com, the request is routed through Tailscale to my Kubernetes cluster. Get the Tailscale DNS name of your Kubernetes cluster from the Tailscale Admin Console under Machines. Open the entry for your service and search for Full domain. Copy the value and create the CNAME record in your DNS provider:

1
2
3
4

CNAME myservice.example.com -> <tailscale-dns-name-of-k8s-cluster>

# Example:
CNAME myservice.example.com -> my-service.tail12345.ts.net

Consider that DNS propagation might take some time depending on your DNS provider. You can use tools like dig to check if the DNS record has propagated:

1

dig +short myservice.example.com

And that’s it! Now I can access my Kubernetes applications securely through Tailscale.

Conclusion

In this blog post, I explained how I set up Tailscale in my Kubernetes cluster using the Tailscale Operator and how I access my Kubernetes applications through Tailscale using the Tailscale Ingress Controller. Tailscale provides a secure and private way to connect to services without exposing them to the public internet. I hope this guide helps you set up Tailscale in your own Kubernetes cluster! Thanks to Josh Noll for his great guide on this topic.

Additional resources

• Josh Nolls’ Guide

Tailscale is a VPN service that makes it easy to create secure networks and connections between devices. The main reason I want to use Tailscale is that I need a secure and private way to access services that should not be publicly accessible. Tailscale allows me to create a virtual private network (VPN) where my devices and services can communicate securely over the internet, without exposing them to the public.

Why I want to use ZITADEL as an OIDC provider

ZITADEL is an IdP that provides identity and access management. It is already my main IdP for other services, so it makes sense to use it for Tailscale as well. By using ZITADEL as my OIDC provider, I can reuse my existing user accounts and authentication mechanisms, which simplifies the overall management of my Tailscale network.

Why I created this guide

Even though setting up Tailscale with an OIDC provider is quite well documented, I still faced some difficulties during the configuration process. Therefore, I decided to create this guide to help others who might face similar challenges when setting up Tailscale with ZITADEL as their OIDC provider.

First, some background on how this works: Tailscale loads the OIDC configuration using WebFinger, based on the domain of the email address you use to log in. For example, if I want to use my own email account name@example.com to authenticate with Tailscale, Tailscale will look for the OIDC configuration at:

https://example.com/.well-known/webfinger

This endpoint must return the following JSON:

1
2
3
4
5
6
7
8
9

{
  "subject": "acct:name@example.com",
  "links": [
    {
      "rel": "http://openid.net/specs/connect/1.0/issuer",
      "href": "https://account.example.com"
    }
  ]
}

The href field points to the issuer URL of the OIDC provider. You can find this value in the OpenID configuration endpoint:

https://account.example.com/.well-known/openid-configuration

Also note that the rel field is a fixed value defined by the OpenID Connect specification.

Using the WebFinger of my main domain

Since I already have a website running at example.com, but did not want to add the WebFinger file to my main website, I tried to set up a dedicated WebFinger server. However, there are only a few WebFinger server implementations available, and none of them worked well for my use case. Most of them are no longer actively maintained, or cannot easily be deployed on Kubernetes.

For example, I tried https://github.com/peeley/carpal, which seemed to be one of the more popular and maintained projects. However, it defines the WebFinger subject (such as acct:name@example.com) directly in the filename. Kubernetes does not allow mounting files with special characters like : and @, which made this approach unusable for me. I also did not want to introduce an external database just for running a WebFinger server.

In the end, I decided to use a static Caddy server and simply serve a static JSON file.

Caddyfile - ConfigMap

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: v1
kind: ConfigMap
metadata:
  name: caddy-config
data:
  Caddyfile: |
    :{{ .Values.service.port | default 8080 }} {
      root * /srv
      try_files {path} /.well-known/webfinger
      file_server

      header {
        Content-Type application/jrd+json
      }
    }

webfinger.json - ConfigMap

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: v1
kind: ConfigMap
metadata:
  name: webfinger-resource
data:
  webfinger.json: |
    {
      "subject": "acct:name@example.com",
      "links": [
        {
          "rel": "http://openid.net/specs/connect/1.0/issuer",
          "href": "https://account.example.com"
        }
      ]
    }

Deployment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webfinger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webfinger
  template:
    metadata:
      labels:
        app: webfinger
    spec:
      containers:
        - name: caddy
          image: caddy:alpine
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 15m
              memory: 10Mi
            limits:
              memory: 16Mi
          volumeMounts:
            # Caddy config
            - name: caddy-config
              mountPath: /etc/caddy/Caddyfile
              subPath: Caddyfile

            # WebFinger JSON served as a FILE
            - name: webfinger-resource
              mountPath: /srv/.well-known/webfinger
              subPath: webfinger.json

      volumes:
        - name: caddy-config
          configMap:
            name: caddy-config

        - name: webfinger-resource
          configMap:
            name: webfinger-resource

Service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: Service
metadata:
  name: webfinger-service
spec:
  selector:
    app: webfinger
  ports:
    - name: http
      port: 8080
      targetPort: 8080

Ingress

You can keep your other ingress for the main website and just add a new path for the webfinger service. However, both need to be accessible on the same host.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "webfinger-path"
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/use-regex: "true" # required to mach the guest path
    kubernetes.io/ingress.class: 'nginx'
    cert-manager.io/cluster-issuer: 'letsencrypt-prod'
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /.well-known
            pathType: ImplementationSpecific
            backend:
              service:
                name: webfinger-service
                port:
                  number: 8080
  tls:
    - hosts:
        - example.com
      secretName: webfinger-service-tls

Configure ZITADEL

Create a new ZITADEL application for Tailscale and make sure that the user you want to use for login has access to this application. It is important that the email address or username matches the subject you configured in the WebFinger setup.

Create a Web Application of type Code and add the following Redirect URI:

https://login.tailscale.com/a/oauth_response

Configure Tailscale

In Tailscale, go to the sign-up page and choose OIDC as the authentication method. Enter the email address associated with your ZITADEL account (for example, name@example.com). Tailscale will use WebFinger to discover the OIDC provider and then redirect you to ZITADEL for authentication.

After a successful login, you will be prompted to enter the clientId and clientSecret of the ZITADEL application you created earlier. Once these credentials are entered, you should be redirected back to Tailscale and logged in successfully.

How to verify that everything works

To verify that everything is working correctly, you can open an incognito or private browsing window in your web browser and try to log in to Tailscale using the same email address.

Keep in mind that you must first enter your email address on the Tailscale sign-in page. After that, you will be redirected to ZITADEL for authentication.

Conclusion

Setting up Tailscale with ZITADEL as a OIDC provider needs some tweaks, especially when it comes to the WebFinger server. However, this only lets you create a basic Tailscale account. If you want to protect your services with Tailscale, you will need to add Tailscale in front of your services, which is a topic for another guide.

Additional resources

• Tailscale OIDC documentation