How to refresh an imagePullSecret token secret, the hard way

January 06, 2024

TL;DR this is a niche issue I encounter because I run a DigitalOcean k8s cluster with a private GitHub registry. If you’re using a public registry or ECR you may not run into those issues.

It’s end of year and deployments to your dev Kubernetes cluster started failing a few days ago. The CI likely passes, but nothing happens. When you check your pods, you directly see that new ones can’t pull the image. Fuck 🤬! You have flashbacks to when your docker registry bullied you into setting an expiry date, because: security, yay!

Screenshot 2024-01-06 at 12.29.01.png

Untitled

You likely had created the secret using the bespoke kubectl command but you forgot, because you do it once a year:

kubectl create secret docker-registry [SECRET_NAME] \
--docker-server=[PRIVATE_REGISTRY_SERVER] \
--docker-username=[REGISTRY_USER] \
--docker-password=[REGISTRY_PASSWORD] \
--docker-email=[REGISTRY_EMAIL] \
--namespace [NAMESPACE]

It turns out there is no “update the token” command, so you’ll either have to recreate it, or use the edit command, which is available in the dashboard as well, if that’s your kind of thing.

kubectl edit secret [SECRET_NAME] --namespace [NAMESPACE]

When you open the secret manifest that was generate with the create secret docker-registry command, you find something like this:

kind: Secret
apiVersion: v1
metadata:
  name: github-regcred
  namespace: default
	<redacted>
data:
  .dockerconfigjson: >-
    <base 64>
type: kubernetes.io/dockerconfigjson

Obviously, it’s the base64 encoded data in .dockerconfigjson that we’ll have to operate on. If we decode it, we can see a JSON of the following structure:

{
	"auths": {
		"ghcr.io": {
			"username":"alice",
			"password":"<token>",
			"email":"[email protected]",
			"auth": "base64 of alice:<token>"
		}
	}
}

As we can see, the decoded secret contains another auth field used by Docker to authenticate with the registry server.

To update the token, you’ll simply have to decode it, replace the token, and reencode it.

Here is a small script to do it:

import sys
import json
import base64
from subprocess import check_output, Popen, PIPE

def main():
    secret_name = sys.argv[1]
    new_token = sys.argv[2]

    # Fetching the secret
    secret = json.loads(check_output(["kubectl", "get", "secret", secret_name, "-o", "json"]))

    # Decoding the existing .dockerconfigjson field
    docker_config_json = base64.b64decode(secret['data']['.dockerconfigjson']).decode()
    config = json.loads(docker_config_json)

    # Assuming 'auths' structure exists and only one registry
    for registry in config['auths']:
        config['auths'][registry]['auth'] = base64.b64encode(f'{secret_name}:{new_token}'.encode()).decode()

    # Encoding the modified .dockerconfigjson field
    secret['data']['.dockerconfigjson'] = base64.b64encode(json.dumps(config).encode()).decode()

    # Preparing the updated secret as json
    updated_secret_json = json.dumps(secret)

    # Invoking kubectl edit
    process = Popen(["kubectl", "edit", "secret", secret_name, "-o", "json"], stdin=PIPE)
    process.communicate(input=updated_secret_json.encode())

if __name__ == "__main__":
    main()

There you have it, see you next year 😉