Kubernetes details

Connecting to Kubernetes

There is more than one way to connect to a Kubernetes cluster.

During development, you will likely be best off using an existing kubectl configuration. In a production setting, you might prefer using a service account and token-based authentication.

Using kubectl configuration

During development, when working from a local development workstation, the easiest way to connect to a cluster is probably to use existing kubectl configuration:

from kubernetes import client, config

# This will initialize the client from an existing Kubectl config file in $HOME/.kube/config 
config.load_kube_config()

k8s_client = client.ApiClient()

Using a service account and token-based authentication

In a production setting, when the kubernetes_job.JobManager is run on the Kubernetes cluster, it is probably best to use a Kubernetes service account and a bearer token. This can be done as follows:

from kubernetes import client

configuration = client.Configuration()
configuration.api_key["authorization"] = '<token>'
configuration.api_key_prefix['authorization'] = 'Bearer'
configuration.host = 'https://<endpoint_of_api_server>'
configuration.ssl_ca_cert = '<path_to_cluster_ca_certificate>'

k8s_client = client.ApiClient(configuration)

How the correct settings for token, endpoint_of_api_server, and the cluster CA certificates can be retrieved is explained in the section below.

Please refer to Python Kubernetes documentation for more details.

The Kubernetes job spec template (e.g. job.yaml)

When Kubernetes-job spawns a new job, the Kubernetes job spec template is used as the base configuration for the new job.

This is an example:

apiVersion: batch/v1
kind: Job
metadata:
  # job name; a unique id will be added when launching a new job based on this template
  name: kubernetes-job
spec:

  # Try 1 time to execute this job
  backoffLimit: 1

  # Active deadline (timeout), in a number of seconds.
  activeDeadlineSeconds: 3600

  # Clean up pods and logs after finishing the job
  ttlSecondsAfterFinished: 3600

  template:
    spec:
      containers:
      - name: kubernetes-job
        image: registry.gitlab.com/roemer/kubernetes-job:latest
      restartPolicy: Never

Please adjust this template to your needs by specifying the right container image, job deadlines, etc. The Kubernetes documentation contains more information.

When Kubernetes-job spawns a new job, three things are added to the template:

1. A unique name, generated by adding a timestamp;

2. The function call, serialized (using Pickle), added as an environment variable;

3. A cmd entry calling JobManager.execute.

A working example can be found in the test/ directory.

Make sure the Docker image in the job template contains the same packaged Python software as the process creating the job! Otherwise the function cannot be executed in the new job pod.

Setting up token-based authentication

Create a service account

First, create a service account:

# Create a service account
kubectl create -f service_account.yml --k8s_namespace=default

An example of service_account.yml can be found here

Kubernetes generates a unique name for the new service account. We need to retrieve that unique name, and to do that, we need to ask Kubernetes for its secrets:

# retrieve secret 
kubectl get secrets --k8s_namespace=default | grep kubernetes-job-service-account

This returns something like this:

kubernetes-job-service-account-token-XXXXX   kubernetes.io/service-account-token   3      66s

kubernetes-job-service-account-token-XXXXX is the name generated by Kubernetes.

Retrieving the access token

Now we are able to retrieve the access token for this service account:

kubectl describe secret/kubernetes-job-service-account-token-XXXXX | grep token

This returns something like:

token:      <token>

This token is the one we’re looking for.

Cluster endpoint and cluster CA certificates

To connect to the cluster we also need the cluster endpoint and the CA certificates. Both can easily be retrieved through the Kubernetes dashboard, through the “cluster details” page.