Kubernetes End-to-End Testing Using Testcontainers and Python

Testing Kubernetes applications often involves dealing with multiple dependencies and a complex environment. While end-to-end (E2E) testing provides a realistic way to validate application behavior, replicating production environments can be challenging. With Testcontainers, you can simplify this process by using lightweight, disposable containers to emulate a Kubernetes cluster and its dependencies.

You can use the testcontainers-python library to perform end-to-end testing for a Kubernetes application.

What Is Testcontainers?

Testcontainers is an open source library that supports running lightweight, disposable containers for testing. With the addition of Kubernetes-focused modules like testcontainers-k3s, you can spin up a Kubernetes cluster as part of your test setup.

Using Testcontainers for Kubernetes testing offers:

1. Realistic testing environments: Emulate Kubernetes clusters and services within isolated containers.
2. Automation: Automate the setup and teardown of dependencies like databases, message brokers or Kubernetes.
3. Efficiency: Run tests in a clean, repeatable environment without manually configuring Kubernetes clusters.
4. Dynamic configurations: Customize dependencies on the fly for each test scenario.

Set Up Testcontainers for Python

Prerequisites

1. Python 3.8 or higher: Install the latest version.
2. Docker: Ensure Docker is installed and running.
3. Install Testcontainers: Use pip to install the library.

pip install testcontainers
pip install kubernetes

Run an E2E Test for a Kubernetes Application

In this step-by-step example, I’ll show you how to test a Kubernetes application that interacts with a PostgreSQL database. The test will:

1. Spin up a Kubernetes cluster using testcontainers-k3s.
2. Deploy the application and database.
3. Validate the application’s behavior through HTTP requests.

1. Create a Python Test Class

‍

The following script sets up a Kubernetes cluster using K3sContainer from Testcontainers:

‍

from testcontainers.k3s import K3sContainer
from testcontainers.postgres import PostgresContainer
import requests
import time
import os


def wait_for_pod_ready(kubeconfig_path, namespace="default", timeout=120):
    """
    Wait for all pods in the namespace to be ready.
    """
    from kubernetes import client, config

    config.load_kube_config(config_file=kubeconfig_path)
    v1 = client.CoreV1Api()

    end_time = time.time() + timeout
    while time.time() < end_time:
        pods = v1.list_namespaced_pod(namespace=namespace)
        if all(p.status.phase == "Running" for p in pods.items):
            return True
        time.sleep(5)
    raise TimeoutError("Timeout waiting for pods to be ready")


def test_kubernetes_app():
    # Spin up PostgreSQL Container
    with PostgresContainer("postgres:15.0") as postgres:
      postgres_host = postgres.get_container_host_ip()
        postgres_port = postgres.get_exposed_port(5432)

        # Set up Kubernetes cluster
        with K3sContainer() as k3s:
            kubeconfig_path = k3s.get_kube_config()
            os.environ["KUBECONFIG"] = kubeconfig_path

            # Apply Kubernetes manifests
            os.system(f"kubectl apply -f kubernetes/app.yaml")
            os.system(f"kubectl apply -f kubernetes/postgres.yaml")

            # Wait for all pods to be ready
            wait_for_pod_ready(kubeconfig_path)

            # Send an HTTP request to the application
            response = requests.get("http://localhost:8080/health")
            assert response.status_code == 200
            assert "healthy" in response.text

‍

2. Create Kubernetes Manifests

‍

Create Kubernetes manifests for the application and PostgreSQL database.

‍

app.yaml

‍

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-app-image:latest
        ports:
        - containerPort: 8080
        env:
        - name: DATABASE_URL
          value: "postgresql://user:password@postgres:5432/mydb"

---
apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  type: NodePort
  selector:
    app: my-app
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

‍

postgres.yaml

‍

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15.0
        env:
        - name: POSTGRES_USER
          value: "user"
        - name: POSTGRES_PASSWORD
          value: "password"
        - name: POSTGRES_DB
          value: "mydb"

---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  ports:
  - port: 5432
  selector:
    app: postgres

‍

3. Run the Test

‍

Execute the test using your preferred Python test runner, such as Pytest:

‍

pytest test_kubernetes_app.py

‍

This test will:

1. Start PostgreSQL in a Testcontainers instance.
2. Spin up a lightweight Kubernetes cluster.
3. Deploy the application and database.
4. Verify the application is reachable and functioning as expected.

Best Practices for Kubernetes E2E Testing

1. Resource management: Ensure your system has enough resources to run containers and the Kubernetes cluster.
2. Namespace isolation: Use separate namespaces for each test to avoid conflicts.
3. Mock external APIs: Mock dependencies that are not directly under test for reliability.
4. Parallelization: Optimize test execution by running independent test cases in parallel.

Conclusion

Due to its distributed nature, end-to-end testing in Kubernetes can be intimidating; however, Testcontainers makes the process easier. By combining the testcontainers-k3s module with Python, you can create isolated, reproducible and production-like environments for thorough testing. This ensures your application behaves as expected in Kubernetes before it goes live.

Adopting this approach lets you catch critical issues earlier in the pipeline and deliver more robust software. Why not give Testcontainers a try and enhance your Kubernetes testing strategy?

About the Author: Manish Saini

Manish is an Andela Technologist and a Senior SDET Consultant with over 9 years of experience. He specializes in empowering testing teams to achieve faster delivery and exceptional quality through automation solutions using Python, JavaScript, and a variety of tools. Manish helps aspiring professionals succeed in software development and quality assurance by using his skills and experience to guide them.

‍