Developing Cluster API with Tilt

Overview

This document describes how to use kind and Tilt for a simplified workflow that offers easy deployments and rapid iterative builds.

Prerequisites

Docker
kind v0.6 or newer (other clusters can be used if preload_images_for_kind is set to false)
kustomize standalone (kubectl kustomize does not work because it is missing some features of kustomize v3)
Tilt
Clone the Cluster API repository locally
Clone the provider(s) you want to deploy locally as well

Getting started

Create a kind cluster

First, make sure you have a kind cluster and that your KUBECONFIG is set up correctly:

$ kind create cluster

Create a tilt-settings.json file

Next, create a tilt-settings.json file and place it in your local copy of cluster-api. Here is an example:

{
  "allowed_contexts": ["kind-kind"],
  "default_registry": "gcr.io/your-project-name-here",
  "provider_repos": ["../cluster-api-provider-aws"],
  "enable_providers": ["aws", "docker"]
}

tilt-settings.json fields

allowed_contexts (Array, default=[]): A list of kubeconfig contexts Tilt is allowed to use. See the Tilt documentation on *allow_k8s_contexts for more details.

default_registry (String, default=””): The image registry to use if you need to push images. See the Tilt *documentation for more details.

provider_repos (Array[]String, default=[]): A list of paths to all the providers you want to use. Each provider must have a tilt-provider.json file describing how to build the provider.

enable_providers (Array[]String, default=[‘docker’]): A list of the providers to enable. See available providers for more details.

kustomize_substitutions (Map{String: String}, default={}): An optional map of substitutions for ${}-style placeholders in the provider’s yaml.

AWSGCP

For example, if the yaml contains ${AWS_B64ENCODED_CREDENTIALS}, you could do the following:

"kustomize_substitutions": {
  "AWS_B64ENCODED_CREDENTIALS": "your credentials here"
}

You can generate a base64 version of your GCP json credentials file using:

base64 -i ~/path/to/gcp/credentials.json

"kustomize_substitutions": {
  "GCP_B64ENCODED_CREDENTIALS": "your credentials here"
}

Run Tilt!

To launch your development environment, run

tilt up

This will open the command-line HUD as well as a web browser interface. You can monitor Tilt’s status in either location. After a brief amount of time, you should have a running development environment, and you should now be able to create a cluster. Please see the Usage section in the Quick Start for more information on creating workload clusters.

Available providers

The following providers are currently defined in the Tiltfile:

core: cluster-api itself (Cluster/Machine/MachineDeployment/MachineSet/KubeadmConfig/KubeadmControlPlane)
docker: Docker provider (DockerCluster/DockerMachine)

tilt-provider.json

A provider must supply a tilt-provider.json file describing how to build it. Here is an example:

{
    "name": "aws",
    "config": {
        "image": "gcr.io/k8s-staging-cluster-api-aws/cluster-api-aws-controller",
        "live_reload_deps": [
            "main.go", "go.mod", "go.sum", "api", "cmd", "controllers", "pkg"
        ]
    }
}

config fields

image: the image for this provider, as referenced in the kustomize files. This must match; otherwise, Tilt won’t build it.

live_reload_deps: a list of files/directories to watch. If any of them changes, Tilt rebuilds the manager binary for the provider and performs a live update of the running container.

additional_docker_helper_commands (String, default=””): Additional commands to be run in the helper image docker build. e.g.

RUN wget -qO- https://dl.k8s.io/v1.14.4/kubernetes-client-linux-amd64.tar.gz | tar xvz
RUN wget -qO- https://get.docker.com | sh

additional_docker_build_commands (String, default=””): Additional commands to be appended to the dockerfile. The manager image will use docker-slim, so to download files, use additional_helper_image_commands. e.g.

COPY --from=tilt-helper /usr/bin/docker /usr/bin/docker
COPY --from=tilt-helper /go/kubernetes/client/bin/kubectl /usr/bin/kubectl

Customizing Tilt

If you need to customize Tilt’s behavior, you can create files in cluster-api’s tilt.d directory. This file is ignored by git so you can be assured that any files you place here will never be checked in to source control.

These files are included after the providers map has been defined and after all the helper function definitions. This is immediately before the “real work” happens.

Under the covers, a.k.a “the real work”

At a high level, the Tiltfile performs the following actions:

Read tilt-settings.json
Configure the allowed Kubernetes contexts
Set the default registry
Define the providers map
Include user-defined Tilt files
Enable providers (core + what is listed in tilt-settings.json)
1. Build the manager binary locally as a local_resource
2. Invoke docker_build for the provider
3. Invoke kustomize for the provider’s config/default directory

Live updates

Each provider in the providers map has a live_reload_deps list. This defines the files and/or directories that Tilt should monitor for changes. When a dependency is modified, Tilt rebuilds the provider’s manager binary on your local machine, copies the binary to the running container, and executes a restart script. This is significantly faster than rebuilding the container image for each change. It also helps keep the size of each development image as small as possible (the container images do not need the entire go toolchain, source code, module dependencies, etc.).

The Cluster API Book