Interact from Terraform with Helm releases via FluxCD

Problem description

When managing Kubernetes clusters via Terraform you often need to get some piece of information to your workload definitions (e.g. deployment) inside k8s. Think of an IP-address, allocated via Terraform, that should be used by your ingress controller of choice (Ambassador for this example). How would someone do that in an automated way?

Using FluxCD with Helm charts

The lovely FluxCD by the kind folks of weaveworks is a tool to manage your k8s workloads in the GitOps way, a term also coined by weaveworks. If you don’t know what it means, stop reading this and go read up, for example here.

One really nice feature of Flux is the Helm operator, which can manage Helm charts (assuming that you know what Helm is and how it works, else read up on that too!). To do so, it defines a CRD (Custom Resource Definition, probably want to read up on that too) in the cluster, the HelmRelease. This HelmRelease installs a chart with values that can come from two different destinations: defined inside the CRD itself or, and this is where it gets interesting, from a ConfigMap. As you might know, Terraform has a kubernetes provider, which, among other things, can manage ConfigMaps.

So the course is clear:

1. Create a needed resource (e.g. IP address) with terraform
2. Create a ConfigMap holding the value in a format that can be used by the Helm chart
3. Install the Helm chart via FluxCD and HelmRelease and reference the ConfigMap.
4. ???
5. profit

Minimal example

Lets say you have a terraform setup which is configured to talk to GCP and a k8s cluster. Then create the required resource, e.g. the static IP that the ingress controller should use:

resource "google_compute_address" "ingress" {
  name = "ingress"
}

Easy enough! Go on creating the ConfigMap. You should use some convention to naming it, since you have to refenrece it later from your k8s manifest. I’m using a $RELEASE_NAME-values convention here:

resource "kubernetes_config_map" "ambassador_values" {
  metadata {
    name      = "ambassador-values"
  }

  data = {
    "values.yaml" = <<-YAML
    service:
      loadBalancerIP: ${google_compute_address.ingress.address}
    YAML
  }
}

On to step three, assuming you have a working FluxCD setup. Create the following HelmRelease referencing the just created ConfigMap:

apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
metadata:
  name: ambassador
spec:
  releaseName: ambassador
  chart:
    repository: https://kubernetes-charts.storage.googleapis.com/
    name: ambassador
    version: 4.4.0
  valuesFrom:
    - configMapKeyRef:
        name: ambassador-values
  values:
    image:
      repository: quay.io/datawire/ambassador
      tag: 0.84.1
      pullPolicy: IfNotPresent
    podDisruptionBudget:
      minAvailable: "50%"

As you can see, you can add additional settings under the values key to the HelmRelease and they will be merged into the values from the ConfigMap. This is very useful, since e.g. the podDisruptionBudget has nothing to do with your infrastructure setup that is managed by terraform and therefore shouldn’t be configured from there.

If for some reason you change the IP address in the future via terraform, the change would be mirrored into the ConfigMap and the Helm operator would pick up this change the next time it reconciles the state of the relase, updating your ingress controller to use the new IP address.