A few weeks ago I had my first opportunity to work with Helm. As part of my assignment, I had to adjust some parts of Helm templates. It looked like a relatively small task, so I decided to rely on a combination of using existing templates as a reference and reading documentation just in time. A common practice in days of an abundance of tools, I believe.

If you have a lot of experience using Helm, you may sum up observations in the rest of this post with a sigh of “of course it works like that, it’s Helm 101”. Yet, I think it’s valuable to gather this kind of outsider feedback because insiders already know all gotchas and tend not to notice them anymore. And, contrary to what some people claim, I don’t think gotchas are facts of nature but quite often stem from the wrong underlying model.

I could have limited myself to writing a few sentences of conclusions and the post wouldn’t lose too much of its purely technical substance but providing some narrative enables you to see how I came to those conclusions.

Use case

In short - I had to create a duplicate of some preexisting Helm-defined job. That duplicate was supposed to have a slightly different configuration from the original one.

Let’s say the original configuration looked like this (a fragment of values.yaml):

application_config:
  host: "localhost"
  port: 1234

In the new job I wanted to use the same application_config with port being overridden to 9876. After a quick lookup I found out mergeOverwrite. Therefore, my first attempt to define ConfigMaps was as follows (templates/configmaps.yaml):

{{ /* The new job's configuration with slighly modified `data` */ }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: port_overriden
data: {{ (mergeOverwrite .Values.application_config (dict "port" 9876)) | toYaml | nindent 2 }}
---
{{ /* The original job's configuration */ }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: original
data: {{ .Values.application_config | toYaml | nindent 2 }}

Problem 1: mergeOverwrite mutates source dictionary

The documentation of mergeOverwrite mentions that:

Nested objects that are merged are the same instance on both dicts. If you want a deep copy along with the merge than use the deepCopy function along with merging.

Since it was my first time using Helm I was not sure if I understood its terminology correctly. Seeing “deep copy” being mentioned I got a vague feeling that Helm allows mutability. To determine if that’s true I ran helm template . which yielded the following result:

# Source: mychart/templates/configmaps.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: port_overriden
data:
  host: localhost
  port: 9876 # That's fine
---
# Source: mychart/templates/configmaps.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: original
data:
  host: localhost
  port: 9876 # I intended it to stay 1234!

The result speaks for itself - mergeOverwrite overridden the original dictionary! I can easily imagine someone not checking the documentation and expecting mergeOverwite to create a new dictionary with the chosen values overridden. Apparently I was not the first one to find this behavior problematic - there is a github issue for that. I must admit that things improved because the sentence I quoted before has been added as resolution and helped me spot the problem right away.

What puzzles me the most here is not mergeOverwrite itself but the very fact that configuration language allows mutability at all. I cannot imagine any use case in which you need to model mutability using configuration language. It stands in contrast to general-purpose languages - most of them have to support mutability in some way because people are supposed to write long-running, stateful programs in them.

However, Helm is clearly not a general-purpose programming language in which you would write a web scraper or web service. helm template should just resolve templates, which is a single atomic operation with no notion of time. There’s only input and output.

Introducing mutability into templating brings a whole class of issues without solving any problem on its own. Have I mentioned that in the example above, if I change the ordering of ConfigMaps so original comes before port_overriden, then I would get the intended result? The thing is that I don’t want semantics of resolution depends on the ordering of fragments. It’s something that Terraform gets right - it builds a graph of dependencies so you don’t have to worry about the ordering of things.

Knowing that problem, we can fix it easily - add deepCopy like this:

data: {{ (mergeOverwrite (deepCopy .Values.application_config) (dict "port" 1235)) | toYaml | nindent 2 }}

Problem 2: ternary evaluates parameters greedily

To appreciate what a Pandora’s box mutability opens, let’s take a look at ternary. This function does what the ternary operator does in many languages - picks one of two expressions depending on a predicate.

Let’s imagine we want to override port in the configuration based on some predicate:

metadata:
  name: port_overriden
data: {{ ternary ((mergeOverwrite .Values.application_config (dict "port" 9876)) | toYaml | nindent 2) .Values.application_config predicate }}

Which overrides port to 9876 if predicate is true and uses original application_config otherwise. How about side effects though? Will application_config be touched in case predicate is false? It took me one helm template run to understand that ternary evaluates its parameters greedily. It means that no matter of predicate value the original application_config will be overriden. In this particular case it contradicts the whole point of using ternary because port will be set to 9876 regardless of the predicate.

Again, I am not assessing the specific design of Helm here but trying to make a larger point. I don’t even know Go, in which Helm is written, so I cannot tell if lazy evaluation would be actually possible here.

A few other problems

Since Helm templates are embedded into YAML it means that whitespaces are significant. It introduces additional complexity, especially with conditionals - you need to be very careful with distinguishing between {{ vs {{-.

Another curious Helm limitation is that conditionals and local variables do not easily compose.

How does it compare to Dhall?

Dhall is a configuration language. You can think of it as a better YAML. It offers variables, functions and types and it embraces immutablity. I’ve written about Dhall long time ago so I am not going to dwell on it too much here but I want to point out that none of the problems described in this post exist in Dhall.

I am aware that comparing Helm to Dhall is not comparing apples to apples. What I am doing here is rather showing very concrete problems with Helm that would not appear in an hypothetical, yet-to-be-written tool based on Dhall. And I am not making this problems up - I stumbled upon all of them just in a few hours of work.