Frequently asked questions

Flux and the GitOps Toolkit frequently asked questions.

Kustomize questions

Are there two Kustomization types?

Yes, the kustomization.kustomize.toolkit.fluxcd.io is a Kubernetes custom resource while kustomization.kustomize.config.k8s.io is the type used to configure a Kustomize overlay.

The kustomization.kustomize.toolkit.fluxcd.io object refers to a kustomization.yaml file path inside a Git repository or Bucket source.

How do I use them together?

Assuming an app repository with ./deploy/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: default
resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml

Define a source of type gitrepository.source.toolkit.fluxcd.io that pulls changes from the app repository every 5 minutes inside the cluster:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: my-app
  namespace: default
spec:
  interval: 5m
  url: https://github.com/my-org/my-app
  ref:
    branch: main

Then define a kustomization.kustomize.toolkit.fluxcd.io that uses the kustomization.yaml from ./deploy/prod to determine which resources to create, update or delete:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: my-app
  namespace: default
spec:
  interval: 15m
  path: "./deploy/prod"
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-app

What is a Kustomization reconciliation?

In the above example, we pull changes from Git every 5 minutes, and a new commit will trigger a reconciliation of all the Kustomization objects using that source.

Depending on your configuration, a reconciliation can mean:

generating a kustomization.yaml file in the specified path
building the kustomize overlay
decrypting secrets
validating the manifests with client or server-side dry-run
applying changes on the cluster
health checking of deployed workloads
garbage collection of resources removed from Git
issuing events about the reconciliation result
recoding metrics about the reconciliation process

The 15 minutes reconciliation interval, is the interval at which you want to undo manual changes .e.g. kubectl set image deployment/my-app by reapplying the latest commit on the cluster.

Note that a reconciliation will override all fields of a Kubernetes object, that diverge from Git. For example, you’ll have to omit the spec.replicas field from your Deployments YAMLs if you are using a HorizontalPodAutoscaler that changes the replicas in-cluster.

Can I use repositories with plain YAMLs?

Yes, you can specify the path where the Kubernetes manifests are, and kustomize-controller will generate a kustomization.yaml if one doesn’t exist.

Assuming an app repository with the following structure:

├── deploy
│   └── prod
│       ├── .yamllint.yaml
│       ├── deployment.yaml
│       ├── service.yaml
│       └── ingress.yaml
└── src

Create a GitRepository definition and exclude all the files that are not Kubernetes manifests:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: my-app
  namespace: default
spec:
  interval: 5m
  url: https://github.com/my-org/my-app
  ref:
    branch: main
  ignore: |
    # exclude all
    /*
    # include deploy dir
    !/deploy
    # exclude non-Kubernetes YAMLs
    /deploy/**/.yamllint.yaml

Then create a Kustomization definition to reconcile the ./deploy/prod dir:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: my-app
  namespace: default
spec:
  interval: 15m
  path: "./deploy/prod"
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-app

With the above configuration, source-controller will pull the Kubernetes manifests from the app repository and kustomize-controller will generate a kustomization.yaml including all the resources found with ./deploy/prod/**/*.yaml.

The kustomize-controller creates kustomization.yaml files similar to:

cd ./deploy/prod && kustomize create --autodetect --recursive

What is the behavior of Kustomize used by Flux?

We referred to the Kustomize v4 CLI flags here, so that you can replicate the same behavior using kustomize build:

---enable-alpha-plugins is disabled by default, so it uses only the built-in plugins.
--load-restrictor is set to LoadRestrictionsNone, so it allows loading files outside the dir containing kustomization.yaml.
--reorder is set to legacy, so the output will have namespaces and cluster roles/role bindings first, CRDs before CRs, and webhooks last.

To replicate the build and apply dry run locally:

kustomize build --load-restrictor=LoadRestrictionsNone --reorder=legacy . \
| kubectl apply --server-side --dry-run=server -f-

kustomization.yaml validation

To validate changes before committing and/or merging, a validation utility script is available, it runs kustomize locally or in CI with the same set of flags as the controller and validates the output using kubeval.

How do I resolve `webhook does not support dry run` errors?

If you’ve installed Kubernetes dynamic admission controls you may see Flux failing to reconcile with an error similar to dry-run failed, error: admission webhook "validation-service.default.svc" does not support dry run.

To fix this issue, you’ll have to find the ValidatingWebhookConfiguration or the MutatingWebhookConfiguration, and set the sideEffects to None or NoneOnDryRun:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: my-webhook.example.com
  sideEffects: None

How do I resolve `some/resource configured` events spam?

If the controller emits change events for a specific resource (even if the resource hasn’t changed), you’ll need to edit your YAML manifests and remove any optional field that is set to null, empty string or empty object.

Example of empty fields that will trigger drift events:

apiVersion: v1
kind: Example
metadata:
  name: example
spec:
  field1: null
  field2: {}
  field3:

The kustomize-controller detects drift between the manifests and the in-cluster resources by running a server-side apply dry-run, removing the empty fields from your manifests will help the controller detect drift correctly.

How to patch CoreDNS and other pre-installed addons?

To patch a pre-installed addon like CoreDNS with customized content. Simply add a shell manifest with only the changed values and kustomize.toolkit.fluxcd.io/prune: disabled annotation into your git repository.

Example CoreDNS with custom replicas, the spec.containers[] empty list is needed for the patch to work and will not override the existing containers:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: kube-dns
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled
  name: coredns
  namespace: kube-system
spec:
  replicas: 5
  selector:
    matchLabels:
      eks.amazonaws.com/component: coredns
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        eks.amazonaws.com/component: coredns
        k8s-app: kube-dns
    spec:
      containers: []

Note that only non-managed fields should be modified else there will be a conflict with the manager of the fields (e.g. eks). For example, while you will be able to modify affinity/antiaffinity fields, the manager (e.g. eks) will revert those changes and that might not be immediately visible to you (with EKS that would be an interval of once every 30 minutes). The deployment will go into a rolling upgrade and Flux will revert it back to the patched version.

Helm questions

How to debug “not ready” errors?

Misconfiguring the HelmRelease.spec.chart, like a typo in the chart name, version or chart source URL would result in a “HelmChart is not ready” error displayed by:

$ flux get helmreleases --all-namespaces
NAMESPACE	NAME   	READY	MESSAGE
default  	podinfo	False 	HelmChart 'default/default-podinfo' is not ready

In order to get to the root cause, first make sure the source e.g. the HelmRepository is configured properly and has access to the remote index.yaml:

$ flux get sources helm --all-namespaces
NAMESPACE  	NAME   	READY	MESSAGE
default   	podinfo	False	failed to fetch https://stefanprodan.github.io/podinfo2/index.yaml : 404 Not Found

If the source is Ready, then the error must be caused by the chart, for example due to an invalid chart name or non-existing version:

$ flux get sources chart --all-namespaces
NAMESPACE  	NAME           	READY	MESSAGE
default  	default-podinfo	False	no chart version found for podinfo-9.0.0

Can I use Flux HelmReleases without GitOps?

Yes, you can install the Flux components directly on a cluster and manage Helm releases with kubectl.

Install the controllers needed for Helm operations with flux:

flux install \
--namespace=flux-system \
--network-policy=false \
--components=source-controller,helm-controller

Create a Helm release with kubectl:

cat << EOF | kubectl apply -f -
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  interval: 30m
  url: https://charts.bitnami.com/bitnami
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  interval: 60m
  releaseName: metrics-server
  chart:
    spec:
      chart: metrics-server
      version: "^5.x"
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  values:
    apiService:
      create: true
EOF

Based on the above definition, Flux will upgrade the release automatically when Bitnami publishes a new version of the metrics-server chart.

How do I resolve a `Request entity too large: limit is 3145728` error during Helm install or upgrade?

This error is returned by Helm when the release that is attempted to be made does not fit in a Secret. Most of the time this is due to exceptionally large (umbrella) charts, as explained in helm/helm#8281.

If you are running into this, confirm first that your chart has all the required excludes in their respective .helmignore and .sourceignore files.

Raspberry Pi questions

How do I resolve a `unable to open the Badger database` that puts `image-reflector-controller` in `CrashLoopBackOff`?

This error is commonly found in low-memory environments where an Out Of Memory condition is likely to be experienced, like on Raspberry Pi 2 and 3 boards which have only 1GB of RAM each.

If you are on Raspbian, the default size of 100MB swap file is not large enough to avoid the fault condition, and can be updated through configuration in /etc/dphys-swapfile: add at least 1GB swap.

Flux v1 vs v2 questions

What are the differences between v1 and v2?

Flux v1 is a monolithic do-it-all operator; Flux v2 separates the functionalities into specialized controllers, collectively called the GitOps Toolkit.

You can find a detailed comparison of Flux v1 and v2 features in the migration FAQ.

How can I migrate from v1 to v2?

The Flux community has created guides and example repositories to help you migrate to Flux v2:

Last modified 2021-10-27: Add server-side apply questions to FAQ (39018eb)