Using Flux on Azure

How to boostrap Flux on Azure AKS with DevOps Git repositories.

AKS Cluster Options

It’s important to follow some guidelines when installing Flux on AKS.

CNI and Network Policy

Previously, there has been an issue with Flux and Network Policy on AKS. ( Upstream Azure Issue) ( Flux Issue) If you ensure your AKS cluster is upgraded, and your Nodes have been restarted with the most recent Node images, this could resolve flux reconciliation failures where source-controller is unreachable. Using --network-plugin=azure --network-policy=calico has been tested to work properly. This issue only affects you if you are using --network-policy on AKS, which is not a default option.

AAD Pod-Identity

Depending on the features you are interested in using with Flux, you may want to install AAD Pod Identity. With AAD Pod-Identity, we can create Pods that have their own cloud credentials for accessing Azure services like Azure Container Registry(ACR) and Azure Key Vault(AKV).

If you do not use AAD Pod-Identity, you’ll need to manage and store Service Principal credentials in K8s Secrets, to integrate Flux with other Azure Services.

As a pre-requisite, your cluster must have --enable-managed-identity configured.

This software can be installed via Helm (unmanaged by Azure). Use Flux’s HelmRepository and HelmRelease object to manage the aad-pod-identity installation from a bootstrap repository and keep it up to date.

Cluster Creation

The following creates an AKS cluster with some minimal configuration that will work well with Flux:

az aks create \
 --network-plugin="azure" \
 --network-policy="calico" \
 --enable-managed-identity \
 --enable-pod-identity \
 --name="my-cluster"

Flux Installation for Azure DevOps

Ensure you can login to dev.azure.com for your proper organization, and create a new repository to hold your Flux install and other Kubernetes resources.

Clone the Git repository locally:

git clone ssh://git@ssh.dev.azure.com/v3/<org>/<project>/<my-repository>
cd my-repository

Create a directory inside the repository:

mkdir -p ./clusters/my-cluster/flux-system

Download the Flux CLI and generate the manifests with:

flux install \
  --export > ./clusters/my-cluster/flux-system/gotk-components.yaml

Commit and push the manifest to the master branch:

git add -A && git commit -m "add components" && git push

Apply the manifests on your cluster:

kubectl apply -f ./clusters/my-cluster/flux-system/gotk-components.yaml

Verify that the controllers have started:

flux check

Create a GitRepository object on your cluster by specifying the SSH address of your repo:

flux create source git flux-system \
  --git-implementation=libgit2 \
  --url=ssh://git@ssh.dev.azure.com/v3/<org>/<project>/<repository> \
  --branch=<branch> \
  --ssh-key-algorithm=rsa \
  --ssh-rsa-bits=4096 \
  --interval=1m

The above command will prompt you to add a deploy key to your repository, but Azure DevOps does not support repository or org-specific deploy keys. You may add the deploy key to a user’s personal SSH keys, but take note that revoking the user’s access to the repository will also revoke Flux’s access. The better alternative is to create a machine-user whose sole purpose is to store credentials for automation. Using a machine-user also has the benefit of being able to be read-only or restricted to specific repositories if this is needed.

If you wish to use Git over HTTPS, then generate a personal access token and supply it as the password:

flux create source git flux-system \
  --git-implementation=libgit2 \
  --url=https://dev.azure.com/<org>/<project>/_git/<repository> \
  --branch=main \
  --username=git \
  --password=${AZ_PAT_TOKEN} \
  --interval=1m

Please consult the Azure DevOps documentation on how to generate personal access tokens for Git repositories. Azure DevOps PAT’s always have an expiration date, so be sure to have some process for renewing or updating these tokens. Similar to the lack of repo-specific deploy keys, a user needs to generate a user-specific PAT. If you are using a machine-user, you can generate a PAT or simply use the machine-user’s password which does not expire.

Create a Kustomization object on your cluster:

flux create kustomization flux-system \
  --source=flux-system \
  --path="./clusters/my-cluster" \
  --prune=true \
  --interval=10m

Export both objects, generate a kustomization.yaml, commit and push the manifests to Git:

flux export source git flux-system \
  > ./clusters/my-cluster/flux-system/gotk-sync.yaml

flux export kustomization flux-system \
  >> ./clusters/my-cluster/flux-system/gotk-sync.yaml

cd ./clusters/my-cluster/flux-system && kustomize create --autodetect

git add -A && git commit -m "add sync manifests" && git push

Wait for Flux to reconcile your previous commit with:

watch flux get kustomization flux-system

Flux Upgrade

To upgrade the Flux components to a newer version, download the latest flux binary, run the install command in your repository root, commit and push the changes:

flux install \
  --export > ./clusters/my-cluster/flux-system/gotk-components.yaml

git add -A && git commit -m "Upgrade to $(flux -v)" && git push

The source-controller will pull the changes on the cluster, then kustomize-controller will perform a rolling update of all Flux components including itself.

Helm Repositories on Azure Container Registry

The Flux HelmRepository object currently supports Chart Repositories as well as fetching HelmCharts from paths in GitRepository sources.

Azure Container Registry has a sub-command ( az acr helm) for working with ACR-Hosted Chart Repositories, but it is deprecated. If you are using these deprecated Azure Chart Repositories, you can use Flux HelmRepository objects with them.

Newer ACR Helm documentation suggests using ACR as an experimental Helm OCI Registry. This will not work with Flux, because using Charts from OCI Registries is not yet supported.

Secrets Management with SOPS and Azure Key Vault

You will need to create an Azure Key Vault and bind a credential such as a Service Principal or Managed Identity to it. If you want to use Managed Identities, install or enable AAD Pod Identity.

Patch kustomize-controller with the proper Azure credentials, so that it may access your Azure Key Vault, and then begin committing SOPS encrypted files to the Git repository with the proper Azure Key Vault configuration.

See the Mozilla SOPS Azure Guide for further detail.

Image Updates with Azure Container Registry

You will need to create an ACR registry and bind a credential such as a Service Principal or Managed Identity to it. If you want to use Managed Identities, install or enable AAD Pod Identity.

You may need to update your Flux install to include additional components:

flux install \
  --components-extra="image-reflector-controller,image-automation-controller" \
  --export > ./clusters/my-cluster/flux-system/gotk-components.yaml

Follow the Image Update Automation Guide and see the ACR specific section for more details.

Your AKS cluster’s configuration can also be updated to allow the kubelets to pull images from ACR without ImagePullSecrets as an optional, complimentary step.

Azure Event Hub with Notification controller

The Notification Controller supports both JWT and SAS based tokens but it also assumes that you will provide the notification-controller with a fresh token when needed.

For JWT token based auth we have created a small example on how to automatically generate a new token that the notification-controller can use.

First you will need to create a Azure Event Hub and bind a credential such as a Service Principal or Managed Identity to it. If you want to use Managed Identities, install or enable AAD Pod Identity.

We have two ways to automatically generate new JWT tokens. Ether running as a deployment or a cronjob.

If you are using Azure Event Hub in Azure we recommend that you use aadpodidentity. If you do you will need to update the AzureIdentity config example.

If you are in none Azure environment like on-prem or another cloud then you can utilize client secret which you will find in the example generic folder. Just like aadpodidentity you can use deployment based or a cronjob.

For more info on how to use Azure Event Hub with the notification controller.