GitOps w/ FluxCD: Automated Image Updates.

In a previous article, we set up a Tekton CI Pipeline and witnessed the creation and addition of the vote app Image in Docker Hub.

The icing on the CI Pipeline will be if FluxCD, through one of its Controllers, could check our Docker Hub for newer versions of Images and, without us updating a docker Image tag in vote-deployment.yaml file, make sure they become the basis of our apps.

How Do We Connect the Left Side to the Right Side?

Through the use of Image Automation Controllers and their trusty side-kick, the Image Reflector Controllers.

In another previous article, we set up a Push-Based Reconciliation for our repo and FluxCD. Therefore, if we successfully establish automated Image scanning, we, in theory, have a fully automated CICD Pipeline.

Enable Image Auto Updates

Check if our installation of FluxCD has the Image Automation and Image Reflector Controllers installed.

flux check

In case, your list does not contain the three names above, you can use the command provided below to include them in your FluxCD installation.

flux bootstrap github
--owner=$GITHUB_USER \
--repository=flux-infra \
--branch=main \
--path=./clusters/staging \
--personal \
--log-level=debug \
--network-policy=false \
-- token-auth
--components-extra=image-reflector-controller,image-automation-controller

Create an Image Repository object which will hold a reference to the source of your image.

For us, that source is Docker Hub.

# Check if any Image Repository objects exist
flux get image repository

Now execute the command to create an Image Repository object.

flux create image repository vote --image=usmanlakhani/vote --interval=1m

Check ImageRepository objects again.

flux get image repository

This is great progress, indeed! We have a ImageRepository object that was able to connect to our public Docker Hub repo.

Now comes the part where we document the criteria that must be used by the ImageRepository to determine if an Image is new and should be auto-updated.

Define an Image Selection Policy

In a previous article, where we configured a Tekton CI Pipeline, the Images saved to Docker Hub follow a naming convention as detailed below.

Image Name = Branch from where code is taken followed by the first 8 characters from git commit and ending in a time stamp in literal format.

We can use the naming convention adopted by our Images and make an ImagePolicy object based on it.

flux create image policy vote \
--image-ref=vote \
--select-numeric=asc \
--filter-regex='^main-[a-f0-9]+-(?P<ts>[0-9]+)' \
--filter-extract='$ts'

• image-ref = the name of the Docker Hub repo to check for Images
• select-numeric = sort the Images in an ascending numeric order
  • We will use the linear timestamp for this sorting logic
• filter-regex = Only Images that satisfy this regex should be selected for sorting in an ascending order
• filter-extract = Once sorted in ascending order, pick the Image with the highest timestamp value

Check the ImagePolicy.

flux get images policy

At this point in our journey, we have a way to

• Keep an eye on our Docker Hub accounts (through ImageRepository/vote) and
• Only pick container Images that satisfy some criteria (through ImagePolicy/vote)

Automated Committing of Image Tags to GitHub

What is the why here? Why do we want to let GitHub know about any Image tags being updated? Since no official reason has been provided, one's own experience can, perhaps, be used to rationalize this task.

At least, two strong reasons come to mind:

1. If we have an updated Image which has a critical defect fix, or a new killer feature, we would want it out in the hands of users. Setting up automated committing of Image tags in a Deployment manifest will create a seamless approach from coding a new Image to releasing it for the world to use.
2. It also feels like this is an additional confirmation step that the right Image was built and by letting GitHub know the tag, we can create an end-to-end verification process for releasing business critical applications.

Place a marker in the Deployment manifest for the app being set up for auto Image tag update.

That's a long-winded (and wordy) way of saying:

''The actual deployment of Pods with a new Image happens through a Deployment manifest, therefore, we have to insert some sort of 'X-marks-the-spot' in them so when the Kustomizations kick in, they know where to look for the Image:tag to be added to the Deployment manifest. For our purpose, since we already set up a kustomization for out vote App, we can simply make relevant additions to it and let the Source and Kustomization Controllers deal with the rest"

• Open the instavote/deploy/vote/staging/kustomization.yaml file.

# instavote/deploy/vote/staging/kustomization.yaml
images:
- name: usmanlakhani/vote
  newTag: v4 # {"$imagepolicy": "flux-system:vote:tag"}

Make an ImageUpdateAutomation object

flux create image update instavote-image-updater \
--interval=30m \
--git-repo-ref=instavote \
--git-repo-path="./deploy/vote/staging" \
--checkout-branch=main \
--push-branch=main \
--author-name=fluxcdbot \
--author-email=fluxcdbot@users.noreply.github.com \
--commit-template="{{range .Updated.Images}}{{println .}}{{end}}"

If you followed all the instructions for making the ImageUpdateAutomation object and still get the error below, it's ok. Rather it's expected.

The cause of this error is the same old same old. You are trying to make an ImageUpdateAutomation object WRITE back to a GitHub repo and, in order, for that to happen successfully, GitHub MUST trust you.

Fortunately, we have seen these kinds of errors before and the solution is simple. We have to create a new Secret through which ImageUpdateAutomation will satisfy GitHub's paranoia and be able to write to it.

Make a new Kubernetes Secret for ImageUpdateAutomation to authenticate with GitHub

# Unless already done, create 2 env variables: one for GitHub Username and # #the second for GitHub PAT
export GITHUB_USER=usmanlakhani
export GIT_PAT=<PAT>

# This time, lets use flux for creating a Secret
flux create secret git flux-github-instavote \
--url=https://github.com/usmanlakhani/instavote \
--username=$GITHUB_USER \
--password=$GIT_PAT

Update the flux-infra/clusters/staging/instavote-staging-gitrepository.yaml to include a reference to the Secret

• Check-in file to GitHub
• Now, either wait for reconciliation to occur as per the documented interval OR manually reconcile the ImageUpdateAutomation

flux reconcile image update flux-github-instavote

Test your work

• Initiate another Tekton CI PipelineRun for the vote app

# Find your way back to the tekton-ci/base folder
kubectl create vote-ci-pipelinerun.yaml

Areas to test

1. Docker Hub should have a new Image added

2. Check GitHub Repo Commit History

3. You can also look at the GitHub copy of the instavote/deploy/vote/staging
   /kustomization.yaml

4. Finally, notice the same Image tag has been used in all three areas

I write to remember, and if, in the process, I can help someone learn about Containers, Orchestration (Docker Compose, Kubernetes), GitOps, DevSecOps, VR/AR, Architecture, and Data Management, that is just icing on the cake.