Gitops w/ FluxCD: Configuring FluxCD for Push-Based Reconciliation.

FluxCD employs a Pull-Based Reconciliation. After changes are made, manifests have to be checked into a repo, from where Kustomizations, periodically, find out what to update in the K8s infrastructure. Sometimes a check-i

FluxCD: Not A One-Trick Pony.

Fortunately, FluxCD can be configured for Push-Based Reconciliation as well.

• The code will be checked in
• The 'code-check-in event' will push (or trigger) the GitOps flow, and reconcile the infrastructure (without waiting for the documented intervals to elapse)

What would a Push-Based Reconciliation look like?

• GitHub sends git events and an authentication token to a Reciever Service
• GitHub sends the git push event to the webhook-receiver Service (which we will configure)
• Notification Controller validates the authenticity of the payload using a token (which we will create)
• Source Controller pulls the changes into the cluster and updates the instavote GitRepository object
• The Kustomize Controller is notified about the revision change
• Kustomize Controller reconciles all the Kustomizations that reference the GitRepository object

Demo: Configure a Push-Based Reconciliation GitOps Flow

This demo introduces the GitRepository Receiver object. This object contains the list of git events, the token used for authenticating messages from GitHub and a reference to the GitRepository Source Object (which for our series has been instavote).

As part of this demo, we will create 2 new objects (a token, shared between GitHub and the Push-Enabling FluxCD components, and a GitRepository Receiver) and edit some of the existing Kustomization manifests (gotk-components.yaml) to expose a public-facing webhook (for GitHub to send messages to).

Edit manifests and increase time intervals

Edit vote-staging-kustomization.yaml

Edit instavote-staging-gitrepository.yaml

Expose a web service that is publicly accessible for GitHub to send messages to

To receive Git push or Helm chart upload events, you must expose a webhook receiver endpoint outside your Kubernetes cluster on a public address.

Surprisingly, FluxCD installs a Service called webhook-receiver as part of bootstrapping which can be used here.

kubectl get svc -n flux-system

The webhook-receiver Service is a ClusterIP service, by default, but needs signals from GitHub and, therefore, must be changed to either a LoadBalancer or NodePort Service type.

Since webhook-receiver is installed as a part of FluxCD bootstrapping, it MUST have a manifest in the flux-infra repo. Browsing through the flux-infra/clusters/staging/flux-system folders, we will find a webhook-receiver's manifest (gotk-components.yaml) file.

We can use our Kustomize experience (remember this article?) to create an overlay patch turning the webhook-receiver from ClusterIP to NodePort.

Title the patch webhook-receiver-service.yaml (code shown below).

apiVersion: v1
kind: Service
metadata:
 name: webhook-receiver
 namespace: flux-system
spec:
 ports:
 - name: http
   port: 80
   protocol: TCP
   targetPort: http-webhook
   nodePort: 31234
 selector:
   app: notification-controller
 type: NodePort

Kustomize needs to know where to look for the files which will be customized.

Edit the kustomization.yaml file in staging/flux-system/.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
patches:
- webhook-receiver-service.yaml

Hold on. What is patchesStrategicMerge? The last time I made a Kustomization file I used patches.

The patches take the contents of the webhook-receiver-service.yaml file, find the exact sections in gotk-components.yaml and gotk-sync.yaml and replace them with the contents of the webhook-receiver-service.yaml manifest.

Still unclear? Let's look at the image below to appreciate the magic of patches.

First, list the files in the clusters/staging/flux-system folder.

When we execute the kustomize build command at the root of flux-system folder, Kustomize gets busy generating an edited manifest for gotk-*.yaml.

We can save the generated YAML into a temporary file using

kustomize build -o temp.yaml

Search for 'webhook-service' in temp.yaml and you should be able to see that webhook-service has been turned into a NodePort.

Git-check new code/manifests

git add *
git commit -m 'comment'
git push origin main

Either manually reconcile the infrastructure or let the set interval elapse before checking the state of reconciliation.

flux get kustomizations

However, the real proof that kustomization worked is to confirm webhook-receiver was converted to a NodePort service.

kubectl get svc -n flux-system

Any traffic sent from GitHub to webhook-receiver Service must be authenticated for security any traffic sent from GitHub to webhook-receiver Service. After all, some malicious man-in-the-middle could corrupt the payload from Git to our cluster.

Generate a new K8s Secret to hold the shared token

# Generate a token
TOKEN=$(head -c 12 /dev/urandom | shasum | cut -d ' ' -f1)

# Save token. It will be used later.
echo $TOKEN

# Turn it into a Secret
kubectl -n flux-system create secret generic webhook-token \	
--from-literal=token=$TOKEN

Create a GitRepository Receiver

This Receiver will accept GitHub events, filter out those required and ignore those that are not.

flux create receiver instavote \
--type github \
--event ping \
--event push \
--secret-ref webhook-token \
--resource GitRepository/instavote \

• 1: git events in scope (ping, push).
• 2: The GitHub repo from where these events are originating.
• 3: The shared token we just generated. This token is validated by the Notification Controller before letting any traffic pass through.
• 4: The type of code repository the receiver is being made for.

At the end of this step, we have FluxCD components alive and ready.

Our focus now shifts towards GitHub.

Set up a webhook on GitHub

Remember that shared token made earlier? We do have to share it with GitHub.

To get the token, echo the value of $TOKEN.

echo $TOKEN

# The token should resemble this:
# e83d76derar7a85f44aabcgdaaea180b1e8c3

We also need the URL of the webhook-receiver Service. This URL will be saved as part of the webhook that will reside in GitHub.

flux get receivers

Using the URL from the image and appending it to the Node IP and NodePort Port Number will give us the Payload URL.

Load the instavote GitHub repository page.

• Go to Settings --> Webhooks.
• Click on Add Webhook.
• Paste the Payload URL in its text box.
• Paste the token in the Secret text box.
• Select the Just the Push event radio button.

Check the Recent Deliveries tab to confirm everything worked.

Test the integration

It's time to put FluxCD's money where its mouth is. We can make a minor modification in the vote app by increasing the replica count from 2 to 5.

Change the replicas in instavote/deploy/vote/staging/vote-deployment.yaml file from 2 to 5.

After code is checked in, check the replicas count.

Another check is to compare the SHA for the commit (in GitHub) and make sure it's the same for the vote-staging Kustomization.

I write to remember, and if, in the process, I can help someone learn about Containers, Orchestration (Docker Compose, Kubernetes), GitOps, DevSecOps, VR/AR, Architecture, and Data Management, that is just icing on the cake.