GitOps w/ FluxCD: Continuous Integration using Tekton.
The CD in FluxCD stands for Continuous Delivery and it is for this reason that FluxCD is considered a critical tool for increasing the productivity of a CICD pipeline. For the first 8 articles in this series, CD was the focus of our investigations. We will now travel upstream and construct the missing CI steps as well and (no surprises) we will utilize Tekton, another CNCF project, for this purpose.
What is Tekton?
Tekton is a set of Custom Resource Definitions installed in a cluster, which performs CI operations as part of a CICD pipeline.
Tektons Architecture
- A Pipeline
- A set of Triggered Tasks
- Each Task may have multiple Steps
As code is checked into a repo, a Trigger event is generated and sent to a Tekton Pipeline. A Pipeline is a sequence of Tasks that are executed, linearly or in parallel, and result in a new app image being deposited in a container registry (like Docker Hub).
Once the image is updated, the CD side of the equation (with reconciliations etc) will be able to pick up the latest and greatest versions of our unicorn product, all without one having to do anything more than check in the application code to a repo.
A Step is an operation in a CI/CD workflow, such as running some unit tests for a Python web app, or the compilation of a Java program. Tekton performs each Step with a container image you provide. For example, you may use the official Go image to compile a Go program in the same manner as you would on your local workstation.
A Task is a collection of Steps in order. Tekton runs a Task in the form of a Kubernetes pod, where each Step becomes a running container in the pod. This design allows you to set up a shared environment for many related Steps; for example, you may mount a Kubernetes volume in a task, which will be accessible inside each step of the Task.
A Pipeline is a collection of Tasks in order. Tekton collects all the Tasks, connects them in a directed acyclic graph (DAG), and executes the graph in sequence. In other words, it creates several Kubernetes Pods and ensures that each Pod completes running successfully as desired. Tekton grants developers full control of the process: one may set up a fan-in/fan-out scenario of Task completion, ask Tekton to retry automatically should a flaky test exist, or specify a condition that a Task must meet before proceeding.
Install Tekton
Before installing Tekton, ensure a default Storage Class is available in the cluster.
But why? What does Tekton have to do with Storage Classes?
Great question and one that needs answering.
Slice it anyway, Tekton Tasks are Pods that run in a set of containers in some sequence. It may be the case that some output from Task A needs to be used by an upcoming Task D. To facilitate this exchange of needed data, a Volume is maintained for each Tekton Task (or Task Pod) to write data to.
Download Tekton Binary
The official website for downloading the Tekton Server-Side is https://tekton.dev/docs/installation/pipelines/.
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yamlAlways use the official documentation site for installing Tekton. *The URL used above was valid on December 18th, 2022.
Kubernetes will create native objects for Tekton, including namespaces, CRDs, Pods, Services, Roles, Secrets, ConfigMaps, etc.
kubectl get crds | grep -i tekton
Optional additional checks could include...
kubectl get all -n tekton-pipeline
kubectl get crds -n tekton-pipelineInstall Tekton Client
Like Kubernetes clusters have an API Server and a Client (our tried, tested and adored kubectl), Tekton has both.
- The official website for Tekton CLI is https://tekton.dev/docs/cli/.
- For this article, the URL used for downloading the CLI is https://github.com/tektoncd/cli/releases/download/v0.35.1/tektoncd-cli-0.35.1_Linux-64bit.deb.
- After installation is complete, verify it.
Set Up Tekton Pipeline Resource
Create the following Tasks for the Pipeline:
- git-clone: clones source repository, marks and informs other tasks about latest commit hashes
tkn hub install task git-clone- kaniko: builds Docker container images and publishes them to a Docker Hub registry
tkn hub install task kaniko- Verify the Tasks were installed
tkn t list
Clone Tekton-CI Git Repository
The core code for Tekton Pipelines is saved at https://github.com/lfs269/tekton-ci.git.
- Clone the repository.
- For this series, the root folder is gitops and we have been cloning repo's inside it. We will continue to do the same for all repos that have to be cloned.
git clone https://github.com/lfs269/tekton-ci.git The project folder structure should resemble the layout in the image below.
- Change the directory to tekton-ci/base and run the kubectl apply command
cd tekton-ci/base
kubectl apply -f instavote-ci-pipeline.yamlIt is indeed true that Tekton will install YAML manifests for Pipelines that were created keeping the Instavote app in mind. I believe the Instavote app is very popular amongst CNCF projects because of its bag of languages and therefore, making examples for both FluxCD and Tekton around it provides readers with a chance to better understand concepts.
In fact, Tekton is very kind to us and, in addition to instavote-ci-pipelines.yaml, it also provides us with templates for:
- docker credentials for use by a Tekton Task to save images in Docker
- result-ci-pipelinerun.yaml
- vote-ci-pipelinerun.yaml
Another question that should be asked at this point? Why did I choose the instavote-ci-pipeline.yaml file?
This YAML manifest is actually a template for Tekton Pipelines.
We can open the file and read the first few lines where it clearly states this is a template to be used by us.
Let's be more inquisitive and peek into instavote-ci Pipeline metadata.
tkn p describe instavote-ciThe terminal should get filled in with particulars that make up the Pipeline.
The template has scaffolding for any Pipeline we might want to run for Instavote.
- The Params section has a list of variables that need to be configured for the Pipeline
- The Workspaces lists the shared storage for Tekton Pipeline Tasks
- The Tasks list shows the default set of Tasks that would likely be used in a CI Pipeline
Another very important section is Pipeline Runs.
So we have a template. Now what?
We will now create a PipelineRun.
Any time, another Pipeline that references the instavote-ci Pipeline template is executed, it is called a PipelineRun.
For our article, open the file vote-ci-pipelinerun.yaml in tekton-ci/base folder.
Before we can successfully execute the vote-ci PipelineRun, we have to configure its values.
The purpose of each parameter is:
- repoUrl: The GitHub repo to run CI PipelineRun against
- revision: The branch in the repo that holds the code base
- sparseCheckoutDirectories: The GitHub repo directory that has the code for the app
- This is a great performance boost to the checkout process in general. Typically, CI tools will check out the entire repo but Tekton allows us to filter out the files that are not needed and focus on those that are.
- imageUrl: The Docker Hub username and the repo where the built code (in the form of an Image) will be saved
Though we are not editing these, it's important to be aware of shared-data and dockerconfig.
- shared-data is the name of the volume where all output-producing Tasks will save their outputs, for upcoming Tasks to use.
- dockerconfig is the name of a Secret that authenticates your PipelineRun for Docker Hub. We did nothing to configure it right now and we will suffer for it soon, but till then, onwards.
Before the PipelineRun is initiated, list existing Pipelines and PipelineRuns.
tkn p list
tkn pr list
Without any more delays, let's run our first PipelineRun.
kubectl create -f vote-ci-pipelinerun.yamlCheck PipelineRun was kicked off.
tkn pr list
The PipelineRun will either go on for too long or exit with a failure.
Load the logs for the PipelineRun, in the hope that it gives us some insight.
tkn pr logs -f -a 
Is there something in the vote-ci-pipelinerun.yaml file that can help us?
This should come as a surprise. One would expect the manifest for the PipelineRun should have the Steps listed but there is no information here that is useful.
Does this mean we won't be able to troubleshoot this problem?
Fear not.
Remember the vote-ci PipelineRun is derived from the instavote-ci Pipeline template, which means that we should be able to look for some clues to our problem in instavote-ci-pipeline.yaml file.
On comparing the logs from our PipelineRun and the contents of the instavote-ci-pipeline.yaml file, we can identify the area which is breaking.
It looks like the shorten-commit-hash Task Step was successful because we do have a hash printed. The next Step however was not getting kicked off and it feels like there may be something to fix here.
Remember dockerconfig?
dockerconfig is a K8s Secret that will be used by the img-build-publish Step to authenticate against our Docker Hub account. Since we didn't set dockerconfig, Docker Hub is probably not letting our PipelineRun access our repo.
Generate a Docker Registry Secret
- Login to Docker
docker loginThis generates a config.json file. You can base64 encode this file's content and save it in a K8s Secret.
- Based64-encode the file's content and keep the output handy.
cat ~/.docker/config.json | base64 | tr -d '\n'- Create a K8s Secret with the base64-encoded string. Fortunately, Tekton provides a Secrets manifest (docker-creds-secret.yaml) that can be used here.
Edit the contents of this file.
apiVersion: v1
kind: Secret
metadata:
name: docker-hub
data:
config.json: <base64 hash here>
Create the Secret.
kubectl apply dockerhub-creds-secret.yamlThis secret is already being referred to from the vote-ci-pipelinerun.yaml file and we can assume that this time when the PipelineRun starts, it will have all the necessary ingredients for a successful run.
Initiate a new PipelineRun
Before we go again, let's take a look at my Docker Hub account
- Cancel the current failed PipelineRun
tkn pr cancel vote-<some hash>- Kick-off a new PipelineRun
kubectl create -f vote-ci-pipelinerun.yamlAssuming all changes were applied correctly, the PipelineRun should finish quickly, and we can confirm its success using our terminal and, of course, our Docker Hub.
Can I confirm that the last PipelineRun did indeed create this image? It could be someone else trying to trick me.
Fair question with a simple answer: Yes we can confirm this image is because of the last PipelineRun by looking at the logs for it.
tkn pr logs --lastScroll to the bottom of the log files and you should see an entry like the one below.
The image published on Docker Hub has the same name as the one we see above.
I write to remember, and if, in the process, I can help someone learn about Containers, Orchestration (Docker Compose, Kubernetes), GitOps, DevSecOps, VR/AR, Architecture, and Data Management, that is just icing on the cake.