Upgrading your Buildkite Pipelines with git diff

In my current role, I've been working with Buildkite quite extensively. One thing I really miss from GitLab is the changes part of step rules. It's super useful for updating Dockerfile while not bogging down your pipeline with things that aren't pertinent. I know there are some people out there who philosophically believe in "running the whole pipeline every time", but who has the time?

I've got a very simple script below (designed for updating Docker builds), that selectively creates steps when certain files have been updated. You can also run a build that will forcibly rebuild all images which you can queue with a pipeline execution on your branch's head and set the environment variable DEPLOY_ALL=true.

This script does the following:

1. Checks if the DEPLOY_ALL environment variable has been set, if it has it will queue all the images for a build.
2. For each image, it will then attempt a match to each changed file to the image. If one is found, it will queue up the build.
3. If a build has been queued due to a changed file, the step file located at .buildkite/container_steps/ will be uploaded as a step.

This won't work all the time, but for some pipelines it just makes sense. I'd rather have the ability for people to skip superfluous steps rather than wait an extra ten minutes. Every minute an Engineer spends waiting for a build to complete is time they could have spent doing something else (and they would enjoy it more too)!

#!/bin/sh
FILES_CHANGED=$(git diff --name-only HEAD~1)
PROJECT_IMAGES=$(ls containers)

if [ "$DEPLOY_ALL" = "true" ]; then
  for image in $PROJECT_IMAGES; do
    echo "'$image' has been queued for build (DEPLOY_ALL)."
    step_file=".buildkite/container_steps/${image}.yaml"
    buildkite-agent pipeline upload "$step_file"
  done
else
  for image in $PROJECT_IMAGES; do
    echo "Checking whether '$image' has been updated."
    image_updated="false"
    for changed_file in $FILES_CHANGED; do
      image_regex="^containers/${image}.*"
      match_count=$(expr "$changed_file" : "$image_regex")

      if [ $match_count -gt 0 ]; then
        image_updated="true"
      fi
    done
    
    if [ "$image_updated" = "true" ]; then
      echo "'$image' has been updated."
      step_file=".buildkite/container_steps/${image}.yaml"
      buildkite-agent pipeline upload "$step_file"
    else
      echo "'$image' has not been updated, skipping."
    fi
  done
fi