The shape of the problem.
chitra.art ships from one repo into several runtime surfaces. There
is the backend API, a scheduled scraper (ArtScout), a digest sender,
a frontend, a generated TypeScript SDK, and the hardware gantry's
own API and SDKs. Each had its own
azure-pipelines.yml and its own trigger. That mapped
cleanly to "one folder, one pipeline" — until you actually
push a commit.
The Azure DevOps free tier gives a private project one parallel job.
One. So the moment a commit touched backend/Common
(which most of them do), three or four pipeline definitions
triggered at once, and the agent picked them off one at a time.
Build the backend image, deploy. Build ArtScout, deploy. Build
DigestSender, deploy. Build the frontend, deploy. Each one waiting
on the last because the agent was busy.
The math we kept doing.
I went back through about a month of runs in Azure DevOps and
pulled representative timings off the per-pipeline summary pages.
Nothing scientific — just typical numbers from typical
commits that touched a shared file like
backend/Common or a top-level config.
Add deploys, queue gaps, and the agent occasionally restarting between jobs and a multi-component commit was easily a twenty-five to thirty-five minute wall-clock wait before everything was live. On a busy afternoon I'd push, go make coffee, come back, and the frontend still hadn't started.
Why the $15-per-agent path didn't pencil.
Microsoft's answer to this is straightforward: pay for additional parallel jobs. For self-hosted pools that's about $15 USD per extra parallel slot per month — cheap on its own, but to get any meaningful speedup the unit of purchase is wrong. One extra slot turns six sequential builds into three sequential pairs. To actually fan out six components I'd need five extra slots, and at that point I'm paying ~$75/month to work around an architectural choice rather than fixing it.
233 Solutions is a side studio. I'd rather spend that money on storage, model calls, or hardware than on parallelism that a small amount of shell scripting can give me for free.
One job, many background processes.
The fix turned out to be smaller than expected. Azure DevOps
charges for parallel jobs, not for what runs inside
a job. So I collapsed every component pipeline into a single
azure-pipelines.yml at the repo root, with one
RunAll job, and used Bash to fan out the work.
run_bg() {
local name="$1"
shift
echo "Starting ${name}..."
( "$@" ) >"${LOG_DIR}/${name}.log" 2>&1 &
PIDS+=("$!")
NAMES+=("$name")
}
wait_all() {
local failed=0
for i in "${!PIDS[@]}"; do
if wait "${PIDS[$i]}"; then
echo "Completed ${NAMES[$i]}"
else
echo "##[error]${NAMES[$i]} failed"
failed=1
fi
done
return "$failed"
}
A short detection step at the top of the job inspects the diff and
sets per-component flags — BUILD_BACKEND_API,
BUILD_FRONTEND, and so on — so the run only
does work for what actually changed. Then the next step launches
every selected build in the background and waits on all of them.
Buildx caches live on the agent's disk; each component has its own
named builder so they don't fight over the cache.
Deploys are wired the same way: a component-build-status.env
file written by the build step tells the deploy step which images
actually produced something, and each az containerapp update
is launched as another background process. The whole thing still
occupies exactly one Azure DevOps parallel slot.
What the runs look like now.
The most common case — a small change to one component — now goes from commit to live container in roughly the time it takes to switch to a browser tab and refresh. Even the worst-case "everything changed" rebuild now finishes in the time the longest single image takes, instead of the sum of all of them. And the bill is still zero.
Why this matters more than it used to.
For most of my career a thirty-minute CI was annoying but tolerable: you batched work, you context-switched, you came back. Working alongside coding agents changes the economics. The loop is no longer "write code, wait, look at logs." It's "ask the model, review the diff, push, observe, ask again." Each cycle is a few minutes of real thinking and a few seconds of typing — the CI is the part with the most slack to give back.
When iteration costs a minute and a half, you'll try the next small experiment. When it costs thirty, you'll bundle three or four changes into one push, lose the bisect-ability when something breaks, and stop trying speculative things at all. The fast loop isn't just nicer — it's what keeps you running experiments instead of avoiding them.
What I'd do differently if I were starting fresh.
-
Start with one pipeline definition at the repo root. The
per-component
azure-pipelines.ymllayout looks tidy and is genuinely a trap on the free tier. - Treat the diff-detection step as a first-class part of the design, not an afterthought. Every minute saved is a minute the parallel slot isn't held.
-
Keep build logs per-component even when everything runs in one
job. Streaming six builds' output to one terminal is unreadable;
writing to
${LOG_DIR}/${name}.logand printing them one at a time afterwait_allis much nicer. - Don't be precious about Bash. The whole orchestrator is about a hundred lines and it has paid for itself many times over.
The full consolidated pipeline lives in
azure-pipelines.yml at the repo root if you want to
crib from it. Most of the file is the component-specific build and
deploy functions; the orchestration is the small part at the top
and bottom.