Challenge: Troubleshoot a failing deploy

Verified lab · 60 min · Path challenge

Path challengeOperate your application

Preview. The set-up script and the fix have been run end to end; the hidden checks are still being confirmed against the fixed shop. Read the brief now; the challenge opens when they're ready.

Goal: Restore a broken service on your own, by finding every fault a bad release brought in and fixing it forward.

You need Modules 1 to 8 done, your lab project and environment with no other services in it, and about an hour.

A shop called Harbour Supply Co. runs on learn-shop-api. Its team shipped a new release, and the shop stopped answering. You're on call. There are no steps: everything you need, you've done in this path.

Set up

A script deploys the working shop as shop in your lab environment, lets it serve for a minute, then ships the bad release. It takes about three minutes. Don't read the script until you're done: it names the faults.

Run the set-up

You need jq: brew install jq on macOS, sudo apt install jq on Ubuntu or Debian. Create an API token with Project access to your lab project, as in lab 5.8.5, and find your IDs with csph projects list and csph environments list --project <project-id>. Then:

git clone https://github.com/computesphere-samples/learn.git
cd learn/labs/shop-api/broken
read -s COMPUTESPHERE_API_TOKEN && export COMPUTESPHERE_API_TOKEN   # paste the token, then Enter
export PROJECT_ID=<lab-project-id> ENVIRONMENT_ID=<lab-environment-id>
./challenge-setup.sh

The script also needs your account ID. It uses the default account csph saved when you signed in, and says which one: “Using account …”. If it stops with “couldn't work out your account ID”, run csph accounts list, then export COMPUTESPHERE_ACCOUNT_ID=<account-id> and run it again.

It ends with “The new release of 'shop' is out, and the shop has stopped working.”

The brief

Before the release, shop answered 200 on /products from 1 spherelet. It had its variables, a SIGNING_KEY secret, and a health check on /healthz, port 8080. Get it back to that by fixing the release forward. The release didn't record a new version, so there's nothing to roll back to.

Check your work

How it's checked

The checks run when you choose Submit, against what's running in your lab environment, and stay hidden until then. Fix what failed and submit again; there's no limit on attempts.

Before you submit, test it yourself: curl the URL's /products a few times.

Doing it on your own

Use the troubleshooting order from module 5, your notes, the ComputeSphere docs and csph --help. Expect more than one fault. Don't trust the status alone: check the URL. Compare what you see with the brief, and write down each fault with its evidence: the response, the log line, the setting, the spherelet count, the secret list.

A time guide
MinutesPart
10Set up; read the status and try the URL
20Logs and settings: find every fault
20Fix it; wait for Running
10Test it yourself, submit

The app takes about two minutes to load its catalog and pass its health check. Stuck for much longer on one part? Reread the module it came from.