Jobs and steps

Description and operation

General

A job is one unit of work in its own sandbox. Its steps run one after the other in that sandbox, so the files of one step are there for the next. Files do not pass from one job to another directly. A job can save its home as a version of a named volume, and a later job can start from it. See Home.

A definition has these limits.

To do these tasks with an AI agent, see Agentic workflows – General.

Order and parallel jobs

The jobs of a pipeline form a graph. A job with no needs starts at once. A job with needs starts when each job that it names has ended. Jobs that do not need each other run in parallel, up to 8 jobs of a run at the same time. A needs list that makes a cycle is refused at save.

Conditions

The platform evaluates the if of a job when the job is about to start. Without if, a job runs only when every job in its needs succeeded. An if that calls no status function adds that rule, so inputs.target == 'prod' runs only after successful needs.

When a run is cancelled, the platform starts no more jobs. It does not evaluate if, so always() jobs do not start.

A condition reads inputs.NAME, matrix.NAME, needs.JOB.result, event and webhook.body.PATH. The operators are !, ==, !=, &&, || and parentheses. The rules are in Expressions.

Matrix

A matrix maps each axis to a list of values. The job runs once for each combination, up to 64. The first axis varies slowest. Each instance has the key job (value, value), for example test (python311). A job that needs a matrix job waits for all its instances. needs.test.result is failure if any instance failed.

The values of an axis must be different from each other.

Runtime

A job uses one of two runtimes. With none, the job runs on the platform base image with the package git.

Each sandbox has access to the internet. A job that passes timeoutMinutes fails with timed out. Each step has the same limit by itself. The log of a job is cut after 8 MiB.

Home

The home of a job is the folder /home/kl. It is the disk of the job sandbox. By default it is empty. fromSnapshot and saveTo set what the home starts with, and where it goes when the job succeeds. Both combine with packages or image.

A reference is name, name@N or a snapshot id. go-base is the newest ready version of the volume go-base. go-base@3 is version 3. See Volumes.

A job with saveTo saves its home as the next version of the volume when it succeeds. A job that fails, times out or is cancelled saves nothing. The job is success only when the save is ready. When the save fails, the job fails with the reason, and the jobs that need it are skipped.

A name that exists but has no ready version fails the job. Only a name that is not made yet starts empty.

The platform finds an unpinned name when the job starts, not when the run is planned. A job that needs the saving job therefore reads the version that the saving job wrote.

Rules for the home:

• saveTo is a volume name. It is a literal: it has no ${{ }}.
• A job with saveTo has no matrix.
• fromSnapshot and saveTo name the same volume. A snapshot id must be a push of that volume.
• Two jobs that save to one name must be ordered with needs.
• A name belongs to the owner of the pipeline. A team pipeline reads and writes the volumes of the team. A job never reads the volume of another owner.
• A volume that is an environment volume is not a start point or a save target.
• A save counts one snapshots unit of the quota of the owner. The job fails at start when the quota has no free unit.
• To write saveTo, you need the same role as to save the pipeline.

The run records the version that each job used and the version that it wrote. The home and saved fields of a job show them. To run a job again from the same version, pin it with an input: fromSnapshot: "${{ inputs.base }}".

Steps

A step has a run command, an optional name, env and cwd. The platform runs each run with sh -e -c, so the first command that fails ends the step and the job. A step starts in the home directory. cwd is a folder relative to the home. The log of the job shows ▶ and the step name before the output of each step. A step with no name shows the first line of its run.

Each step runs in its own subshell. Only files pass to the next step. A cd or an export does not.

Variables

Step env values are added to the job env, and a step value replaces a job value of the same name. A value can read ${{ inputs.NAME }}, ${{ matrix.NAME }} and ${{ needs.JOB.result }}. A step run, cwd and env can also read ${{ env.NAME }}. The fields image, packages and fromSnapshot cannot read needs. The platform also sets these variables in every step.

For secrets in variables, see Secrets, registries and environments.

Tasks

Run jobs in parallel and then in order

Reason for the task: Do independent work at the same time, then run one job that uses all the results.

Job set-up

• Access: you. For a team, an admin, and an active member to start the run.
• References: Pipeline definition – Jobs, Create a pipeline.

Procedure

1. Write this definition to site-check.json. The jobs api and docs run in parallel, and summary runs after both.

   jsonCopy

   {  "name": "site-check",  "jobs": {    "api": {      "packages": ["curl"],      "steps": [{ "name": "probe", "run": "curl -fsS -o /dev/null https://api.github.com" }]    },    "docs": {      "packages": ["curl"],      "steps": [{ "name": "probe", "run": "curl -fsS -o /dev/null https://docs.github.com" }]    },    "summary": {      "needs": ["api", "docs"],      "if": "always()",      "steps": [        {          "name": "report",          "run": "echo \"api=$API docs=$DOCS\"",          "env": { "API": "${{ needs.api.result }}", "DOCS": "${{ needs.docs.result }}" }        }      ]    }  }}

2. Create the pipeline from the file, as in Create a pipeline.

3. Start a run and wait for it to end.

   PythonCopy

   run = kl.pipelines.run(owner="acme", pipeline="site-check")print(run.wait(), [(j["key"], j["state"]) for j in run.jobs])

   TypeScriptCopy

   const run = await kl.pipelines.run({ owner: "acme", pipeline: "site-check" });console.log(await run.wait(), run.jobs.map((j) => [j.key, j.state]));

   GoCopy

   run, err := kl.Pipelines.Run(kloudlite.RunPipeline{Owner: "acme", Pipeline: "site-check"})if err != nil {    log.Fatal(err)}state, err := run.Wait(kloudlite.WaitRun{})if err != nil {    log.Fatal(err)}fmt.Println(state)for _, j := range run.Jobs {    fmt.Println(j.Key, j.State)}

   RubyCopy

   run = kl.pipelines.run(owner: "acme", pipeline: "site-check")puts run.waitrun.jobs.each { |j| puts "#{j["key"]} #{j["state"]}" }

   JavaCopy

   var run = kl.pipelines().run(new Kloudlite.RunPipeline().owner("acme").pipeline("site-check"));System.out.println(run.waitFor());for (var j : run.jobs()) System.out.println(j.get("key") + " " + j.get("state"));

   CLICopy

   RUN=$(kloudlite pipelines run acme/site-check --json | jq -r .id)kloudlite pipelines wait-run acme site-check "$RUN"kloudlite pipelines get-run acme site-check "$RUN"

Close-out

1. Make sure that api and docs have close startedAtMs values, and that summary started after both ended.
2. Read the log of summary. It shows the result of each needed job.

Run a job only on a condition

Reason for the task: Do a step of work only for some runs, for example only for the production target.

Job set-up

• Access: you. For a team, an admin.
• Conditions: you have the definition of task 001.
• References: Conditions, Inputs.

Procedure

1. Add the input target and the job announce to site-check.json.

   jsonCopy

   {  "name": "site-check",  "inputs": { "target": { "options": ["staging", "prod"], "default": "staging" } },  "jobs": {    "api": {      "packages": ["curl"],      "steps": [{ "name": "probe", "run": "curl -fsS -o /dev/null https://api.github.com" }]    },    "announce": {      "needs": ["api"],      "if": "inputs.target == 'prod'",      "steps": [{ "name": "announce", "run": "echo \"production check passed\"" }]    }  }}

2. Send the definition as the next version. See Save a new version.

3. Start a run with target set to prod.

   PythonCopy

   run = kl.pipelines.run(owner="acme", pipeline="site-check", inputs={"target": "prod"})print(run.wait())

   TypeScriptCopy

   const run = await kl.pipelines.run({ owner: "acme", pipeline: "site-check", inputs: { target: "prod" } });console.log(await run.wait());

   GoCopy

   run, err := kl.Pipelines.Run(kloudlite.RunPipeline{Owner: "acme", Pipeline: "site-check", Inputs: map[string]string{"target": "prod"}})if err != nil {    log.Fatal(err)}state, err := run.Wait(kloudlite.WaitRun{})if err != nil {    log.Fatal(err)}fmt.Println(state)

   RubyCopy

   run = kl.pipelines.run(owner: "acme", pipeline: "site-check", inputs: { "target" => "prod" })puts run.wait

   JavaCopy

   var run = kl.pipelines().run(new Kloudlite.RunPipeline().owner("acme").pipeline("site-check").input("target", "prod"));System.out.println(run.waitFor());

   CLICopy

   RUN=$(kloudlite pipelines run acme/site-check --input target=prod --json | jq -r .id)kloudlite pipelines wait-run acme site-check "$RUN"

Close-out

1. Make sure that announce is success for prod.
2. Start a run with no inputs. Make sure that announce is skipped, because the default target is staging.

Run a job for each value of a matrix

Reason for the task: Run the same work on several versions of a tool, or on several data sets, in parallel.

Job set-up

• Access: you. For a team, an admin.
• References: Matrix, Pipeline definition – Matrix.

Procedure

1. Write this definition to python-check.json. The axis python names a nixpkgs package for each instance.

   jsonCopy

   {  "name": "python-check",  "jobs": {    "test": {      "matrix": { "python": ["python311", "python312"] },      "packages": ["${{ matrix.python }}"],      "steps": [{ "name": "version", "run": "python3 --version" }]    }  }}

2. Create the pipeline from the file, and start a run. See Start a run and follow it.

3. Read the log of one instance. Use the key of the instance as the job.

   PythonCopy

   print(run.logs(job="test (python312)"))

   TypeScriptCopy

   console.log(await run.logs({ job: "test (python312)" }));

   GoCopy

   out, err := run.Logs("test (python312)")if err != nil {    log.Fatal(err)}fmt.Print(out)

   RubyCopy

   puts run.logs(job: "test (python312)")

   JavaCopy

   System.out.print(run.logs("test (python312)"));

   CLICopy

   kloudlite pipelines logs acme python-check "$RUN" "test (python312)"

Close-out

1. Make sure that the run lists the jobs test (python311) and test (python312).
2. Make sure that each log shows the matching Python version.

Set the runtime and the time limit of a job

Reason for the task: Run a job in a container image of your choice, and stop it when it takes too long.

Job set-up

• Access: you. For a team, an admin.
• References: Runtime.

Procedure

1. Give the job an image and a timeoutMinutes. Use only one of packages and image.

   jsonCopy

   {  "name": "image-check",  "jobs": {    "check": {      "image": "python:3.12-slim",      "timeoutMinutes": 30,      "steps": [{ "name": "version", "run": "python -c 'import sys; print(sys.version)'" }]    }  }}

2. Create the pipeline from the file, and start a run. See Start a run and follow it.

Close-out

1. Read the log of check. Make sure that it shows Python 3.12.

Save the home of a job

Reason for the task: Keep the files of a job as a version of a volume. Start a later job or a later run from them.

Job set-up

• Access: you. For a team, an admin.
• References: Home, Volumes, Pipeline definition – Jobs.

Procedure

1. Write this definition to go-base.json. The job base saves its home to go-base; the job build starts from it.

   jsonCopy

   {  "name": "go-base",  "jobs": {    "base": {      "packages": ["go"],      "saveTo": "go-base",      "steps": [{ "name": "fetch", "run": "mkdir -p cache && date > cache/stamp" }]    },    "build": {      "needs": ["base"],      "fromSnapshot": "go-base",      "steps": [{ "name": "read", "run": "cat cache/stamp" }]    }  }}

2. Create the pipeline from the file, and start a run. See Start a run and follow it.

3. Show the versions of the volume.

   PythonCopy

   for v in kl.volumes.versions(owner="acme", name="go-base"):    print(v["version"], v["phase"], v["saved_by"])

   TypeScriptCopy

   console.log(await kl.volumes.versions({ owner: "acme", name: "go-base" }));

   GoCopy

   versions, err := kl.Volumes.Versions("acme", "go-base")if err != nil {    log.Fatal(err)}fmt.Println(versions)

   RubyCopy

   puts kl.volumes.versions(owner: "acme", name: "go-base")

   JavaCopy

   System.out.println(kl.volumes().versions("acme", "go-base"));

   CLICopy

   kloudlite volumes versions acme/go-base

Close-out

1. Make sure that each run adds a version. Make sure that saved_by names the job base.
2. Read the log of build. Make sure that it shows the date that base wrote.
3. Make sure that the run shows saved on base and home on build with the same version.

Troubleshooting