Pipelines

Description and operation

General

A pipeline automates work that runs without a person at a terminal. It is a definition of jobs, written as JSON data and stored in Kloudlite. Each job runs its commands in its own sandbox, with the packages, image or snapshot that it names. You create, change and start pipelines from the SDK, the CLI or the API. The automation lives in code, like the rest of your platform.

A pipeline starts when you call it, on a schedule, from a webhook call, or after another pipeline ends. Each start makes a run. A run records which version of the definition it used, so you always know what ran.

The exhaustive format is in Pipeline definition. The routes are in the pipelines API.

To do these tasks with an AI agent, see Agentic workflows – General.

How a run works

1. A start request reads one stored version. It checks the inputs and the secrets, and expands the matrix into jobs.

2. The platform records the run as queued and answers 202 at once. If the version cannot run, the call fails with 422 and records nothing.

3. A worker takes the run. Each job starts when the jobs in its needs have ended, with up to 8 jobs of a run at the same time.

4. Each job runs in a new sandbox. Its steps run in order in that sandbox, each in its own shell, so cd and export do not carry to the next step. Its output goes to the log of the job.

5. When the last job ends, the run ends as success, failure or cancelled. The platform ends every sandbox of the run.

Jobs that do not need each other run in parallel. A job whose needed job failed is skipped, unless its if condition says otherwise. Files do not pass from one job to another: each job has its own sandbox.

States

Access

A pipeline belongs to an owner: you, or a team. The owner of a pipeline also owns its runs.

The web console shows pipelines, versions and runs, and has no control that changes them. Use the SDK, the CLI or the API for every change.

Quota and limits

The sandboxes of a run count against the quota of the owner of the pipeline.

Where to go next

• Definitions and versions: change a pipeline and go back to an earlier version.
• Triggers and inputs: start runs on a schedule, from a webhook or after another pipeline.
• Runs: pin a version, follow logs and cancel a run.

Tasks

Create a pipeline

Reason for the task: Store a definition of jobs that the platform runs every night and on request.

Job set-up

• Access: you. For a team, an admin.
• References: Pipeline definition, POST /v1/pipelines/{owner}, SDK pipelines.create.

Procedure

1. Write this definition to the file nightly-report.json.

   jsonCopy

   {  "name": "nightly-report",  "triggers": { "schedule": ["0 2 * * *"] },  "inputs": {    "source": { "default": "https://api.github.com/repos/nixos/nixpkgs" }  },  "jobs": {    "fetch": {      "packages": ["curl", "jq"],      "steps": [        { "name": "download", "run": "curl -fsSL \"$SOURCE\" -o data.json", "env": { "SOURCE": "${{ inputs.source }}" } },        { "name": "summarize", "run": "jq '{stars: .stargazers_count, issues: .open_issues_count}' data.json" }      ]    },    "report": {      "needs": ["fetch"],      "if": "always()",      "steps": [        { "name": "result", "run": "echo \"fetch ended with $RESULT\"", "env": { "RESULT": "${{ needs.fetch.result }}" } }      ]    }  }}

2. Create the pipeline from the file.

   PythonCopy

   import jsonfrom kloudlite import Kloudlitekl = Kloudlite()with open("nightly-report.json") as f:    doc = kl.pipelines.create(owner="acme", definition=json.load(f))print(doc["name"], doc["version"])

   TypeScriptCopy

   import { readFileSync } from "node:fs";import { Kloudlite } from "./index.ts";const kl = new Kloudlite();const definition = JSON.parse(readFileSync("nightly-report.json", "utf8"));const doc = await kl.pipelines.create({ owner: "acme", definition });console.log(doc.name, doc.version);

   GoCopy

   kl := kloudlite.New(kloudlite.Options{})raw, err := os.ReadFile("nightly-report.json")if err != nil {    log.Fatal(err)}var d kloudlite.PipelineDefinitionif err := json.Unmarshal(raw, &d); err != nil {    log.Fatal(err)}doc, err := kl.Pipelines.Create("acme", d)if err != nil {    log.Fatal(err)}fmt.Println(doc.Name, doc.Version)

   RubyCopy

   require "json"require_relative "kloudlite"kl = Kloudlite::Client.newdoc = kl.pipelines.create(owner: "acme", definition: JSON.parse(File.read("nightly-report.json")))puts doc.values_at("name", "version").join(" ")

   JavaCopy

   var kl = new Kloudlite();// The definition is JSON already, so send it as the body of the create route.System.out.println(kl.request(new Kloudlite.RawRequest().method("POST").path("/v1/pipelines/acme")    .body(Files.readString(Path.of("nightly-report.json")))));

   CLICopy

   kloudlite pipelines create --owner acme --file nightly-report.json

Close-out

1. Show the pipelines of the owner.

   PythonCopy

   print(kl.pipelines.list(owner="acme"))

   TypeScriptCopy

   console.log(await kl.pipelines.list({ owner: "acme" }));

   GoCopy

   list, err := kl.Pipelines.List("acme")if err != nil {    log.Fatal(err)}fmt.Printf("%+v\n", list)

   RubyCopy

   p kl.pipelines.list(owner: "acme")

   JavaCopy

   System.out.println(kl.pipelines().list("acme"));

   CLICopy

   kloudlite pipelines list --owner acme

2. Make sure that nightly-report shows latestVersion 1 and the schedule 0 2 * * *.

Start a run and follow it

Reason for the task: Run the pipeline now, without waiting for its schedule, and read the output of its jobs.

Job set-up

• Access: you. For a team, an active member.
• Conditions: the pipeline nightly-report exists. See Create a pipeline.
• References: POST /v1/pipelines/{owner}/{pipeline}/runs, SDK pipelines.run, Run.follow, Run.wait, Runs.

Procedure

1. Start a run of the newest version.

   PythonCopy

   run = kl.pipelines.run(owner="acme", pipeline="nightly-report")print(run.id, run.state)

   TypeScriptCopy

   const run = await kl.pipelines.run({ owner: "acme", pipeline: "nightly-report" });console.log(run.id, run.state);

   GoCopy

   run, err := kl.Pipelines.Run(kloudlite.RunPipeline{Owner: "acme", Pipeline: "nightly-report"})if err != nil {    log.Fatal(err)}fmt.Println(run.ID, run.State)

   RubyCopy

   run = kl.pipelines.run(owner: "acme", pipeline: "nightly-report")puts run.id, run.state

   JavaCopy

   var run = kl.pipelines().run(new Kloudlite.RunPipeline().owner("acme").pipeline("nightly-report"));System.out.println(run.id() + " " + run.state());

   CLICopy

   RUN=$(kloudlite pipelines run acme/nightly-report --json | jq -r .id)

2. Read the log of the job fetch while it runs.

   PythonCopy

   for chunk in run.follow(job="fetch"):    print(chunk, end="")

   TypeScriptCopy

   for await (const chunk of run.follow({ job: "fetch" })) process.stdout.write(chunk);

   GoCopy

   err = run.Follow(kloudlite.FollowLogs{Job: "fetch"}, func(chunk string) bool {    fmt.Print(chunk)    return true})if err != nil {    log.Fatal(err)}

   RubyCopy

   run.follow(job: "fetch") { |chunk| print chunk }

   JavaCopy

   run.follow("fetch").forEach(System.out::print);

   CLICopy

   kloudlite pipelines logs acme nightly-report "$RUN" fetch --follow

3. Wait for the run to end. The call returns the final state.

   PythonCopy

   print(run.wait())

   TypeScriptCopy

   console.log(await run.wait());

   GoCopy

   state, err := run.Wait(kloudlite.WaitRun{})if err != nil {    log.Fatal(err)}fmt.Println(state)

   RubyCopy

   puts run.wait

   JavaCopy

   System.out.println(run.waitFor());

   CLICopy

   kloudlite pipelines wait-run acme nightly-report "$RUN"

Close-out

1. Make sure that the final state is success. The log of fetch shows the stars and issues counts.
2. If the state is failure, read detail of the run and failedStep of each failed job. See Troubleshooting.

Troubleshooting