YAML and JSON pipelines

Use compilePipelineDocument(document, registry) from tubeless/project to turn a parsed YAML or JSON document and a registry of functions into ordinary pipelines. The document owns step identities, dependencies, targets, and policies; application code owns the handlers, schemas, and command parameters.

The project compiler remains dependency-free. It does not parse YAML, import modules, or run application code. Use your runtime’s YAML support or a parser of your choice. JSON works through JSON.parse without another package.

Validate quickly

Validate a document without importing any handlers:

tubeless validate pipelines.yaml
tubeless validate --json pipelines.yaml

In this repository, use make validate FILE=examples/declarative/peloton.yaml, or bunx tubeless validate --json examples/declarative/peloton.yaml. From a Tubeless checkout, bun run tubeless -- runs the local build. Exit code 0 means valid structure; 4 means a file, parse, or document validation failure; 1 means invalid command usage. JSON output includes ok, the file path, pipeline IDs and metadata on success, or a diagnostic with a document path when available.

This checks structure and metadata only. To check registered handler names, dependency references, cycles, and targets, load the compiled command with tubeless inspect or tubeless plan. Domain schemas run only during execution. The CLI’s Bun YAML parser can overwrite duplicate mapping keys; choose a parser with duplicate-key rejection when that check is required.

Downloadable JSON Schema

The pipeline document JSON Schema describes version 1 using JSON Schema draft 2020-12. The website serves the same file at https://tubeless.io/schemas/pipeline-document-v1.schema.json, with a current alias at https://tubeless.io/pipeline-document.schema.json. It is also packaged at docs/pipeline-document.schema.json. The website’s llms.txt links to the versioned URL for agents. Validate parsed YAML against it using a JSON Schema validator with date-format checking enabled.

For YAML language-server autocomplete and diagnostics, associate the schema:

# yaml-language-server: $schema=https://tubeless.io/schemas/pipeline-document-v1.schema.json
version: 1
metadata:
  name: Race weekend
  description: Prepare the riders and publish a validated start list.
  authors: [Race operations]
  date: "2026-09-17"
pipelines:
  # Pipeline definitions go here.

JSON documents can use a top-level $schema string; YAML documents may include it too. Tubeless accepts that association but never fetches it. version selects the document format.

Document metadata

The optional metadata object supports name, description, authors (an array of names or organizations), and date (a quoted YYYY-MM-DD calendar date). All fields are optional. Dates and authors are maintained by the author; they are not inferred from Git and do not schedule execution.

tubeless validate --json preserves and reports metadata. Document compilation produces the same pipelines regardless of metadata. The compiled value exposes validated, immutable metadata, including an immutable authors snapshot. Reuse its name and description when registering a project:

import { compilePipelineDocument, defineProject } from "tubeless/project";

const compiled = compilePipelineDocument(document, registry);
const project = defineProject("data-jobs", compiled.pipelines, {
  name: compiled.metadata?.name,
  description: compiled.metadata?.description,
});

// Ordinary object construction controls overrides.
const production = defineProject("production", compiled.pipelines, {
  ...compiled.metadata,
  name: "Production data jobs",
  cwd: "./jobs",
});

Project IDs and execution cwd belong to registration. Project metadata is immutable; project.name defaults to its ID when no name is supplied. Passing undefined uses that default and leaves description absent; there is no implicit metadata inheritance. Document authors and date remain available as descriptive data on compiled.metadata. Each pipeline definition may separately declare name and description; compiled commands inherit them. Top-level document metadata does not override pipeline labels, IDs, options, or run timestamps.

For custom CLI inputs, build a command directly from compiled.get(id). It can run independently or be registered directly in the project’s entry list:

import { definePipelineCommand } from "tubeless/cli";

const pipeline = compiled.get("yaml-import");
const command = definePipelineCommand(pipeline, {
  params: { lines: { type: "string" } },
  mapOptions: ({ lines }) => ({ lines: lines.split(",") }),
});

export default defineProject("yaml-jobs", [command]);

Only the selected pipeline is registered. Its compiled children still run through composition, without becoming separate project entries.

Try the example

From the repository root, use the registered examples:

bunx tubeless plan --project examples/project/tubeless.project.ts yaml-import --target normalize --explain
bunx tubeless run --project examples/project/tubeless.project.ts yaml-import -- --lines " Alpha , Beta , "
bunx tubeless ui examples/project/tubeless.project.ts

Studio lists Import rows from YAML and Preview rows from YAML. Their command parameters supply the input forms; the normal preview and run controls use the compiled pipelines. Browser launches record the same events as TypeScript-authored pipelines.

The loader and commands use Bun’s native YAML imports. The document describes four pipelines sharing an explicit handler registry. Its import command uses fromPipeline, whose child uses forEachPipeline to normalize each row. Both composed steps use registered application adapters, and empty input is an intentional policy skip. Edit the YAML and restart the command or Studio to load the new definition. The CLI accepts the command module or registered ID, not a bare YAML file.

Peloton demo

For a larger graph, try YAML Peloton. Its seven-step document declares rider discovery, weather, kit normalization, bike inspection, a separate car audit, tech validation, and publication. The handlers simulate all work; no network requests or filesystem writes occur.

# Visible progress, concurrent bike inspections, and a simulated radio retry
bun examples/yaml-peloton.ts --delay 200 --concurrency 2

# Substitute the weather preview handler and skip publication
bun examples/yaml-peloton.ts --dry-run --delay 0

# Select validation without publishing or auditing cars
bun examples/yaml-peloton.ts --target validate-tech --delay 0

# An independent audit failure allows publication, but the run still fails
bun examples/yaml-peloton.ts --fail-audit --continue-on-error --delay 0

# Tech failure blocks publication even when independent work may continue
bun examples/yaml-peloton.ts --fail-tech --continue-on-error --delay 0

Both failure examples intentionally exit with an execution error. The finalizer returns a partial summary so dry runs and target selection do not require a published start list. Option and rider-output schemas validate the dynamic document boundaries.

Open make ui STUDIO=examples/project/tubeless.project.ts and select Peloton from YAML to use the same demo in Studio. Its form exposes delay, inspection concurrency, and both failure switches. Keep a nonzero delay to watch progress or try cancellation. Plan without running any handlers:

make plan FILE=yaml-peloton PROJECT=examples/project/tubeless.project.ts ARGS="--target publish-start-list --explain"

The publication target selects the tech gate and its prerequisites, but omits the independent audit. Per-rider inspection runs through runConcurrent inside one ordinary step, with explicit progress details and retry events. This demo chooses lightweight in-handler concurrency; the smaller YAML recipe shows declarative child-pipeline composition and fan-out when each item needs its own child lifecycle.

Document format

version: 1
pipelines:
  import:
    optionsSchema: lines
    targets: [normalize]
    steps:
      - id: normalize
        fromPipeline:
          pipeline: normalize-all
          adapter: allRows
        skip: noRows
    finalize:
      run: normalizedRows
      requireOutputs: [normalize]

  normalize-all:
    optionsSchema: lines
    steps:
      - id: normalize
        forEachPipeline:
          pipeline: normalize-one
          adapter: eachRow
        skip: noRows
    finalize:
      run: normalizedRows
      requireOutputs: [normalize]

  normalize-one:
    optionsSchema: line
    steps:
      - id: normalize
        run: normalizeLine
    finalize:
      run: normalizedLine
      requireOutputs: [normalize]

Each key in pipelines is a pipeline ID. Step IDs are local to that pipeline; registry names can be reused across pipelines. Each pipeline requires at least one step and a finalizer. Step and pipeline forward references are allowed. Dependencies determine execution order using the existing engine, not the order in the file alone. Composition cycles are rejected.

Every step declares exactly one of run, fromPipeline, or forEachPipeline. Child references resolve only pipelines in the same document. The adapter is application code: a fromPipeline adapter supplies mapOptions and optional controls and mapResult; a forEachPipeline adapter supplies items, key, mapOptions, and optional controls, concurrency, progress, and mapResult. These are the same hooks as the TypeScript builders, so a static adapter concurrency must be a positive integer and is rejected at the document reference. Parent plans still expose one opaque wrapper with nestedPipeline metadata, and execution retains the normal nested progress and failure behavior.

Omitted dryRun runs the normal handler or child workflow during a dry run. Mark writes with dryRun: skip; ordinary steps can instead supply a side-effect-free preview handler. A skipped required output can prevent finalization. A runtime skip is a successful policy skip, can publish a value, and unlocks dependents. The compiler preserves the engine’s existing semantics. See core concepts.

Compile and select pipelines

import { compilePipelineDocument } from "tubeless/project";

// `document` is the unknown result of your YAML or JSON parser.
// `registry` explicitly imports and registers your application functions.
const compiled = compilePipelineDocument(document, registry);
const pipeline = compiled.get("import");

const plan = pipeline.plan({ targets: ["normalize"] });
const result = await pipeline.runOrThrow({ lines: [" Alpha ", "Beta"] });

Step handlers receive (inputs, context). Required and available optional dependency outputs are keyed by step ID, e.g. inputs.load. Domain options live in context.options. Use the normal context.log, context.signal, context.sleep, and progress APIs. Finalizers receive (outputs, context); without requireOutputs, they must handle absent outputs themselves.

compilePipelineDocument validates the document and compiles its pipelines. It returns an immutable CompiledPipelineDocument with a readonly pipelines collection, get(id), and optional PipelineDocumentMetadata. Lookup throws a clear error for an unknown ID. Repeated lookup, collection entries, and references from parent pipelines share one compiled instance per pipeline ID. Compilation does not register a project; defineProject collects whichever existing pipelines you choose. The registry has separate steps and finalizers function maps, plus optional skipPredicates, fromPipelineAdapters, forEachPipelineAdapters, optionsSchemas, and schemas maps. Schemas use Standard Schema v1. optionsSchemas must accept and produce objects, matching Tubeless domain options. schemas can validate any step-output or final-result shape. Functions are resolved once during compilation; modifying a registry later does not replace handlers in an already compiled pipeline.

Skip predicates receive (inputs, context) and return the normal StepSkipDecision. A single-child adapter’s mapOptions and dynamic controls receive the same arguments. A fan-out adapter receives the normal items, key, mapOptions, controls, and result-mapping arguments documented for forEachPipeline. Keep stable and unique keys, and return the complete parent-facing result array from a valued fan-out skip.

YAML cannot provide TypeScript’s inferred graph wiring. Handler inputs and pipeline results are unknown, handler options are object, and pipeline IDs are string. Narrow them or validate at the application boundary. Schemas validate runtime values; the compiler does not prove compatibility between two schemas.

User decisions, confirmations, and text

Declarative pipelines can wait for user input through registered async run handlers. Use waitForInput(...).run from createSteps to give those handlers the same waiting progress and cancellation behavior as TypeScript input steps. The application supplies a terminal, browser form, or other input adapter.

Version 1 uses ordinary run references; it has no YAML waitForInput, question, or choices fields. Messages, choices, input normalization, and UI rendering belong to the registered handler and its adapter. CLI and Studio do not automatically render these questions.

For example, a confirmation can refer to an upstream result:

- id: approval
  run: confirmProceeding
  dependsOn: [output]
  outputSchema: yesNo
  dryRun: skip
- id: proceed
  run: proceed
  dependsOn: [output, approval]
  skip: declined
  dryRun: skip

Register the input handler, output schema, and decline predicate:

import { createSteps } from "tubeless";
import type { ProjectRegistry } from "tubeless/project";

const { waitForInput } = createSteps<object>();
const registry: ProjectRegistry = {
  steps: {
    prepareOutput: () => "xyz",
    confirmProceeding: waitForInput("approval", {
      read: ({ output }: Record<string, unknown>, context) => {
        if (typeof output !== "string") throw new Error("Expected output");
        return readFromYourUI(`Output was ${output}. Confirm proceeding?`, context.signal);
      },
    }).run,
    proceed: ({ output }) => output,
  },
  schemas: {
    yesNo: {
      "~standard": {
        version: 1,
        vendor: "example",
        validate: (value) =>
          typeof value === "boolean"
            ? { value }
            : { issues: [{ message: "Expected a boolean confirmation" }] },
      },
    },
  },
  skipPredicates: {
    declined: ({ approval }) => (approval === false ? "User declined" : false),
  },
  finalizers: {},
};

The UI adapter should return an actual boolean, converting its yes/no text or button selection as needed. A false confirmation is a valid answer: the predicate skips proceed and the pipeline can finish successfully. Invalid answers fail output validation before dependent work starts. To keep asking after invalid input, put that retry behavior in the adapter.

The YAML compiler registers the helper’s .run callback only. Declare dependsOn, outputSchema, and dryRun in the document; the helper’s default dry-run skip does not carry over. Use dryRun: skip to avoid prompting, or dryRun: { run: previewApproval } with a registered preview handler. If skipped input leaves outputs absent, choose a finalizer that handles that absence, or require those outputs when a missing answer must prevent finalization.

While input is pending the run stays active. Forward context.signal to the adapter so cancellation releases its UI/I/O resources; a late reply is ignored. The application owns persistence and recovery across process restarts.

The complete YAML input recipe and document demonstrate all three question types, confirmation of xyz, declined execution, and a finalizer that distinguishes an unanswered question from a deliberate false.

Validation and limits

Parsing errors belong to the YAML parser. Configure duplicate-key rejection when available: a compiler cannot recover keys that a parser already overwrote. Bun’s native YAML loader does not provide this guarantee. For strict authoring, use a parser configured to reject duplicate mapping keys before compilation.

Malformed documents, unknown fields, and unresolved names throw PipelineDocumentError, with code TUBELESS_DOCUMENT_INVALID and a document path. Source line/column locations are not retained by this compiler. Duplicate step IDs, cycles, conflicting edges, and invalid target/finalizer combinations go through core and throw the existing PipelineDefinitionError.

Compilation and plan() never invoke handlers or schemas. Business validation happens during run(). Loading an application registry can still execute module initialization code. Registry functions remain trusted application code.

Version 1 supports ordinary steps, single-child composition, child fan-out, and runtime skip predicates. Remote steps, expressions, inline code, and external pipeline references are not document features. Use TypeScript authoring for those workflows. Register selected compiled pipelines with defineProject and export that project from a project module for CLI and Studio loading. Pipelines whose options schemas expose Standard JSON Schema input metadata receive automatic commands. Custom or schema-less inputs need definePipelineCommand adapters registered directly in the project’s entry list. A YAML file alone does not grant execution access or define a new Studio protocol.

Graph metadata on individual pipelines and steps is currently a TypeScript API feature. Document version 1 rejects those fields; its document-level metadata remains unchanged. See graph metadata compatibility.