# Agent guide for `tubeless`

Use this guide when generating or modifying pipelines, pipeline-backed scripts,
or shared helpers in this repository.

## Workflow

1. Read the [recipe index](https://tubeless.io/docs/recipes.md) and open the smallest matching example.
2. Copy file layout and stable IDs from the
   [project manifest](https://github.com/chetmancini/tubeless/blob/main/examples/catalog/tubeless.project.ts).
3. Declare stable IDs and operational descriptions. Add `name` only when printed
   output needs a friendlier display name.
4. Model data dependencies before failure policy or CLI concerns.
5. Typecheck the example or consumer, then run focused tests.
6. Run `make check` from the package root after changing the package or its learning surface.

## Primitive selection

- Use `createSteps<TDomainOptions>()` once per pipeline and `definePipeline` once
  after declaring its steps. Domain option types contain domain input only;
  callers pass those options and optional built-in controls to
  `run(options, controls?)`, while `plan(controls)` accepts controls alone.
- Treat a `PipelineDefinitionError` during module loading as an authoring bug;
  static graph mistakes are rejected when `definePipeline` is called.
- Use `dependsOn` when the output is required, `optionalDependsOn` when absence
  is expected, and `skipAfterFailureOf` for a failure gate that supplies no data.
- Use `createSteps(optionsSchema)` when external domain options need Standard
  Schema validation or transformation. Add `outputSchema` only at step
  boundaries that receive untrusted or independently checked values, and
  `resultSchema` when the finalized public result must be checked. Core imports
  no schema library.
- Set `dryRun: "skip"` on filesystem writes, database mutations, publication,
  email, and other steps whose normal `run` must not execute in a dry run. Use a
  typed `dryRun` handler when the step should produce a preview value instead.
- Use `step.skippable` only for an intentional successful outcome. Handle its
  resulting `T | undefined` output type explicitly.
- Use `fromPipeline` for one independently useful child workflow and
  `forEachPipeline` for runtime fan-out with stable keys and bounded concurrency.
  Use `forEachPipeline.skippable` only when the whole fan-out may be intentionally
  omitted; handle its `readonly T[] | undefined` output explicitly.
  Async `fromPipeline` result mappings publish resolved values; use that resolved
  shape for dependent inputs and policy-skip values.
  Parent plans keep these steps opaque but expose `nestedPipeline` with the child
  pipeline id, declared step ids, and single/fan-out mode for presentation.
  The interactive CLI automatically expands selected child steps and fan-out
  items into nested progress rows and retains completed states. Inner progress
  counts and details propagate through both adapters; do not forward raw child
  hooks or duplicate this bookkeeping in consumers. Fan-outs materialize up to
  32 live item groups by default and emit the full retained tree once at settlement.
  Set `progress.detailLimit` to override that live cap and cap the final snapshot.
- Use `fromRemote` for a unit of work that lives on another engine. Required
  fields are `adapter`, `mapInput`, and `outputSchema`. Omitting `dryRun`
  contacts the engine during a pipeline dry run; the adapter and remote
  worker must honor `context.dryRun` and authors prove the flag crossed the
  boundary in `mapInput`. Host-embed with `pipeline.runOrThrow` and pass
  `runId` / `parentRunId` when the graph must outlive the process. Parent
  plans expose `remote` with `engine` and optional `target`. Adapters may
  forward remote lines through `context.log` and must rethrow remote
  failures as `Error` with `cause` / `code`. Copy the native-fetch boundary in
  [`remote-steps.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/remote-steps.ts), including validation of
  unknown JSON and forwarding the signal. Use
  [`host-embedding.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/host-embedding.ts) for host-owned invocation;
  correlation IDs do not provide persistence or checkpoint/resume.
- Use `runConcurrent` for bounded lightweight functions that do not need child
  lifecycle events. Use `runConcurrentSettled` when the caller needs completed
  results and the first failure instead of a throw.
- Use `definePipelineCommand` for scripts centered on a pipeline. Do not parse
  `process.argv` manually or redeclare built-in dry-run, `--step`, or `--target`
  flags. `mapOptions`, validation, and hooks receive `stepIds` and `targets`,
  not `step` or `target`. Omit `mapOptions` when validated flags already satisfy
  same-name pipeline options; provide it when names, types, defaults, or derived
  values differ.
  The returned command exposes an immutable `descriptor`; UI adapters should
  render that structured parameter contract instead of parsing help text.
  Use `command.plan()` or `tubeless plan` for a selection-only preview. Do not
  simulate planning with `--plan`.
- Use `pipeline.toMermaid()` or `command.toMermaid()` when documentation needs
  the static graph; do not duplicate dependency edges by hand.
- Use `tubeless list` for the explicit project command inventory. Use
  `tubeless inspect <registered-id>` for a registered module inventory,
  `tubeless plan` to preview selection without domain options or execution, and
  `tubeless graph` when generating documentation. `inspect`, `plan`, and `graph`
  accept a pipeline or a marked command and prefer the command when both are
  exported. Let the workbench discover the sole matching export or pass
  `--export`.
- Use `tubeless run <registered-id> -- <command-args>` for project commands, or
  the existing file form only for modules exporting a
  `definePipelineCommand`. Keep application flags after `--`; the command must
  continue to own domain validation and option mapping. Pass `--trace <path>` for
  NDJSON traces (`-` writes NDJSON to stdout and moves command output to stderr)
  and `--store` for SQLite; they compose. Use `tubeless history` to list or show
  recorded runs from SQLite, or pass `--trace` to inspect a finished NDJSON
  artifact without importing it.
  Filter shared history with `--pipeline <recorded-pipeline-id>` in any output
  mode. This is the pipeline definition's ID, not its registered command ID.

## Runtime rules

- Library entrypoints are ESM-only and require Node.js 22 or later. The
  `tubeless` CLI requires Bun 1.3.14 or later; its `#!/usr/bin/env node`
  trampoline relaunches the binary with Bun under Node and reports an
  actionable install message when Bun is missing.
- Use `context.log`, never direct `console` calls inside steps.
- Pass `context.signal` into network calls, batching, retry, rate limiting, and
  long waits. Use `context.sleep` for retry-aware or testable delays.
- Call `context.reportProgress` for long loops and `context.reportAttempt` for
  retries that operators should see.
- Resolve relative files from `context.cwd`; prefer helpers from
  `tubeless/node`.
- Use `runOrThrow` when every step must succeed and the caller expects a value.
  It always throws for an unsuccessful run, including `continueOnError` runs.
  Use `run` when the caller must inspect failures, skips, timings, or best-effort
  output. Its versioned `PipelineRun` exposes `runId`, terminal status and
  timestamps, errors, and timestamped step reports with correlated attempt IDs;
  structural skips have no attempt ID or start timestamp. Use hooks or tracing
  for streaming logs and progress. Exporter failures warn once per emitter and
  do not fail the run; pass `onExporterError` to observe the first drop. Use
  `plan` when nothing may run.
- Declare supported downstream goals with `targets: [step]` on
  `definePipeline`, then select their literal IDs through run controls. Required
  inputs and failure gates are selected recursively. Use `stepIds` only when
  exact low-level filtering of any step is intentional; never combine the two.
- Read `PipelinePlanStep.selectionReasons` when explaining selection. It already
  includes originating targets and immediate dependents; do not reconstruct
  provenance by walking dependency arrays in application or CLI code. Use
  `renderPipelinePlan` from `tubeless/render` for shared human or JSON output
  instead of maintaining another selection-reason formatter. Use
  `createPipelineReporter` / `createRunReporter` from `tubeless/reporter` for
  TTY presentation; do not import reporters from the root kernel.
- Use focused hooks for ordinary observation: `onStepStart`,
  `onStepProgress`, `onStepComplete`, `onStepSkip`, `onStepCancel`, and
  `onStepFail`. Their event metadata is already narrowed. Use additive
  `onStepStatus` only when one consumer genuinely needs the whole discriminated
  lifecycle, such as a status-aware renderer or event store.
- Use `createPipelineTestRuntime` from `tubeless/testing` for deterministic
  pipeline tests. Inspect its structured logs, statuses, and latest progress;
  keep test-framework matchers outside the package.
- Branch on `PipelineError.code`, `phase`, and `kind`, never message prose.
  Underlying thrown codes live in `sourceCode`. Failed and cancelled reports
  expose the structured error under `error`; skipped reports expose `reason`,
  optional `message`, and optional `dependencyId`. `PipelineError.cause` is a
  bounded JSON-safe snapshot; a thrown `PipelineExecutionError` retains the
  original value through native `Error.cause`. Use `renderPipelineError` when a
  diagnostic crosses a human or JSON presentation boundary.
- Inspect `error.fanOut` in reports or recorded trace history for bounded keyed failures from `forEachPipeline`. Check
  `omittedFailureCount` and `keyTruncated` before selecting rerun inputs; unstarted
  items are not failures. Reruns remain caller-owned new runs.
- Standard Schema failures use `kind: "validation"`, retain normalized `issues`,
  and have boundary-specific codes for options, step outputs, and final results.
  Async schemas run during `run`; synchronous `plan()` previews graph and
  selection only and never invokes schemas.
- Wrap normal finalizers in `requireOutputs` when a valid result requires
  specific step outputs. Use a plain finalizer only when partial output is a
  valid domain result.
- Preserve the dependency-free runtime. Keep application telemetry SDKs at the
  exporter boundary. Use `composeTraceExporters` from `tubeless/tracing` when
  one run must fan out to multiple destinations; `onExporterError` reports the
  first partial drop, the failed destination is retired, and healthy exporters
  keep receiving events.
- Keep durable local observation opt-in. Use the append-only adapter from
  `tubeless/run-store/sqlite` or `tubeless run --store`; inspect recorded runs
  with `tubeless history`. Use the strictly read-only adapter from
  `tubeless/run-store/ndjson`, `tubeless history --trace`, or the `--trace`
  option to `tubeless ui` for a finished portable trace. Treat trace files as sensitive: logs,
  errors, and attributes are displayed as recorded, and malformed or oversized
  artifacts are rejected. `tubeless history` inspects a finished artifact and
  refuses a store with a live writer or multiple hard links. SQLite `export()`
  may return before the row is on disk. Other connections cannot see that tail
  until a batch of 64, `flush()`, same-instance `listEvents`/`clearHistory`, or
  `close()`. A crash can lose up to 63 unflushed events; `tubeless run --store`
  flushes at completion so finished runs are durable. Do not make pipeline
  definitions depend on storage or the studio. Recorded history keeps the last
  `reportProgress` `details` plus `detail_count`, and opaque child steps keep
  `nested_pipeline` with the original `step_count`. Studio renders those
  snapshots; it does not flatten child DAGs into the parent step.
  Observed definitions pick the latest `pipeline.started` by `timestampMs`, then
  store-local id.
- Use `createPipelineRunProjector` from `tubeless/run-store` when a custom
  reader pages `listEvents({ afterId })`. Append each newer page and call
  `snapshot()`; a refresh with no new ids returns the cached view. Duplicate
  and out-of-order ids are ignored; `0` is a valid first id. Pass
  `{ retainLogs: false }` only when the snapshot should keep `logCount`
  without log bodies. Use `projectPipelineRunStore` only for a one-shot fold
  of a complete list. See
  [`local-observability.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/local-observability.ts).
- Pass caller-owned `runId` and `parentRunId` values through `PipelineContext`
  when joining an external execution tree. Do not derive correlation from step
  IDs or timestamps.
- Treat `tubeless ui` as a local projection with no execution capability by
  default. Use `definePipelineProject` for a checked-in command catalog with
  stable registered IDs, and register only explicit `definePipelineCommand`
  modules; never make execution require the studio server or infer executable
  modules from observed history.
  Every studio request, including reads, must send a `Host` that matches the
  bound authority, or, on a wildcard bind, `localhost` or a literal IP on the
  same port. Browser plan, launch, cancel, and clear-history also send
  `x-tubeless-studio-*` headers; those are same-origin guards, not
  authentication. Studio HTTP errors contain stable `code`, `message`, `hint`,
  and a backward-compatible `error` alias; the local-only contract is published
  at `https://tubeless.io/openapi.json`. Cancel a live top-level launch from the running detail pane;
  that abort is process-local, leaves sibling launches running, and is not
  crash-resume.

## Failure and safety rules

- Do not use `step.skippable` to swallow an exception.
- Do not treat dry run as rollback; external side effects require
  `dryRun: "skip"` or a side-effect-free custom `dryRun` handler.
- Do not publish after a failed or cancelled validation step. Make validation
  required, or use `skipAfterFailureOf` when its output is intentionally optional;
  the gate blocks both unsuccessful terminal statuses.
- Do not hide absent required finalizer outputs behind defaults. Declare them
  with `requireOutputs`; handle optional outputs explicitly.
- Keep step IDs stable because reports, hooks, traces, and CLI selection use them.
- Treat `name` as presentation only. Dependencies, outputs, traces, and CLI
  selection continue to use the stable step ID.

## Required references

- Read [core concepts](https://tubeless.io/docs/concepts.md) for skip, failure, or selection changes.
- Read [the CLI](https://tubeless.io/docs/cli.md) for list, inspect, plan, graph, run, history, and exit codes.
- Read [the studio](https://tubeless.io/docs/studio.md) before changing `tubeless ui`,
  `definePipelineProject`, or `definePipelineStudio` compatibility.
- Read [child composition](https://tubeless.io/docs/child-pipeline-composition.md) before changing child
  propagation, progress, or parent/child selection.
- Read the relevant executable example linked from the
  [recipe index](https://tubeless.io/docs/recipes.md) before writing new usage.
- Copy consumer layout, export names, and IDs from the
  [project manifest](https://github.com/chetmancini/tubeless/blob/main/examples/catalog/tubeless.project.ts).
- Use the [generated API inventory](https://tubeless.io/docs/api-reference.md) only to verify exports;
  it is not implementation guidance.

## Validation

For package changes:

```sh
make check
```

For a consumer-only change, run its focused tests and the repository typecheck.
If the public surface changes intentionally, regenerate the checked API artifacts
with `bun run api:generate` from the package root.