TUBELESS v0.2.2
Markdown

Agent guide for tubeless

Use this guide when generating or modifying pipelines, pipeline-backed scripts, or shared helpers in this repository.

Workflow

  1. Read the recipe index and open the smallest matching example.
  2. Copy file layout and stable IDs from the project manifest.
  3. Declare stable IDs and operational descriptions. Add name only when printed output needs a friendlier display name.
  4. Model data dependencies before failure policy or CLI concerns.
  5. Typecheck the example or consumer, then run focused tests.
  6. Run make check from the package root after changing the package or its learning surface.

Primitive selection

  • Use createSteps<TDomainOptions>() once per pipeline and definePipeline once after declaring its steps. Domain option types contain domain input only; callers pass those options and optional built-in controls to run(options, controls?), while plan(controls) accepts controls alone.
  • Treat a PipelineDefinitionError during module loading as an authoring bug; static graph mistakes are rejected when definePipeline is called.
  • Use dependsOn when the output is required, optionalDependsOn when absence is expected, and skipAfterFailureOf for a failure gate that supplies no data.
  • Use createSteps(optionsSchema) when external domain options need Standard Schema validation or transformation. Add outputSchema only at step boundaries that receive untrusted or independently checked values, and resultSchema when the finalized public result must be checked. Core imports no schema library.
  • Set dryRun: "skip" on filesystem writes, database mutations, publication, email, and other steps whose normal run must not execute in a dry run. Use a typed dryRun handler when the step should produce a preview value instead.
  • Use step.skippable only for an intentional successful outcome. Handle its resulting T | undefined output type explicitly.
  • Use fromPipeline for one independently useful child workflow and forEachPipeline for runtime fan-out with stable keys and bounded concurrency. Use forEachPipeline.skippable only when the whole fan-out may be intentionally omitted; handle its readonly T[] | undefined output explicitly. Async fromPipeline result mappings publish resolved values; use that resolved shape for dependent inputs and policy-skip values. Parent plans keep these steps opaque but expose nestedPipeline with the child pipeline id, declared step ids, and single/fan-out mode for presentation. The interactive CLI automatically expands selected child steps and fan-out items into nested progress rows and retains completed states. Inner progress counts and details propagate through both adapters; do not forward raw child hooks or duplicate this bookkeeping in consumers. Fan-outs materialize up to 32 live item groups by default and emit the full retained tree once at settlement. Set progress.detailLimit to override that live cap and cap the final snapshot.
  • Use fromRemote for a unit of work that lives on another engine. Required fields are adapter, mapInput, and outputSchema. Omitting dryRun contacts the engine during a pipeline dry run; the adapter and remote worker must honor context.dryRun and authors prove the flag crossed the boundary in mapInput. Host-embed with pipeline.runOrThrow and pass runId / parentRunId when the graph must outlive the process. Parent plans expose remote with engine and optional target. Adapters may forward remote lines through context.log and must rethrow remote failures as Error with cause / code. Copy the native-fetch boundary in remote-steps.ts, including validation of unknown JSON and forwarding the signal. Use host-embedding.ts for host-owned invocation; correlation IDs do not provide persistence or checkpoint/resume.
  • Use runConcurrent for bounded lightweight functions that do not need child lifecycle events. Use runConcurrentSettled when the caller needs completed results and the first failure instead of a throw.
  • Use definePipelineCommand for scripts centered on a pipeline. Do not parse process.argv manually or redeclare built-in dry-run, --step, or --target flags. mapOptions, validation, and hooks receive stepIds and targets, not step or target. Omit mapOptions when validated flags already satisfy same-name pipeline options; provide it when names, types, defaults, or derived values differ. The returned command exposes an immutable descriptor; UI adapters should render that structured parameter contract instead of parsing help text. Use command.plan() or tubeless plan for a selection-only preview. Do not simulate planning with --plan.
  • Use pipeline.toMermaid() or command.toMermaid() when documentation needs the static graph; do not duplicate dependency edges by hand.
  • Use tubeless list for the explicit project command inventory. Use tubeless inspect <registered-id> for a registered module inventory, tubeless plan to preview selection without domain options or execution, and tubeless graph when generating documentation. inspect, plan, and graph accept a pipeline or a marked command and prefer the command when both are exported. Let the workbench discover the sole matching export or pass --export.
  • Use tubeless run <registered-id> -- <command-args> for project commands, or the existing file form only for modules exporting a definePipelineCommand. Keep application flags after --; the command must continue to own domain validation and option mapping. Pass --trace <path> for NDJSON traces (- writes NDJSON to stdout and moves command output to stderr) and --store for SQLite; they compose. Use tubeless history to list or show recorded runs from SQLite, or pass --trace to inspect a finished NDJSON artifact without importing it. Filter shared history with --pipeline <recorded-pipeline-id> in any output mode. This is the pipeline definition's ID, not its registered command ID.

Runtime rules

  • Library entrypoints are ESM-only and require Node.js 22 or later. The tubeless CLI requires Bun 1.3.14 or later; its #!/usr/bin/env node trampoline relaunches the binary with Bun under Node and reports an actionable install message when Bun is missing.
  • Use context.log, never direct console calls inside steps.
  • Pass context.signal into network calls, batching, retry, rate limiting, and long waits. Use context.sleep for retry-aware or testable delays.
  • Call context.reportProgress for long loops and context.reportAttempt for retries that operators should see.
  • Resolve relative files from context.cwd; prefer helpers from tubeless/node.
  • Use runOrThrow when every step must succeed and the caller expects a value. It always throws for an unsuccessful run, including continueOnError runs. Use run when the caller must inspect failures, skips, timings, or best-effort output. Its versioned PipelineRun exposes runId, terminal status and timestamps, errors, and timestamped step reports with correlated attempt IDs; structural skips have no attempt ID or start timestamp. Use hooks or tracing for streaming logs and progress. Exporter failures warn once per emitter and do not fail the run; pass onExporterError to observe the first drop. Use plan when nothing may run.
  • Declare supported downstream goals with targets: [step] on definePipeline, then select their literal IDs through run controls. Required inputs and failure gates are selected recursively. Use stepIds only when exact low-level filtering of any step is intentional; never combine the two.
  • Read PipelinePlanStep.selectionReasons when explaining selection. It already includes originating targets and immediate dependents; do not reconstruct provenance by walking dependency arrays in application or CLI code. Use renderPipelinePlan from tubeless/render for shared human or JSON output instead of maintaining another selection-reason formatter. Use createPipelineReporter / createRunReporter from tubeless/reporter for TTY presentation; do not import reporters from the root kernel.
  • Use focused hooks for ordinary observation: onStepStart, onStepProgress, onStepComplete, onStepSkip, onStepCancel, and onStepFail. Their event metadata is already narrowed. Use additive onStepStatus only when one consumer genuinely needs the whole discriminated lifecycle, such as a status-aware renderer or event store.
  • Use createPipelineTestRuntime from tubeless/testing for deterministic pipeline tests. Inspect its structured logs, statuses, and latest progress; keep test-framework matchers outside the package.
  • Branch on PipelineError.code, phase, and kind, never message prose. Underlying thrown codes live in sourceCode. Failed and cancelled reports expose the structured error under error; skipped reports expose reason, optional message, and optional dependencyId. PipelineError.cause is a bounded JSON-safe snapshot; a thrown PipelineExecutionError retains the original value through native Error.cause. Use renderPipelineError when a diagnostic crosses a human or JSON presentation boundary.
  • Inspect error.fanOut in reports or recorded trace history for bounded keyed failures from forEachPipeline. Check omittedFailureCount and keyTruncated before selecting rerun inputs; unstarted items are not failures. Reruns remain caller-owned new runs.
  • Standard Schema failures use kind: "validation", retain normalized issues, and have boundary-specific codes for options, step outputs, and final results. Async schemas run during run; synchronous plan() previews graph and selection only and never invokes schemas.
  • Wrap normal finalizers in requireOutputs when a valid result requires specific step outputs. Use a plain finalizer only when partial output is a valid domain result.
  • Preserve the dependency-free runtime. Keep application telemetry SDKs at the exporter boundary. Use composeTraceExporters from tubeless/tracing when one run must fan out to multiple destinations; onExporterError reports the first partial drop, the failed destination is retired, and healthy exporters keep receiving events.
  • Keep durable local observation opt-in. Use the append-only adapter from tubeless/run-store/sqlite or tubeless run --store; inspect recorded runs with tubeless history. Use the strictly read-only adapter from tubeless/run-store/ndjson, tubeless history --trace, or the --trace option to tubeless ui for a finished portable trace. Treat trace files as sensitive: logs, errors, and attributes are displayed as recorded, and malformed or oversized artifacts are rejected. tubeless history inspects a finished artifact and refuses a store with a live writer or multiple hard links. SQLite export() may return before the row is on disk. Other connections cannot see that tail until a batch of 64, flush(), same-instance listEvents/clearHistory, or close(). A crash can lose up to 63 unflushed events; tubeless run --store flushes at completion so finished runs are durable. Do not make pipeline definitions depend on storage or the studio. Recorded history keeps the last reportProgress details plus detail_count, and opaque child steps keep nested_pipeline with the original step_count. Studio renders those snapshots; it does not flatten child DAGs into the parent step. Observed definitions pick the latest pipeline.started by timestampMs, then store-local id.
  • Use createPipelineRunProjector from tubeless/run-store when a custom reader pages listEvents({ afterId }). Append each newer page and call snapshot(); a refresh with no new ids returns the cached view. Duplicate and out-of-order ids are ignored; 0 is a valid first id. Pass { retainLogs: false } only when the snapshot should keep logCount without log bodies. Use projectPipelineRunStore only for a one-shot fold of a complete list. See local-observability.ts.
  • Pass caller-owned runId and parentRunId values through PipelineContext when joining an external execution tree. Do not derive correlation from step IDs or timestamps.
  • Treat tubeless ui as a local projection with no execution capability by default. Use definePipelineProject for a checked-in command catalog with stable registered IDs, and register only explicit definePipelineCommand modules; never make execution require the studio server or infer executable modules from observed history. Every studio request, including reads, must send a Host that matches the bound authority, or, on a wildcard bind, localhost or a literal IP on the same port. Browser plan, launch, cancel, and clear-history also send x-tubeless-studio-* headers; those are same-origin guards, not authentication. Studio HTTP errors contain stable code, message, hint, and a backward-compatible error alias; the local-only contract is published at https://tubeless.io/openapi.json. Cancel a live top-level launch from the running detail pane; that abort is process-local, leaves sibling launches running, and is not crash-resume.

Failure and safety rules

  • Do not use step.skippable to swallow an exception.
  • Do not treat dry run as rollback; external side effects require dryRun: "skip" or a side-effect-free custom dryRun handler.
  • Do not publish after a failed or cancelled validation step. Make validation required, or use skipAfterFailureOf when its output is intentionally optional; the gate blocks both unsuccessful terminal statuses.
  • Do not hide absent required finalizer outputs behind defaults. Declare them with requireOutputs; handle optional outputs explicitly.
  • Keep step IDs stable because reports, hooks, traces, and CLI selection use them.
  • Treat name as presentation only. Dependencies, outputs, traces, and CLI selection continue to use the stable step ID.

Required references

  • Read core concepts for skip, failure, or selection changes.
  • Read the CLI for list, inspect, plan, graph, run, history, and exit codes.
  • Read the studio before changing tubeless ui, definePipelineProject, or definePipelineStudio compatibility.
  • Read child composition before changing child propagation, progress, or parent/child selection.
  • Read the relevant executable example linked from the recipe index before writing new usage.
  • Copy consumer layout, export names, and IDs from the project manifest.
  • Use the generated API inventory only to verify exports; it is not implementation guidance.

Validation

For package changes:

make check

For a consumer-only change, run its focused tests and the repository typecheck. If the public surface changes intentionally, regenerate the checked API artifacts with bun run api:generate from the package root.