# Tubeless documentation Version: 0.2.2 Index: https://tubeless.io/llms.txt Read the agent guide before writing pipelines. Documentation below is generated from the same sources as the human site. --- Source: https://tubeless.io/docs/agent-guide.md # Agent guide for `tubeless` Use this guide when generating or modifying pipelines, pipeline-backed scripts, or shared helpers in this repository. ## Workflow 1. Read the [recipe index](https://tubeless.io/docs/recipes.md) and open the smallest matching example. 2. Copy file layout and stable IDs from the [project manifest](https://github.com/chetmancini/tubeless/blob/main/examples/catalog/tubeless.project.ts). 3. Declare stable IDs and operational descriptions. Add `name` only when printed output needs a friendlier display name. 4. Model data dependencies before failure policy or CLI concerns. 5. Typecheck the example or consumer, then run focused tests. 6. Run `make check` from the package root after changing the package or its learning surface. ## Primitive selection - Use `createSteps()` once per pipeline and `definePipeline` once after declaring its steps. Domain option types contain domain input only; callers pass those options and optional built-in controls to `run(options, controls?)`, while `plan(controls)` accepts controls alone. - Treat a `PipelineDefinitionError` during module loading as an authoring bug; static graph mistakes are rejected when `definePipeline` is called. - Use `dependsOn` when the output is required, `optionalDependsOn` when absence is expected, and `skipAfterFailureOf` for a failure gate that supplies no data. - Use `createSteps(optionsSchema)` when external domain options need Standard Schema validation or transformation. Add `outputSchema` only at step boundaries that receive untrusted or independently checked values, and `resultSchema` when the finalized public result must be checked. Core imports no schema library. - Set `dryRun: "skip"` on filesystem writes, database mutations, publication, email, and other steps whose normal `run` must not execute in a dry run. Use a typed `dryRun` handler when the step should produce a preview value instead. - Use `step.skippable` only for an intentional successful outcome. Handle its resulting `T | undefined` output type explicitly. - Use `fromPipeline` for one independently useful child workflow and `forEachPipeline` for runtime fan-out with stable keys and bounded concurrency. Use `forEachPipeline.skippable` only when the whole fan-out may be intentionally omitted; handle its `readonly T[] | undefined` output explicitly. Async `fromPipeline` result mappings publish resolved values; use that resolved shape for dependent inputs and policy-skip values. Parent plans keep these steps opaque but expose `nestedPipeline` with the child pipeline id, declared step ids, and single/fan-out mode for presentation. The interactive CLI automatically expands selected child steps and fan-out items into nested progress rows and retains completed states. Inner progress counts and details propagate through both adapters; do not forward raw child hooks or duplicate this bookkeeping in consumers. Fan-outs materialize up to 32 live item groups by default and emit the full retained tree once at settlement. Set `progress.detailLimit` to override that live cap and cap the final snapshot. - Use `fromRemote` for a unit of work that lives on another engine. Required fields are `adapter`, `mapInput`, and `outputSchema`. Omitting `dryRun` contacts the engine during a pipeline dry run; the adapter and remote worker must honor `context.dryRun` and authors prove the flag crossed the boundary in `mapInput`. Host-embed with `pipeline.runOrThrow` and pass `runId` / `parentRunId` when the graph must outlive the process. Parent plans expose `remote` with `engine` and optional `target`. Adapters may forward remote lines through `context.log` and must rethrow remote failures as `Error` with `cause` / `code`. Copy the native-fetch boundary in [`remote-steps.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/remote-steps.ts), including validation of unknown JSON and forwarding the signal. Use [`host-embedding.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/host-embedding.ts) for host-owned invocation; correlation IDs do not provide persistence or checkpoint/resume. - Use `runConcurrent` for bounded lightweight functions that do not need child lifecycle events. Use `runConcurrentSettled` when the caller needs completed results and the first failure instead of a throw. - Use `definePipelineCommand` for scripts centered on a pipeline. Do not parse `process.argv` manually or redeclare built-in dry-run, `--step`, or `--target` flags. `mapOptions`, validation, and hooks receive `stepIds` and `targets`, not `step` or `target`. Omit `mapOptions` when validated flags already satisfy same-name pipeline options; provide it when names, types, defaults, or derived values differ. The returned command exposes an immutable `descriptor`; UI adapters should render that structured parameter contract instead of parsing help text. Use `command.plan()` or `tubeless plan` for a selection-only preview. Do not simulate planning with `--plan`. - Use `pipeline.toMermaid()` or `command.toMermaid()` when documentation needs the static graph; do not duplicate dependency edges by hand. - Use `tubeless list` for the explicit project command inventory. Use `tubeless inspect ` for a registered module inventory, `tubeless plan` to preview selection without domain options or execution, and `tubeless graph` when generating documentation. `inspect`, `plan`, and `graph` accept a pipeline or a marked command and prefer the command when both are exported. Let the workbench discover the sole matching export or pass `--export`. - Use `tubeless run -- ` for project commands, or the existing file form only for modules exporting a `definePipelineCommand`. Keep application flags after `--`; the command must continue to own domain validation and option mapping. Pass `--trace ` for NDJSON traces (`-` writes NDJSON to stdout and moves command output to stderr) and `--store` for SQLite; they compose. Use `tubeless history` to list or show recorded runs from SQLite, or pass `--trace` to inspect a finished NDJSON artifact without importing it. Filter shared history with `--pipeline ` in any output mode. This is the pipeline definition's ID, not its registered command ID. ## Runtime rules - Library entrypoints are ESM-only and require Node.js 22 or later. The `tubeless` CLI requires Bun 1.3.14 or later; its `#!/usr/bin/env node` trampoline relaunches the binary with Bun under Node and reports an actionable install message when Bun is missing. - Use `context.log`, never direct `console` calls inside steps. - Pass `context.signal` into network calls, batching, retry, rate limiting, and long waits. Use `context.sleep` for retry-aware or testable delays. - Call `context.reportProgress` for long loops and `context.reportAttempt` for retries that operators should see. - Resolve relative files from `context.cwd`; prefer helpers from `tubeless/node`. - Use `runOrThrow` when every step must succeed and the caller expects a value. It always throws for an unsuccessful run, including `continueOnError` runs. Use `run` when the caller must inspect failures, skips, timings, or best-effort output. Its versioned `PipelineRun` exposes `runId`, terminal status and timestamps, errors, and timestamped step reports with correlated attempt IDs; structural skips have no attempt ID or start timestamp. Use hooks or tracing for streaming logs and progress. Exporter failures warn once per emitter and do not fail the run; pass `onExporterError` to observe the first drop. Use `plan` when nothing may run. - Declare supported downstream goals with `targets: [step]` on `definePipeline`, then select their literal IDs through run controls. Required inputs and failure gates are selected recursively. Use `stepIds` only when exact low-level filtering of any step is intentional; never combine the two. - Read `PipelinePlanStep.selectionReasons` when explaining selection. It already includes originating targets and immediate dependents; do not reconstruct provenance by walking dependency arrays in application or CLI code. Use `renderPipelinePlan` from `tubeless/render` for shared human or JSON output instead of maintaining another selection-reason formatter. Use `createPipelineReporter` / `createRunReporter` from `tubeless/reporter` for TTY presentation; do not import reporters from the root kernel. - Use focused hooks for ordinary observation: `onStepStart`, `onStepProgress`, `onStepComplete`, `onStepSkip`, `onStepCancel`, and `onStepFail`. Their event metadata is already narrowed. Use additive `onStepStatus` only when one consumer genuinely needs the whole discriminated lifecycle, such as a status-aware renderer or event store. - Use `createPipelineTestRuntime` from `tubeless/testing` for deterministic pipeline tests. Inspect its structured logs, statuses, and latest progress; keep test-framework matchers outside the package. - Branch on `PipelineError.code`, `phase`, and `kind`, never message prose. Underlying thrown codes live in `sourceCode`. Failed and cancelled reports expose the structured error under `error`; skipped reports expose `reason`, optional `message`, and optional `dependencyId`. `PipelineError.cause` is a bounded JSON-safe snapshot; a thrown `PipelineExecutionError` retains the original value through native `Error.cause`. Use `renderPipelineError` when a diagnostic crosses a human or JSON presentation boundary. - Inspect `error.fanOut` in reports or recorded trace history for bounded keyed failures from `forEachPipeline`. Check `omittedFailureCount` and `keyTruncated` before selecting rerun inputs; unstarted items are not failures. Reruns remain caller-owned new runs. - Standard Schema failures use `kind: "validation"`, retain normalized `issues`, and have boundary-specific codes for options, step outputs, and final results. Async schemas run during `run`; synchronous `plan()` previews graph and selection only and never invokes schemas. - Wrap normal finalizers in `requireOutputs` when a valid result requires specific step outputs. Use a plain finalizer only when partial output is a valid domain result. - Preserve the dependency-free runtime. Keep application telemetry SDKs at the exporter boundary. Use `composeTraceExporters` from `tubeless/tracing` when one run must fan out to multiple destinations; `onExporterError` reports the first partial drop, the failed destination is retired, and healthy exporters keep receiving events. - Keep durable local observation opt-in. Use the append-only adapter from `tubeless/run-store/sqlite` or `tubeless run --store`; inspect recorded runs with `tubeless history`. Use the strictly read-only adapter from `tubeless/run-store/ndjson`, `tubeless history --trace`, or the `--trace` option to `tubeless ui` for a finished portable trace. Treat trace files as sensitive: logs, errors, and attributes are displayed as recorded, and malformed or oversized artifacts are rejected. `tubeless history` inspects a finished artifact and refuses a store with a live writer or multiple hard links. SQLite `export()` may return before the row is on disk. Other connections cannot see that tail until a batch of 64, `flush()`, same-instance `listEvents`/`clearHistory`, or `close()`. A crash can lose up to 63 unflushed events; `tubeless run --store` flushes at completion so finished runs are durable. Do not make pipeline definitions depend on storage or the studio. Recorded history keeps the last `reportProgress` `details` plus `detail_count`, and opaque child steps keep `nested_pipeline` with the original `step_count`. Studio renders those snapshots; it does not flatten child DAGs into the parent step. Observed definitions pick the latest `pipeline.started` by `timestampMs`, then store-local id. - Use `createPipelineRunProjector` from `tubeless/run-store` when a custom reader pages `listEvents({ afterId })`. Append each newer page and call `snapshot()`; a refresh with no new ids returns the cached view. Duplicate and out-of-order ids are ignored; `0` is a valid first id. Pass `{ retainLogs: false }` only when the snapshot should keep `logCount` without log bodies. Use `projectPipelineRunStore` only for a one-shot fold of a complete list. See [`local-observability.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/local-observability.ts). - Pass caller-owned `runId` and `parentRunId` values through `PipelineContext` when joining an external execution tree. Do not derive correlation from step IDs or timestamps. - Treat `tubeless ui` as a local projection with no execution capability by default. Use `definePipelineProject` for a checked-in command catalog with stable registered IDs, and register only explicit `definePipelineCommand` modules; never make execution require the studio server or infer executable modules from observed history. Every studio request, including reads, must send a `Host` that matches the bound authority, or, on a wildcard bind, `localhost` or a literal IP on the same port. Browser plan, launch, cancel, and clear-history also send `x-tubeless-studio-*` headers; those are same-origin guards, not authentication. Studio HTTP errors contain stable `code`, `message`, `hint`, and a backward-compatible `error` alias; the local-only contract is published at `https://tubeless.io/openapi.json`. Cancel a live top-level launch from the running detail pane; that abort is process-local, leaves sibling launches running, and is not crash-resume. ## Failure and safety rules - Do not use `step.skippable` to swallow an exception. - Do not treat dry run as rollback; external side effects require `dryRun: "skip"` or a side-effect-free custom `dryRun` handler. - Do not publish after a failed or cancelled validation step. Make validation required, or use `skipAfterFailureOf` when its output is intentionally optional; the gate blocks both unsuccessful terminal statuses. - Do not hide absent required finalizer outputs behind defaults. Declare them with `requireOutputs`; handle optional outputs explicitly. - Keep step IDs stable because reports, hooks, traces, and CLI selection use them. - Treat `name` as presentation only. Dependencies, outputs, traces, and CLI selection continue to use the stable step ID. ## Required references - Read [core concepts](https://tubeless.io/docs/concepts.md) for skip, failure, or selection changes. - Read [the CLI](https://tubeless.io/docs/cli.md) for list, inspect, plan, graph, run, history, and exit codes. - Read [the studio](https://tubeless.io/docs/studio.md) before changing `tubeless ui`, `definePipelineProject`, or `definePipelineStudio` compatibility. - Read [child composition](https://tubeless.io/docs/child-pipeline-composition.md) before changing child propagation, progress, or parent/child selection. - Read the relevant executable example linked from the [recipe index](https://tubeless.io/docs/recipes.md) before writing new usage. - Copy consumer layout, export names, and IDs from the [project manifest](https://github.com/chetmancini/tubeless/blob/main/examples/catalog/tubeless.project.ts). - Use the [generated API inventory](https://tubeless.io/docs/api-reference.md) only to verify exports; it is not implementation guidance. ## Validation For package changes: ```sh make check ``` For a consumer-only change, run its focused tests and the repository typecheck. If the public surface changes intentionally, regenerate the checked API artifacts with `bun run api:generate` from the package root. --- Source: https://tubeless.io/docs/getting-started.md # Getting started ## Install Library consumers install the package and import it from Node.js 22 or later: ```sh npm install tubeless ``` The same package works with `pnpm add tubeless`, `yarn add tubeless`, and `bun add tubeless`. The README quick start is a complete first program. The `tubeless` CLI is separate: it requires Bun 1.3.14 or later. ```sh bunx tubeless --help ``` Command-by-command usage is in [the CLI](https://tubeless.io/docs/cli.md). ## Runtime support Library imports (`tubeless` and every subpath) are ESM-only and require Node.js 22 or later. They are dependency-free compiled JavaScript. The `tubeless` CLI requires Bun 1.3.14 or later. Its shebang is `#!/usr/bin/env node`: the binary relaunches itself with Bun when run under Node, and Bun loads TypeScript pipeline modules directly. When Bun is missing, the binary prints install instructions instead of a raw interpreter error. Supported operating systems are Linux and macOS. Windows is untested. ## 1. Define domain options Describe only domain input. Built-in controls are a separate argument to `run(options, controls?)` and `runOrThrow(options, controls?)`. `plan(controls)` accepts controls alone. ```ts interface ImportOptions { lines: readonly string[]; } const lines = [" First row ", "", "Second row"]; ``` ## 2. Create typed steps Call `createSteps` once for the pipeline. Declare dependencies with step object references; their output types become the dependent step's input types. ```ts import { createSteps } from "tubeless"; const step = createSteps(); const load = step("load", { description: "Read source rows", run: (_inputs, context) => context.options.lines, }); const normalize = step("normalize", { dependsOn: [load], description: "Normalize non-empty rows", run: ({ load: rows }) => rows.map((row) => row.trim().toLowerCase()).filter(Boolean), }); ``` IDs should remain stable: plans, reports, hooks, tracing, and CLI selection all use them. ## 3. Define the result ```ts import { definePipeline, requireOutputs } from "tubeless"; const ImportPipeline = definePipeline({ id: "import", steps: [load, normalize], targets: [normalize], finalize: requireOutputs([normalize], ({ normalize }) => ({ count: normalize.length, rows: normalize, })), }); ``` `finalize` receives the outputs that completed. `requireOutputs` declares which ones a valid result needs, makes them non-optional in the callback, and reports a finalization error when a dry run, exact filter, or failure leaves one absent. Use a plain finalizer when partial output is itself a valid result. `targets` on the definition declares the pipeline's supported downstream goals. Only those literal IDs are accepted by target selection and shown by pipeline commands. When `requireOutputs` is used, definition construction rejects a declared target whose dependency closure cannot produce the required result. ## 4. Visualize the static graph `toMermaid` returns dependency-free Mermaid flowchart source without planning or running the pipeline. It uses `name` when present and otherwise displays the stable step ID. ```ts const diagram = ImportPipeline.toMermaid({ direction: "LR" }); ``` Required data dependencies use solid arrows. Optional inputs and failure gates use labeled dotted arrows. Set `includeDescriptions: true` when the extra node text is useful. The same source is available from `tubeless graph`; see [the CLI](https://tubeless.io/docs/cli.md). ## 5. Pick an execution method ```ts const value = await ImportPipeline.runOrThrow({ lines }); const report = await ImportPipeline.run({ lines }, { continueOnError: true }); const plan = ImportPipeline.plan({ dryRun: true }); const normalizePlan = ImportPipeline.plan({ targets: ["normalize"] }); ``` - `runOrThrow` returns the finalized value only when the run succeeds and throws `PipelineExecutionError` for every unsuccessful run. `continueOnError` may let independent steps finish, but it never makes their failures successful. - `run` always returns the structured pipeline result. - `plan` describes executor selection without requiring or validating domain options. - `targets` selects declared downstream goals plus their required inputs and failure gates. `stepIds` is an exact low-level filter and cannot be combined with `targets`. See [core concepts](https://tubeless.io/docs/concepts.md) for selection provenance. Library callers invoke the pipeline directly. Scripts should use `definePipelineCommand` so help, `--target`, `--step`, dry run, and cancellation stay generated instead of hand-parsed. See [the CLI](https://tubeless.io/docs/cli.md) and [`cli-job.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/cli-job.ts). ## 6. Test with deterministic runtime plumbing ```ts import { createPipelineTestRuntime } from "tubeless/testing"; const test = createPipelineTestRuntime({ cwd: "/workspace" }); const result = await test.run(ImportPipeline, { lines }); test.clock.now(); test.logs; test.statuses; test.latestProgress; ``` The default sleep advances the test clock immediately. The harness also owns an abort controller and provides typed `runOrThrow` and `plan` helpers. It has no test-framework dependency; use ordinary assertions against its structured captures and pipeline results. ## Next Continue with the [recipe index](https://tubeless.io/docs/recipes.md), or read [core concepts](https://tubeless.io/docs/concepts.md) before implementing failure-sensitive writes. For untrusted boundary values, use the Standard Schema support in [`validated-boundaries.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/validated-boundaries.ts). Persist and inspect local runs with [the studio](https://tubeless.io/docs/studio.md), or open a finished portable trace with `tubeless history --trace run.ndjson`. --- Source: https://tubeless.io/docs/recipes.md # Recipe index Every linked TypeScript example is compiled by the package typecheck. `pack:verify` also executes these modules from the published tarball. Start with the smallest example matching the workflow rather than assembling primitives from the API inventory. | Intent | Executable recipe | Main primitives | | ---------------------------------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | | Sequential import or ETL | [`typed-import.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/typed-import.ts) | `createSteps`, `dependsOn`, `requireOutputs`, `targets` | | Validate options, outputs, and results | [`validated-boundaries.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/validated-boundaries.ts) | Standard Schema, `outputSchema`, `resultSchema` | | Inspect, plan, or graph a pipeline or command | [`typed-import.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/typed-import.ts) | `tubeless inspect`, `tubeless plan`, `tubeless graph`, `toMermaid` | | Safe write/publish preview | [`publish-with-gates.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/publish-with-gates.ts) | `dryRun`, `optionalDependsOn`, `skipAfterFailureOf` | | Deliberately omit unnecessary work | [`conditional-step.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/conditional-step.ts) | `step.skippable`, valued skip, skip-aware output typing | | Preserve independent work after failure | [`best-effort.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/best-effort.ts) | `continueOnError`, structured `run` result | | Compose one reusable workflow | [`child-pipeline.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/child-pipeline.ts) | `fromPipeline`, `mapOptions`, resolved async `mapResult` | | Call a real HTTP service | [`remote-steps.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/remote-steps.ts) | `fromRemote`, fetch cancellation, validated HTTP output | | Host a pipeline in a durable engine | [`host-embedding.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/host-embedding.ts) | `runOrThrow`, pass `runId` / `parentRunId` | | Fan out over runtime items | [`fan-out-progress.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/fan-out-progress.ts) | `forEachPipeline.skippable`, stable keys, concurrency, progress | | Inspect keyed fan-out failures | [`fan-out-progress.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/fan-out-progress.ts) | `error.fanOut`, bounded diagnostics, caller-directed reruns | | Show determinate progress | [`fan-out-progress.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/fan-out-progress.ts) | `reportProgress`, bounded live CLI rows, complete final trees | | Watch the live TTY reporter | [`live-tui.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/live-tui.ts) | named steps, nested `details`; persist with `--store` | | Retry and rate-limit remote calls | [`resumable-enrichment.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/resumable-enrichment.ts) | `withRetry`, `RateLimiter`, injected sleep and signal | | Resume durable long-running work | [`resumable-enrichment.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/resumable-enrichment.ts) | `dryRun`, `openCheckpoint`, `withCheckpointedBatch` | | Expose and run a typed command-line program | [`cli-job.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/cli-job.ts) | `definePipelineCommand`, conditional `mapOptions`, `tubeless run` | | Render plans and diagnostics | [`rendering.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/rendering.ts) | `renderPipelinePlan`, `renderPipelineError` | | Handle cancellation and deterministic testing | [`cancellation-and-testing.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/cancellation-and-testing.ts) | `createPipelineTestRuntime`, captured status/progress | | Export JSON or OpenTelemetry lifecycle events | [`tracing.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/tracing.ts) | trace context, exporter composition, JSON / OTel, `onExporterError` | | Persist, port, inspect, launch, or cancel runs | [`local-observability.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/local-observability.ts) | SQLite / NDJSON stores, `tubeless history --pipeline `, studio projector | | Watch many primitives in one run | [`peloton.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/peloton.ts) | delays, logs, children, fan-out, retry, gates, test runtime | | Project layout, IDs, and command manifest | [`tubeless.project.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/catalog/tubeless.project.ts) | `pipelines/`, `scripts/`, `definePipelineProject`, `tubeless list` | ## Selection rules 1. Use an ordinary step for one unit of domain work. 2. Set `dryRun: "skip"` or provide a side-effect-free `dryRun` handler before exposing side-effecting work through CLI dry-run support. 3. Use `step.skippable` only for a successful policy decision, never to hide an error. 4. Use a child pipeline when the child has value independently; use a normal helper function when it does not. 5. Use `forEachPipeline` when every item needs child-pipeline lifecycle and reporting. Opt into `forEachPipeline.skippable` when policy may omit the whole fan-out and a canonical skipped disposition matters. Use `runConcurrent` for lightweight worker functions that should throw on the first failure. Use `runConcurrentSettled` when the caller needs completed results plus that failure without throwing. Use fromRemote when a unit of work lives on another engine but the parent DAG still runs in this process. Omit dryRun only when the adapter and the remote worker are side-effect free under context.dryRun. Embed the whole pipeline in Temporal/Lambda/a worker when the graph must outlive the process. 6. Use `definePipelineCommand` for pipeline scripts; use `defineCommand` only when the script is not centered on a pipeline. Preview selection with `command.plan()` or `tubeless plan`; do not simulate planning with `--plan`. `--step` and `--target` are argv flags; `mapOptions` and hooks read `stepIds` and `targets`. 7. Declare public goals with `targets: [step]` on the pipeline, select their IDs for goal-oriented execution, and use `stepIds` only for an exact filter. Use `requireOutputs` when the final domain result is not meaningful without specific step outputs. Read plan `selectionReasons` instead of recreating target-closure logic in a CLI or application. 8. Use `tubeless/render` when plans or diagnostics cross a human or JSON presentation boundary; do not duplicate selection-reason or error formatting. Use `tubeless/reporter` for optional TTY run reporters; do not import them from `tubeless`. 9. Copy consumer file layout, export names, and IDs from the [project manifest](https://github.com/chetmancini/tubeless/blob/main/examples/catalog/tubeless.project.ts). Register project commands explicitly; do not infer executable modules from run history or the filesystem. Cancel only a live launch owned by the current studio process; it is not crash-resume and does not abort sibling launches. For semantics behind these choices, read [core concepts](https://tubeless.io/docs/concepts.md). For workbench commands, read [the CLI](https://tubeless.io/docs/cli.md). For the local run UI, read [the studio](https://tubeless.io/docs/studio.md). --- Source: https://tubeless.io/docs/concepts.md # Core concepts ## Definition model A pipeline is an ordered declaration of typed steps plus a finalizer. `definePipeline` immediately rejects invalid static graphs (duplicate/reserved IDs, missing or contradictory dependencies, and cycles) and stores a compiled graph. Planning applies option-dependent step selection against that compiled graph and does not re-validate the definition. Dependency arrays are snapshotted in that compiled storage; caller-owned step objects are left unchanged. Literal duplicate step IDs are also rejected by TypeScript at the `definePipeline` call. Runtime definition validation remains the backstop for widened arrays and dynamically assembled definitions. Use one `createSteps()` factory per pipeline. A step object is both its definition and its typed dependency token. The ID is the stable machine identity used for dependencies, output keys, selection, and tracing. An optional `name` overrides how reporters and printed plans display the step without changing that identity. ## Dependency choices | Field | Supplies data | Blocks after failure | Typical use | | -------------------- | ------------- | -------------------- | ----------------------------------- | | `dependsOn` | Required | Yes | Transformation needs upstream value | | `optionalDependsOn` | When present | No | Fallback or partial rerun | | `skipAfterFailureOf` | No | Yes | Publish/write safety gate | Prefer required dependencies unless partial execution is an intentional part of the workflow. A failure gate blocks after either `failed` or `cancelled`; both mean the guarded prerequisite did not safely complete. Shorter aliases such as `needs`, `uses`, and `gatedBy` were evaluated and not adopted. `uses` does not communicate optionality, and aliases would leave two vocabularies for the same graph. The explicit fields above remain canonical. ## Three kinds of non-execution - **Structural skip:** a dependency, dry-run rule, filter, abort, or fail-fast decision prevents execution. Dependents that require the step do not run. - **Policy skip:** a `step.skippable` predicate deliberately decides that work is unnecessary. It may publish a value and unlock dependents. - **Failure:** the step ran and threw. Fail-fast stops new work by default; `continueOnError` lets independent work proceed. `continueOnError` changes scheduling, not success: the structured run result remains unsuccessful and `runOrThrow` still throws. Inspect `run()` when a best-effort final value is useful despite recorded failures. Any `step.skippable` step has output type `T | undefined`, even if current skip paths publish a value. Dependents must handle the absence explicitly. ## Selection and finalization Pipeline definitions declare supported downstream goals with step references: ```ts definePipeline({ id: "publish", steps: [build, validate, publish], targets: [publish], finalize: requireOutputs([publish], ({ publish }) => publish), }); ``` At call sites, `targets` accepts only those declared literal IDs. The planner recursively selects each target's required inputs and failure gates, while leaving optional-only inputs out. `pipeline.targetIds` and CLI `--target` discovery expose the same declared set. Internal steps are not public goals; `stepIds` remains an exact filter for partial reruns and other advanced workflows and does not add prerequisites. Empty selections, unknown, undeclared, or duplicate IDs, and a run that supplies both fields fail during planning. Every planned step includes `selectionReasons`, a stable discriminated union that explains inclusion or omission without requiring callers to reconstruct the graph. A target plan can report a direct `target`, a `required-dependency`, a `failure-gate`, an excluded `optional-only` input, or an `outside-target-closure` omission. Exact filters use `exact` and `not-selected`; unfiltered plans use `all`. ```ts const plan = pipeline.plan({ targets: ["publish"] }); const load = plan.steps.find((step) => step.id === "load"); // A shared prerequisite can explain every path that selected it. load?.selectionReasons; // [{ kind: "required-dependency", dependentId: "build", targetId: "publish" }] ``` `selected` remains the convenient boolean and `skipReason` describes execution disposition such as `filtered`, `dry-run`, or `unmet-dependency`. Render a plan at the presentation boundary instead of duplicating its selection reason switch in each command or tool: ```ts import { renderPipelinePlan } from "tubeless/render"; const plan = pipeline.plan({ targets: ["publish"] }); const terminalText = renderPipelinePlan(plan); const machineJson = renderPipelinePlan(plan, { format: "json" }); ``` Human output explains target and dependency provenance by default; pass `{ explain: false }` for a compact disposition-only view. JSON output preserves the original structured plan fields for machine consumers. ## Step statuses and structured errors Steps have one canonical, enforced lifecycle: ```text planned ──┬──> running ──┬──> complete │ ├──> skipped │ ├──> cancelled │ └──> failed ├──> skipped └──> cancelled ``` The runtime derives focused hooks from this state machine. Use `onStepStart`, `onStepProgress`, `onStepComplete`, `onStepSkip`, `onStepCancel`, or `onStepFail` for ordinary pipeline integration; each callback receives metadata narrowed to that case. `onStepPlan` is available for plan-aware tooling. `onStepStatus` is an additive catch-all for consumers that need the complete discriminated lifecycle, such as renderers and event stores. Registering it does not replace the focused callbacks. A `running` status may be published repeatedly as progress changes. Every selected step ends in exactly one terminal report; cancellation is distinct from failure, including selected steps that were cancelled before they started. ```ts const hooks: PipelineHooks = { onStepProgress({ progress, step }) { renderProgress(step.id, progress); }, onStepSkip({ reason, step }) { explainSkip(step.id, reason); }, onStepFail({ error, step }) { reportError(step.id, error); }, onStepCancel({ error, step }) { reportCancellation(step.id, error); }, }; ``` Every `PipelineError` has package-owned `code`, `phase`, and `kind` fields. Branch on those fields rather than message text. `phase` locates the lifecycle boundary (`definition`, `planning`, `execution`, or `finalization`); `kind` distinguishes definition, selection, step, dependency, cancellation, child, and finalization problems. If a thrown error supplied its own machine code, it is retained separately as `sourceCode`. Native `Error.cause` chains are copied into a bounded, cycle-safe `PipelineErrorCause` tree containing only message, name, source code, and nested cause fields, so reports and trace events stay JSON-safe. ```ts const result = await pipeline.run(options); const first = result.errors[0]; if (first?.code === "TUBELESS_STEP_FAILED" && first.sourceCode === "ENOENT") { // A step failed because its underlying operation could not find a file. } ``` `runOrThrow` raises `PipelineExecutionError` with the full structured result on `result` and the original thrown value on native `cause`. Its default message identifies the pipeline, phase, package code, step, and deepest normalized cause. `PipelineDefinitionError` uses the same diagnostic summary for every rejected definition issue. `renderPipelineError` from `tubeless/render` exposes that same diagnostic formatting directly and can emit the structured error as JSON. `PipelineStepReport` is the terminal-state union. Successful reports carry timing, skipped reports carry `reason` plus optional `message` and `dependencyId`, and failed/cancelled reports carry a structured `error`. Hooks and tracing consume the same terminal objects rather than reconstructing status from separate callbacks. ## Versioned run records `run()` returns one versioned `PipelineRun`. Every execution has a `runId`, terminal `status`, start and finish timestamps, structured errors, and one terminal report per planned step. Pass `context.runId` and `context.parentRunId` when an external orchestrator owns correlation; otherwise core generates an opaque run ID. The same IDs are available in step contexts and optional trace exports. An actual step execution receives one `attemptId`. It appears on the `PipelineStepContext`, its terminal `PipelineStepReport`, and trace lifecycle records. Executed reports also carry start and finish timestamps. Structural skips and steps cancelled before starting have neither an attempt ID nor a start timestamp because their handlers never ran. `context.reportAttempt()` describes retry activity within that execution; it does not create another persisted attempt model. The terminal record deliberately does not copy log or progress streams. Use the injected logger and hooks for live observation, or tracing when durable event export is required. Errors and cancellation remain represented directly in the run and terminal step reports. A failing `tracing.exporter` does not fail the run: the executor warns once on the first export or flush error for that emitter. A standalone failure drops later events. `composeTraceExporters` instead retires a failed destination after reporting the first partial drop and keeps sending to healthy destinations. Nested child runs each construct their own emitter, so `tracing.onExporterError` fires once per nested run rather than once for the whole parent tree. The run currently has `version: 2`, also exported as `RUN_MODEL_VERSION`; persist that field and branch on it before decoding a stored run. Projected `StoredPipelineRun` snapshots stamp the same version. Durations are derived by subtracting the relevant timestamps. ```ts const result = await pipeline.run(options, undefined, { ...defaultPipelineContext(), runId: externalJobId, }); result.runId; result.status; result.steps.find((step) => step.attemptId)?.attemptId; result.finishedAtMs - result.startedAtMs; ``` ### Local event store and studio The local studio is a composition of existing boundaries, not part of pipeline execution: ```text pipeline definition → dependency-free executor → trace exporter ↘ append-only SQLite ↘ local studio ↘ portable NDJSON ↗ ↘ injected command launcher (SQLite only) ``` `openSqlitePipelineRunStore()` implements `PipelineTraceExporter` and appends every lifecycle record to one versioned SQLite event table. Database triggers reject updates and deletes. Current run state, history, step attempts, progress, logs, errors, and observed definition graphs are projections of that immutable stream; they are not competing executor models. The SQLite adapter also exposes an explicit `clearHistory()` maintenance reset. It deletes the complete event history and compacts the database while restoring the append-only triggers in the same transaction; individual event updates and deletes remain forbidden. The workbench injects this capability into its loopback studio with a destructive confirmation and refuses to clear while it knows a browser-launched run is still live. Persisted runs left active by an interrupted process can still be cleared after confirmation. A directly embedded studio stays history-immutable unless its caller explicitly injects the same capability and may report its own known live writers through `isBusy()`. `openNdjsonPipelineRunStore()` gives a finished trace artifact the same read-only event-query boundary. It validates and bounds the artifact, assigns store-local ids in file order, closes the file, and never exposes an exporter or maintenance capability. History and Studio can therefore project CI or support traces directly without copying them into SQLite. Trace contents are not redacted and should be handled as sensitive operational data. Trace-enabled contexts route calls made through `context.log` into correlated `pipeline.log` events while still forwarding them to the injected logger. Definition metadata is attached to `step.planned` records, so the studio can render observed graphs without importing application pipeline modules. Opaque child steps include `nested_pipeline` on those records. Progress-bearing `step.running` events carry the parent summary plus bounded `details` rows so history can show the same per-item status as the live TTY. The first planned step from a newer run replaces that pipeline ID's observed targets and steps, so a changed definition does not retain older structure while a run that fails validation before planning does not erase the last usable graph. The main `tubeless` entrypoint never imports SQLite or the UI. The optional `tubeless/run-store/sqlite` adapter selects the runtime-provided SQLite implementation, `tubeless/run-store/ndjson` reads portable artifacts, and `tubeless/run-store/ui` is a separate HTTP projection. It is read-only unless the caller injects a `PipelineRunStudioLauncher`. Likewise, ordinary `tubeless run` remains unchanged; pass `--store` to record a run, `--trace` for NDJSON, `tubeless history` to inspect SQLite, `tubeless history --trace` to inspect portable output, and `tubeless ui` only when a browser view is useful. Observed definitions are never treated as executable registrations: an event stream does not contain a trusted module path or a command's domain contract. `tubeless ui --command ./path/to/command.ts` explicitly loads only marked `definePipelineCommand` exports and passes bounded structured form values through their validation, option mapping, and execution path. No shell or argv round trip is involved. Launch-enabled workbench servers are restricted to loopback, while the UI HTTP module stays execution-agnostic through its injected capability. Each command exposes an immutable, JSON-safe descriptor derived from the same effective schema used by its parser, including built-in flags. The studio uses that descriptor for checkboxes, numeric inputs, constrained selects, paths, and repeatable values. Planning is a separate read-only capability: **Preview plan** passes the current dry-run and step/target values to the real pipeline planner and expands the result beneath the launch form. Domain parameters remain visible for the eventual run but are not interpreted by the planner, no run is recorded, and previewing is optional. Opaque `fromPipeline` and `forEachPipeline` steps carry `nestedPipeline` metadata so the preview can distinguish ordinary work from a single child pipeline or runtime fan-out without flattening child selection. Studio snapshots page through the store once and then fetch only events after the last observed sequence. Those pages go through `createPipelineRunProjector` from `tubeless/run-store`; a refresh with no newer ids returns the cached snapshot instead of re-folding history. Store-local event ids are monotonic and may start at `0`. Duplicate or out-of-order ids are ignored. Use `projectPipelineRunStore` when the caller already holds a complete event list and does not need incremental refresh. History therefore continues past an adapter's per-query safety cap without reloading the entire database on every refresh. For a reusable project interface, `definePipelineProject` declares stable command IDs and versioned module references in one dependency-free manifest. `tubeless list`, `inspect`, `plan`, `graph`, and `run` resolve those IDs without scanning for executable files. Module paths and the optional execution `cwd` resolve from the manifest instead of the caller's shell directory; duplicate or malformed declarations fail when it loads. Presentation-name overrides never change registered, run, or pipeline identity. Legacy `definePipelineStudio` catalogs remain accepted by `ui` with their historical UI-only identities. Cancellation is classified from an actual abort error or propagated child cancellation, not merely from the signal's current state. An unrelated failure that races with `abort()` remains a step or finalization failure. Finalizer inputs remain partial because dry-run policy, exact filtering, and best-effort execution can omit outputs. Wrap a normal finalizer with `requireOutputs([stepA, stepB], callback)` to declare which output slots make a valid result. The callback receives those values as required properties, and the run fails finalization with the missing step IDs if any slot is absent. A successfully published `undefined` still counts as an output; structural absence is determined by whether the slot was published. When a pipeline uses `requireOutputs`, every declared target is checked during definition construction. Its required-input and failure-gate closure must contain all required finalizer steps, so an advertised target cannot complete its work and then fail only because it intentionally omitted the pipeline's result. Plain finalizers remain appropriate when selected goals intentionally produce a partial domain result. ### Why targets are declared Three contracts were evaluated after dependency-aware selection shipped: - expose every step as a target and keep the global finalizer; - add a separate selected-execution runner returning raw output slots; or - declare the pipeline's meaningful public goals and retain one finalized domain result. Declared targets were selected because they preserve the existing `run`/`runOrThrow` result contract, add only one short definition field, keep steps as implementation details, and let definition validation prove `requireOutputs` compatibility. A second runner would create competing execution/result semantics, while goal-specific finalizers add ceremony that current production pipelines do not need. Exact `stepIds` remains the explicit escape hatch for low-level partial work. ## Dry runs Every step has one normal `run` handler and an optional dry-run policy: - Omit `dryRun` when the normal handler is safe and useful during a dry run. - Set `dryRun: "skip"` for filesystem writes, database mutation, publication, email, and other external side effects. The step is reported as structurally skipped and its required dependents do not run. - Provide a `dryRun(inputs, context)` handler to substitute a side-effect-free preview. It must return the same output type as `run`, so dependents can use the preview normally. ```ts const publish = step("publish", { dependsOn: [build], dryRun: ({ build }) => ({ id: `preview:${build.id}` }), run: ({ build }) => publishArtifact(build), }); ``` Dry run is not a rollback mechanism. Read-only validation and resolution steps may run normally so the preview remains useful. ## Validated boundaries `tubeless` implements the small Standard Schema V1 structural contract and does not import a validation library. Pass a domain-options schema to `createSteps(schema)`, an `outputSchema` to an ordinary step, or a `resultSchema` to `definePipeline`: ```ts const step = createSteps(optionsSchema); const parse = step("parse", { outputSchema: parsedRowsSchema, run: (_inputs, context) => readRows(context.options.source), }); const pipeline = definePipeline({ id: "validated-import", steps: [parse], resultSchema, finalize: requireOutputs([parse], ({ parse }) => summarize(parse)), }); ``` Schema input and output types are inferred independently. Callers pass the options schema's input type, steps read its validated output type, a step's `run` and dry-run handler return its schema input type, dependents receive its schema output type, and `runOrThrow` returns the result schema's output type. Built-in run controls are separate from domain-options validation, so strict object schemas do not need to know about `dryRun`, `targets`, or other executor policy. Callers pass those controls alongside domain fields; the executor removes the reserved control keys before validation while preserving class methods, getters, inherited and non-enumerable properties, and symbol keys. Steps receive the validated output object directly; the executor neither flattens it nor overlays control fields onto it. Options are validated once after structural planning and before any step starts. Step values are validated before publication, including values from a custom dry-run handler or policy skip. Finalized values are validated before a run is marked successful. Sync and async Standard Schema validators are both supported during `run`; `plan()` stays synchronous and does not invoke schemas. Failures retain normalized issue messages and paths on `PipelineError.issues`. Use the stable codes `TUBELESS_OPTIONS_VALIDATION_FAILED`, `TUBELESS_STEP_OUTPUT_VALIDATION_FAILED`, and `TUBELESS_FINAL_RESULT_VALIDATION_FAILED` to distinguish boundaries. A step-output failure is reported on that step; final-result failure uses `__finalize__`. Pipeline definitions cannot mix steps from different options-schema factory scopes. ## Child pipelines Use `step.fromPipeline` for one child run and `step.forEachPipeline` when runtime items each need the same child. Children are opaque to the parent's plan and hooks; their activity is summarized as progress on the parent step. See [child-pipeline composition](https://tubeless.io/docs/child-pipeline-composition.md) for propagation and selection boundaries. When the whole mapped fan-out is optional, use `step.forEachPipeline.skippable`; only that opt-in constructor widens the mapped array output with `undefined`. ## Remote steps Use `step.fromRemote` when one parent step's work lives on another engine. The parent plan stays local and opaque; `remote.engine` and optional `remote.target` are presentation only. Dry-run remains a side-effect gate: omitting `dryRun` still calls the adapter. See [remote-step composition](https://tubeless.io/docs/remote-step-composition.md). ## Mermaid diagrams Call `pipeline.toMermaid()` to generate a static flowchart without run options or execution. Nodes use `name ?? id`; generated internal node identifiers keep arbitrary user-facing text out of Mermaid syntax. Required dependencies render as solid arrows, while optional inputs and failure gates render as labeled dotted arrows. Use `direction` to change layout and `includeDescriptions` to add operational descriptions to node labels. ## Module workbench Command-by-command usage lives in [the CLI](https://tubeless.io/docs/cli.md). The optional local UI is documented in [the studio](https://tubeless.io/docs/studio.md). `tubeless inspect`, `tubeless plan`, and `tubeless graph` load a registered project command, a pipeline, or a marked `definePipelineCommand` without executing steps or requiring domain options, and prefer the command when both are exported. `tubeless run` deliberately accepts only a command created by `definePipelineCommand`, because that export carries the application-owned parser, validation, option mapping, reporter, and result summary needed for safe execution. A raw pipeline is not executable through the workbench. `tubeless history` reads the optional local SQLite store; `tubeless run --trace` writes NDJSON without requiring a store. `definePipelineCommand` defaults to a same-name mapping when its validated flag values structurally satisfy the pipeline's domain options. Command-only values such as `resume`, `stepIds`, and `targets` are removed before execution; bridge-owned selection and failure controls are applied separately. The argv flags stay `--resume`, `--step`, and `--target`. If flags do not satisfy required domain options, `mapOptions` remains required at compile time. Keep explicit mapping for renamed fields, file loading, adaptive defaults, prompts, or other derived values. The returned command's `descriptor` is the non-terminal view of that contract: name, description, and immutable parameter metadata for flags, types, defaults, choices, numeric constraints, repeatability, paths, and positional support. It is presentation-neutral; consumers still submit argv through `parse` or `run` so one validation and execution path remains authoritative. `command.plan()` always exposes structural planning from `PipelineRunControls` (`dryRun`, `stepIds`, and `targets`) only. It deliberately skips domain parsing, option mapping, execution, and persistence. Use `command.plan()` or `tubeless plan`; do not simulate planning with `--plan`. Keep the workbench's module-selection arguments before the `--` boundary and the command's application arguments after it: ```sh tubeless run --export PublishCommand ./scripts/publish.ts -- --source input.json --target publish ``` The workbench forwards SIGINT through the command context and classifies exits as validation, planning, execution, or cancellation without parsing diagnostic messages. ## Runtime context Use the step context instead of global facilities: - `context.options` for validated domain options and `context.dryRun` for the active mode. - `context.log` so interactive reporters can preserve output. - `context.signal` for cooperative cancellation. - `context.sleep` for cancellation-aware, testable delays. - `context.reportProgress` and `context.reportAttempt` for observability. - `context.cwd` for caller-controlled path resolution. Callers may inject the logger, clock, sleep function, signal, hooks, and tracing. This makes a pipeline embeddable and deterministic under test. ## Deterministic testing `createPipelineTestRuntime` from `tubeless/testing` packages the ordinary runtime injection points into a framework-neutral harness. Its default sleep advances a monotonic clock immediately, its logger stays silent while capturing calls, and its canonical status hook records lifecycle events and latest progress. The harness owns an `AbortController` and delegates `run`, `runOrThrow`, and `plan` to the real pipeline, preserving option and result inference without implementing alternate execution semantics. Create one runtime per test. Customize `sleep` when a test needs to pause, interleave, or advance time differently; call `test.abort()` to exercise the normal cancellation path. Assertions remain application- or framework-owned. ## Stable boundaries - Keep step IDs stable, use `name` only for a friendlier display label, and keep descriptions operationally useful. - Generate documentation from `pipeline.toMermaid()` instead of hand-maintaining a second copy of the graph. - Return small domain results; inspect `run()` reports for execution metadata. - Put remote API mechanics in retry/rate-limit helpers, not in the executor. - Put CLI parsing at the script edge with `definePipelineCommand`. - Keep application-owned telemetry SDKs outside the dependency-free core. --- Source: https://tubeless.io/docs/cli.md # CLI The `tubeless` binary inspects, plans, graphs, and runs exported pipeline modules. It requires Bun 1.3.14 or later. Bun loads TypeScript pipeline files directly. ```sh bunx tubeless --help npx tubeless --help # Node launcher; relays through Bun automatically bunx --bun tubeless --help # Bun-only machines: force the Bun runtime ``` Library imports stay dependency-free compiled JavaScript. The CLI is the Bun workbench around those same exports. The binary's Node shebang makes it invocable through `npx` and any Node environment; when Bun is missing it prints install instructions instead of failing with `env: bun: No such file or directory`. On machines with Bun but no Node, `bunx` cannot use the Node shebang — pass `--bun` to force the Bun runtime. ## Commands | Command | Accepts | Does | | ------------------ | ----------------------------------- | ----------------------------------------------------------------------------- | | `tubeless list` | A project manifest | Lists explicitly registered command IDs without loading their modules | | `tubeless inspect` | A pipeline or command export | Prints identity (`id`, targets, exact steps) plus the default structural plan | | `tubeless plan` | A pipeline or command export | Previews selection without executing or requiring domain options | | `tubeless graph` | A pipeline or command export | Writes Mermaid flowchart source | | `tubeless run` | A `definePipelineCommand` export | Executes the command's own validated CLI contract | | `tubeless history` | An optional run id | Lists or shows recorded runs from SQLite or a finished NDJSON trace | | `tubeless ui` | An optional project/Studio manifest | Serves the local run studio; see [studio](https://tubeless.io/docs/studio.md) | Use a checked-in `tubeless.project.ts` to address commands by stable project ID: ```sh bunx tubeless list bunx tubeless inspect import-rows bunx tubeless plan import-rows --target normalized-import --explain bunx tubeless graph import-rows --markdown bunx tubeless run import-rows -- --source ../rows.txt --target normalized-import ``` The default manifest is only checked in the current directory; Tubeless never walks parent directories. Pass `--project ` to every command when the manifest has another name or location. With no `--project`, an existing file argument wins over manifest identity, then a bare name may resolve from the default manifest. Paths containing `/`, starting with `.`, or having an extension always remain file arguments. `--export` applies only to file mode; manifest registrations own export selection. The workbench discovers a single matching export automatically. Pass `--export Name` when the file exports more than one. `inspect`, `plan`, and `graph` prefer a marked command when a module exports both a pipeline and a command. ```sh bunx tubeless inspect ./scripts/import.ts bunx tubeless plan ./scripts/import.ts --target normalize --explain bunx tubeless graph ./scripts/import.ts --markdown bunx tubeless run ./scripts/import.ts -- --source rows.txt --target normalize ``` Workbench flags stay before `--`. Application flags belong after it. `tubeless run` never executes a raw pipeline or guesses domain options. ```sh tubeless run --export ImportCommand ./scripts/import.ts -- --source rows.txt --target normalize ``` For command help: ```sh tubeless run ./scripts/import.ts -- --help ``` ## List ``` tubeless list [options] ``` - `-p, --project ` selects the manifest (default `./tubeless.project.ts`) - `--json` emits its version, resolved manifest path, `cwd`, and registrations `list` evaluates the explicit manifest but does not load any registered command module. It never scans the filesystem or run history for executables. ## Inspect ``` tubeless inspect [options] ``` - `-e, --export ` selects a pipeline or command export - `-p, --project ` resolves the positional as a registered command ID - `--json` emits identity and the default plan as JSON ## Plan ``` tubeless plan [options] ``` - `-e, --export ` selects a pipeline or command export - `-p, --project ` resolves the positional as a registered command ID - `-t, --target ` selects a declared target and its prerequisites (repeatable) - `-s, --step ` selects exact internal steps (repeatable) - `--dry-run` shows each step's dry-run disposition - `--explain` includes target/dependency selection provenance - `--json` emits the structured plan Do not simulate planning with a `--plan` flag on the command itself. Use `command.plan()` or `tubeless plan`. `--target` and `--step` cannot be combined. ## Graph ``` tubeless graph [options] ``` - `-e, --export ` selects a pipeline or command export - `-p, --project ` resolves the positional as a registered command ID - `-d, --direction ` is `BT`, `LR`, `RL`, `TB`, or `TD` (default `TD`) - `--descriptions` includes step descriptions in node labels - `--markdown` wraps the result in a fenced Mermaid block The same graph is available in process as `pipeline.toMermaid()` or `command.toMermaid()`. ## Run ``` tubeless run [options] [-- ] ``` - `-e, --export ` selects a command export - `-p, --project ` resolves the positional as a registered command ID - `--store ` appends run events to a local SQLite database - `--trace ` writes NDJSON traces to a file, or `-` for stdout `--store` and `--trace` can be combined. Traces stay off stdout unless `--trace -` is set. When `--trace -` is set, the TTY reporter and command result go to stderr so stdout stays valid NDJSON. The binary does not attach OpenTelemetry; keep that SDK at the application exporter boundary. Programmatic callers can use `composeTraceExporters` from `tubeless/tracing` to fan one event stream out to multiple destinations with the same failure isolation. `run` accepts only a `definePipelineCommand` export. That export owns parsing, validation, option mapping, reporting, and the result summary. Omit `mapOptions` when validated flags already satisfy same-name pipeline options; keep it when names, types, defaults, or derived values differ. `--step` and `--target` stay the flag names; the parsed keys are `stepIds` and `targets`. A first script is [`cli-job.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/cli-job.ts). Local history with `--store` is covered in [the studio](https://tubeless.io/docs/studio.md) and `tubeless history`. ## Nested progress In a capable TTY, composed pipelines automatically show indented substeps: ```text ⠋ build-database ✓ prepare-artifacts ⠋ build-database [████░░░░] 50% 2/4 · validate-and-promote ``` `fromPipeline` shows child steps; `forEachPipeline` adds an item-key row above each child's steps. Deeper composition stays nested. Completed rows remain after parents settle, and failed, cancelled, and skipped work retain distinct states. Inner progress bars require the child to call `context.reportProgress`; lifecycle rows work without instrumentation. Tall trees use a live window with omitted-row counts and print the full retained tree at completion. Fan-out progress snapshots show at most 32 live item groups by default, prioritizing active items and failures; the complete tree is emitted once at settlement. `progress.detailLimit` overrides the live cap and caps the final snapshot too. Plain/non-TTY output uses aggregate progress messages. See [child composition](https://tubeless.io/docs/child-pipeline-composition.md) for item-group display limits and [fan-out progress](https://github.com/chetmancini/tubeless/blob/main/examples/fan-out-progress.ts) for an example. ## History ``` tubeless history [options] [run-id] ``` - `--store ` selects the SQLite database (default `.tubeless/runs.sqlite`) - `--trace ` selects a finished NDJSON trace artifact - `--pipeline ` filters by the exact recorded pipeline ID, not a registered command ID - `--json` emits the projected run list, or one projected run when `run-id` is set - `--events` emits raw store events as NDJSON (run-scoped when `run-id` is set) `--store` and `--trace` cannot be combined. `--json` and `--events` cannot be combined. `--pipeline` applies to every output mode and both artifact sources. With `run-id`, both selectors must match; a mismatch is an unknown run (exit `1`). A pipeline with no recorded runs returns an empty list or event stream (exit `0`). History reads recorded IDs directly without loading a project catalog or command module. The default print is the projection: a run list, or one run's steps, logs, and error. A missing store exits `2`. A store that cannot be opened read-only — a pending `-wal` or `-journal`, multiple hard links, or a file that is not a versioned run store — also exits `2` with an `Error:` line. An unknown run id exits `1`. An unflushed `export()` does not create that sidecar; `tubeless run --store` flushes at completion so a finished run is durable. A hard crash before then can lose up to 63 tail events. NDJSON is opened read-only, validated before projection, and assigned zero-based event ids in file order. By default, artifacts over 64 MiB, individual event lines over 1 MiB, and traces over 100,000 events are refused. Diagnostics name the line and invalid field without echoing its contents. Trace artifacts may contain sensitive logs, errors, and attributes; inspect only files you trust and avoid exposing Studio beyond the intended host. ```sh bunx tubeless run --store .tubeless/runs.sqlite --trace run.ndjson ./scripts/import.ts -- --source rows.txt bunx tubeless history bunx tubeless history --pipeline import bunx tubeless history --json bunx tubeless history --events bunx tubeless history --trace run.ndjson ``` ## Exit codes | Code | Meaning | | ---- | ------------ | | `0` | Success | | `1` | Usage | | `2` | Load | | `3` | Definition | | `4` | Validation | | `5` | Planning | | `6` | Execution | | `7` | Cancellation | These values are also exported as `TUBELESS_WORKBENCH_EXIT_CODE` from `tubeless/cli`. SIGINT is forwarded through the command context. Help (`--help` or a command's own help) exits `0`. --- Source: https://tubeless.io/docs/studio.md # Local studio Normal pipeline and CLI execution is storage-free. The studio is an optional local projection of an append-only SQLite event store or a finished NDJSON trace. Core does not import it. Record one workbench run by placing `--store` before the command file: ```sh bunx tubeless run --store .tubeless/runs.sqlite ./scripts/import.ts -- --source rows.txt ``` Inspect recorded runs without a browser: ```sh bunx tubeless history --store .tubeless/runs.sqlite bunx tubeless history --json bunx tubeless history --trace run.ndjson ``` Open the studio only when you want a browser view: ```sh bunx tubeless ui --store .tubeless/runs.sqlite bunx tubeless ui --trace run.ndjson ``` Both forms can be read-only. The NDJSON form is always read-only: it does not offer clear-history or accept a studio catalog / `--command`. Studio never guesses executable modules from recorded definitions. Register marked `definePipelineCommand` exports to make a **Run pipeline** action available: ```sh bunx tubeless ui \ --store .tubeless/runs.sqlite \ --command ./scripts/import.ts \ --command ./scripts/publish.ts ``` `--export` selects the export when you register exactly one `--command`. Launch forms render each command's structured parameter contract: booleans as checkboxes, constrained strings as selects, numbers with their bounds, and paths or unconstrained values as text fields. Submitted values go through the normal typed command parser without a shell. **Preview plan** uses the dry-run and stepIds/targets values from the same form; it never creates a run. Default bind address is `127.0.0.1`, default port is `4317`, and the default store is `.tubeless/runs.sqlite`. Browser-triggered execution requires a loopback host (`127.0.0.1`, `::1`, or `localhost`). Non-loopback binding is rejected when any command is registered. A non-loopback `--host` without commands is allowed and stays read-only: the store is visible to anyone who can reach the port, and clear-history is not wired. That bind is a risk you enable; see [SECURITY.md](https://github.com/chetmancini/tubeless/blob/main/SECURITY.md). NDJSON traces can include sensitive log messages, structured errors, and attributes. They are validated and bounded before serving, but their contents are not redacted; keep Studio on loopback unless disclosure is intentional. When `--host` is `0.0.0.0` or `::`, a matching `Host` cannot be the bind address, so the studio also accepts `localhost` or a literal IP on the same port. DNS names are still refused. ## Checked-in project manifest Declare stable IDs and module references once for every workbench surface: ```ts // tubeless.project.ts import { definePipelineProject } from "tubeless/workbench/project"; export default definePipelineProject({ cwd: ".", commands: [ { id: "import-rows", file: "./scripts/import.ts", export: "ImportCommand", name: "Import rows", }, { id: "publish", file: "./scripts/publish.ts", export: "PublishCommand" }, ], }); ``` ```sh bunx tubeless list bunx tubeless inspect import-rows bunx tubeless run import-rows -- --source rows.txt bunx tubeless ui --store .tubeless/runs.sqlite ./tubeless.project.ts ``` Command paths and `cwd` are relative to the manifest file. Empty or duplicate IDs and duplicate module registrations fail when the manifest loads. Presentation-name overrides do not change the registered, run, or pipeline identities. The Studio uses the stable registered ID in its local protocol. Legacy `definePipelineStudio` catalogs remain accepted by `tubeless ui`; their historical `file#export` Studio identity is unchanged. They are UI-only and do not participate in `list`, `inspect`, `plan`, `graph`, or `run` identity lookup. ## What the UI shows The studio combines active and historical runs in one running-first view. Child pipeline executions stay beneath their top-level run. Run details include step attempts, progress, logs, and structured errors. Nested pipeline steps are labeled with the child pipeline and its declared steps; runtime fan-out is identified without guessing its item count. Recorded progress keeps the last per-item `details` rows under the parent bar. Truncated lists keep the original `detail_count` and nested `step_count` so history can say how many rows were omitted. A writer's unflushed tail stays in that process; read-only history and studio inspect committed rows only and refuse after a drain creates a WAL, not on a single unflushed `export()`. Observed definitions use the latest `pipeline.started` timestamp, not insert order. Every studio request, including reads, requires a `Host` header that matches the bound authority (or, on a wildcard bind, `localhost` or a literal IP on the same port). Browser plan, launch, cancel, and clear-history also send `x-tubeless-studio-plan`, `x-tubeless-studio-launch`, `x-tubeless-studio-cancel`, and `x-tubeless-studio-clear-history`. Plan and launch require `application/json`. A successful launch response (HTTP 202) means the run id is already queryable from the store. Cancel aborts one live studio launch without stopping the server; it is not crash-resume. The Cancel run control appears only for top-level launches this studio process still owns. The header names are part of the local studio protocol. They are same-origin guards, not authentication. ## HTTP contract and errors The published [OpenAPI document](https://tubeless.io/openapi.json) describes the local Studio HTTP API. Its server URL is loopback-only; `tubeless.io` hosts the contract but does not execute pipelines. Unsuccessful Studio API responses use `application/json` and include four fields: stable `code`, human-readable `message`, actionable `hint`, and the backward-compatible `error` alias. Use `code` for control flow. The `error` and `message` fields currently contain the same text. Programmatic callers can compose the same pieces from `tubeless/run-store/sqlite`, `tubeless/run-store/ndjson`, and `tubeless/run-store/ui`. See [`local-observability.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/local-observability.ts). --- Source: https://tubeless.io/docs/comparison.md # Comparison Tubeless is an in-process typed DAG you import from TypeScript or run from a Bun CLI. It is not a hosted workflow engine, a job queue, or a warehouse scheduler. ## Same job: local typed steps These all run work in one process. The difference is how much graph, typing, and preview you get without writing it yourself. | Feature | tubeless | roll your own | p-graph | listr2 | | --------------------------------------- | -------------------------- | --------------- | ------- | --------------------- | | Typed step outputs through dependencies | Yes | If you write it | No | Weak (shared context) | | Invalid graphs rejected at definition | Yes | No | Partial | No | | Plan / preview without executing | Yes | No | No | No | | Dry-run and write gates | Yes | If you write it | No | No | | Declared targets and partial rerun | Yes | If you write it | No | No | | Child pipelines and fan-out | Yes | If you write it | No | Nested tasks | | Inspect / plan / graph / run CLI | Yes (Bun) | No | No | TTY task renderer | | Local run history | Optional SQLite or NDJSON | No | No | No | | Crash-resume the graph | No (file checkpoints only) | If you write it | No | No | Roll your own `await` / `Promise.all` is enough for two or three linear steps with no dry-run, no partial rerun, and no typed fan-out. p-graph is topo-order plus concurrency. listr2 is a terminal task list (pretty TTY, rollback), not a reusable typed data graph. File checkpoints (`openCheckpoint` in `tubeless/node`) record an "already done" set for batch API work. They do not replay a crashed process the way a durable workflow engine does. ## Different job | If you need… | Use | | ----------------------------------------------------- | --------------------------------------- | | Typed local DAG, plan, dry-run, write gates | tubeless | | Two or three `await`s and failure is "throw and exit" | roll your own | | Pretty CLI spinners, not a typed data graph | listr2 | | Survive process death, sleep for days, wait on humans | Temporal, Inngest, Trigger.dev, or DBOS | | Thousands of the same job with retries across workers | BullMQ, pg-boss, or graphile-worker | | Org-wide schedule, catalog, warehouse assets | Airflow, Dagster, or dbt | | Record-at-a-time streams | Node streams or RxJS | Those last four can still _call_ a tubeless pipeline. A queue worker or a durable step can run `pipeline.runOrThrow(...)` when the hard part inside the job is a gated, typed graph. | Need | Composition | Who drives the DAG | Who survives process death | | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------------------------- | | The graph must outlive the process, sleep for days, or wait on humans | Host embedding: a Temporal workflow, Lambda handler, or queue worker calls `pipeline.runOrThrow(...)` and passes `runId` / `parentRunId` | The engine | The engine | | Some steps run elsewhere; the rest stay local | `fromRemote`: one opaque parent step per remote unit of work | The tubeless process | Only the remote job, not the parent `PipelineRun` | ## When tubeless is the wrong default - The pipeline must outlive the process. Use a durable engine as the orchestrator. - The unit of work is one item, tens of thousands of times. Queue the items. - The consumer is a data platform, not a TypeScript repo. Use the platform. - You need a stable 1.x API or Windows. This package is `0.1.0`, and Windows is untested. The CLI runs through Bun; `npx tubeless` works wherever Bun is installed and otherwise prints Bun install instructions. --- Source: https://tubeless.io/docs/child-pipeline-composition.md # Child-pipeline composition ## What ships Compose a child pipeline as one typed parent step with `createSteps().fromPipeline(...)`. Overloads cover identity and mapped results. An async `fromPipeline` `mapResult` publishes its resolved value; dependents receive `Awaited` and skippable adapters accept that resolved shape as their skip value. Child lifecycle hooks stay isolated, and child work reports through parent-step progress. Interactive reporters display selected child steps as an indented tree, retaining their terminal states and inner work counts. Map a runtime item list onto the same child pipeline with `createSteps().forEachPipeline(...)`. Item keys are stable, concurrency is bounded, and results keep input order even when children finish out of order. All running children settle before an aggregate failure is returned, so the parent cannot finalize while child side effects are still in flight. Opaque parent steps pass outputs through typed dependencies, skip side effects in dry-run, and convert domain validation errors into failed parent steps. ## Why not call the child from an ordinary step Invoking a child pipeline from an ordinary parent step and passing the parent `PipelineStepContext` gives the child the parent's runtime seams, but it also gives the child the parent's raw hooks and parent-only `reportProgress` callback. The resulting lifecycle stream is flat. A direct nested run emits the child pipeline, step, finalize, and completion events between the parent wrapper step's start and completion events. Most events have no pipeline path or run identity. The interactive reporter also owns one plan, one step map, and one result. A child start replaces its visible plan, and a child completion disposes the reporter before the parent completes. The package needs a composition contract that preserves typing and runtime behavior without exposing two pipeline frames through hooks designed for one pipeline. ## Goals - Model a child pipeline as one typed step in the parent DAG. - Infer parent dependency inputs, parent options, child options, the child result, and any mapped result. - Preserve cancellation, logging, timing, working-directory, dry-run, progress, and failure behavior. - Keep raw child lifecycle hooks inside the adapter while reporting useful progress through the opaque parent step. - Keep the first slice bounded: one opaque child, or a mapped set of the same child. ## Supported: policy skip on opaque steps Policy skip (`step.skippable` / `StepSkipDecision`) is part of the shipped API and applies to ordinary steps, `fromPipeline` adapters, and mapped `forEachPipeline` adapters (the opaque parent step is a normal step under the hood): - Return a non-empty string or `{ reason, value? }` from `skip` to skip without calling `run` (or without running the child). Reporters show a yellow skip whose terminal report has `status: "skipped"` and `reason: "policy"`. - Policy skips unlock required dependents. A bare string (or `{ reason }` without `value`) publishes `undefined` as the step output. - Any step that declares `skip` is typed so dependents see `TOut | undefined` (including `PipelineResultOf | undefined` for `fromPipeline` and `readonly PipelineResultOf[] | undefined` for `forEachPipeline`, both without `mapResult`). Prefer `{ reason, value }` on every skip path when dependents need a real output. With `mapResult`, skip `value` is the parent-facing mapped output — one `TOut` for `fromPipeline`, or the complete `readonly TOut[]` for `forEachPipeline`. Policy skip does not call `mapResult`. Use `fromPipeline.skippable` or `forEachPipeline.skippable` to policy-skip optional child work and publish a concrete disabled result via `{ reason, value }`. The ordinary constructors reject `skip` and retain their non-optional output types. ## Non-goals - Flatten child steps into the parent plan. - Add hierarchical or versioned external lifecycle events. - Support namespaced selection such as `parent.child-step`. - Add parallel DAG execution, remote checkpoints, or subcommands. - Injected runner overrides (substituting a child pipeline implementation at test time via parent options) — separate from policy skip. ## Why this shape | Model | Typing | Planning | Hooks and reporting | Selective execution | Failure | Dry run | Cost | | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Continue manual shared-context calls | Each wrapper can map options and return a typed value, but each wrapper repeats the contract and structural context passing makes unsafe fields easy to forward. | The parent sees only the wrapper step; the child creates a separate plan at runtime. | Raw child events enter the parent's flat hook stream. The current interactive reporter replaces the parent frame and disposes on child completion. | Parent selection can select the wrapper. Child selection depends on ad hoc mapped `stepIds`. | `runOrThrow` can fail the wrapper, but consistent child-identifying messages and progress require repeated wrapper code. | Each wrapper must remember parent/child dry-run precedence and per-step policy. | Low per call, with continuing duplication and reporter breakage. | | Flatten the child DAG into the parent DAG | The API must translate child option and dependency types into a parent definition without losing inference. | Child ids need stable namespacing, and runtime option mapping can change which child plan is valid. | A flat plan fits current hooks only after defining pipeline identity, paths, and collision rules. | Namespaced filters need semantics for parent dependencies, child dependencies, and finalization. | The parent must define how child failures, partial outputs, and child finalization affect the flattened DAG. | The parent must reconcile its dry-run rules with every flattened child step. | High. It requires event identity and selection semantics before the first useful slice. | | Opaque typed child-step adapter | A parent-scoped factory can infer dependency inputs and parent options while a typed child pipeline constrains mapped options and results. | The parent plan contains one stable step. The adapter validates the child plan immediately before running it. | Parent hooks see the opaque step only. An internal bridge translates child terminal events into parent-step progress. | Selecting the opaque parent step runs the child according to mapped child options. Nested selection is deferred. | An unusable child result throws a parent-step error that retains the child pipeline and failing step identity. | Framework-owned parent `dryRun` overrides the mapper, then the child applies its own per-step rules. | Medium. It uses existing pipeline and hook primitives without changing the external event model. | Shared raw hooks are rejected because they already corrupt the interactive reporter's single frame and cannot distinguish nested runs. Immediate flattening is rejected because it needs stable identity, selection, dependency, failure, and finalization rules that the current event and plan models don't provide. The opaque adapter addresses the existing composition case without deciding those broader contracts. ## Contract The parent DAG contains one ordinary step. That step maps its typed parent inputs and context into typed child options, validates the child plan, runs the child with an isolated internal hook bridge, and returns either the child result or a mapped parent result. The bridge owns child lifecycle events. Parent and custom hooks receive only the parent pipeline and opaque parent-step events. The bridge reports terminal child-step counts through the parent step's `reportProgress`, so existing reporters keep one live frame. ## API The parent-scoped step factory owns the adapter: ```ts const seedIndexStage = step.fromPipeline("seed-index", { pipeline: IndexSeedPipeline, dependsOn: [seedCatalogStage], description: "Seed a precomputed search index", mapOptions: (_inputs, context) => ({ indexDir: context.options.indexDir, syncSchema: false, }), mapResult: () => ({ ran: true, stageId: "seed-index" }), }); ``` `StepFactory.fromPipeline` and `StepFactory.fromPipeline.skippable` infer the child result when `mapResult` is absent, and infer the mapped result when it is present. For runtime-selected sets (any domain — shards, files, jobs, catalog rows): ```ts const processShards = step.forEachPipeline("process-shards", { pipeline: ShardPipeline, dependsOn: [resolveShards], items: ({ "resolve-shards": shards }) => shards, key: (shard) => shard.id, concurrency: (_inputs, context) => context.options.concurrency, // Optional presentation only. Defaults to the noun "items". progress: { itemNoun: "shards" }, mapOptions: (shard, _index, _inputs, context) => ({ shardPath: shard.path, outputRoot: context.options.outputRoot, }), }); ``` When the whole fan-out may be intentionally unnecessary, opt into the widened output explicitly: ```ts const processShards = step.forEachPipeline.skippable("process-shards", { pipeline: ShardPipeline, dependsOn: [resolveShards], skip: ({ "resolve-shards": shards }) => shards.length === 0 ? { reason: "no shards selected", value: [] } : false, items: ({ "resolve-shards": shards }) => shards, key: (shard) => shard.id, mapOptions: (shard) => ({ shardPath: shard.path }), }); ``` Only `.skippable` produces `readonly ChildResult[] | undefined` (or `readonly TOut[] | undefined` with `mapResult`). A valued skip supplies the complete parent-facing array and bypasses `items`, child execution, and `mapResult`. The parent sees one opaque `process-shards` step. Progress is domain-neutral by default: a one-line item/concurrency summary plus structured `details` rows for each item and its selected child steps. Interactive reporters render these as an indented tree: parent step, item key, child steps, and any deeper composition. Pending, running, completed, failed, cancelled, and skipped rows retain their states; filtered steps are omitted. Completed item groups remain in the tree. `fromPipeline` provides the same child-step breakdown without an item-key level. Inner `reportProgress` counts and details survive both composition boundaries. The progress bar advances on terminal child steps so long fan-out work does not look hung at 0%. Override `progress.itemNoun` or `progress.formatMessage` for domain labels without changing scheduling. Duplicate keys fail before any child starts. Parent dry-run overrides the mapped child run object's `dryRun` value. Live fan-out snapshots show up to 32 item groups by default, prioritizing active items and then failures while keeping displayed groups in input order. Descendants stay with their item; an overflow row reports omitted groups. All groups are retained and the complete default tree is emitted once after the fan-out settles. `progress.detailLimit` overrides the live cap and also caps the final snapshot; larger values increase per-event presentation work. Progress payloads are not constructed when neither hooks nor tracing observe them. The standalone `mappedChildProgressDetails` helper continues to format only the active entries supplied in its snapshot, with no default cap. Progress `details` use preorder rows with `depth: 0` (or omitted) for direct children. Each composition level adds one to descendant depths. IDs are stable within their containing group; `name` provides optional display text. `completed` and `total` describe inner work independently of the parent's terminal-step count. Reporters cap visual indentation at 32 levels. Trace snapshots preserve these fields within 128-row, 4096-character-per-field, and 256 KiB encoded-payload bounds, with `detail_count` recording the pre-truncation row count. The byte budget includes UTF-8 and escaping inside the enclosing JSON event, leaving room for event metadata below the default 1 MiB NDJSON event limit. The interactive CLI keeps completed details after their parent settles. Trees larger than the terminal use a live window around active work, with omitted-row counts; the full retained tree prints at completion. Plain/non-TTY reporting continues to emit aggregate progress messages. Presentation does not change parent plans, child selection, hook isolation, scheduling, or result types. ## Semantics matrix | Concern | Contract | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Parent plan | The child is one opaque parent step. Its `nestedPipeline` metadata identifies the child pipeline, declared child step ids, and single/fan-out mode; child steps are not added to `PipelinePlan.steps` and runtime selection is not predicted. | | Option mapping | `mapOptions(inputs, parentContext)` returns the child's complete run object: domain fields plus any child-specific controls. The adapter applies parent `dryRun` last so framework semantics win. | | Other run controls | The adapter does not automatically propagate parent `stepIds`, `targets`, or `continueOnError`. `mapOptions` may choose child-specific values; child `targets` are limited to that child's declared public goals. Mapping parent selection would usually be invalid because parent and child ids occupy different namespaces. Mapping `continueOnError` lets independent child work finish but does not make a failed child usable by the parent. | | Runtime context | The adapter forwards `cwd`, `log`, `now`, `sleep`, and `signal` field by field. It does not forward parent hooks or the parent-only `reportProgress` field. | | Hook isolation | Parent and custom hooks see only the parent pipeline and opaque parent step. Raw child lifecycle events go only to an internal bridge. | | Progress | Single-child `fromPipeline` bridges terminal child steps into the opaque step (`completed` / plan step count, messages prefixed with `childPipelineId/childStepId`). Mapped `forEachPipeline` reports domain-neutral fan-out progress via `toMappedChildStepProgress`: `completed` counts terminal child steps across items, `total` is `items × stepsPerItem`, `message` is a one-line summary, and `details` contains a bounded live item tree and a complete default final tree. Presentation is optional (`progress.itemNoun` / `progress.formatMessage` / `progress.detailLimit`); neither path creates another live reporter frame. | | Result | The default output is the typed child result. `mapResult` may transform it into a domain-specific parent-step output. A child result is usable only when its status is `"completed"`; best-effort child results must be handled through an explicit ordinary step that calls `run()` and inspects the structured result. | | Failure | An unusable child result fails the opaque parent step. Its error names the child pipeline and the first failing child step and message, with `code: "TUBELESS_CHILD_FAILED"`, `phase: "execution"`, and `kind: "child"`; its JSON-safe cause chain retains the actionable child failure. When mapped children contain both cancellation and genuine failure, failure takes precedence; the aggregate is cancelled only when every unsuccessful child was cancelled. | | Cancellation | The child receives the exact parent `AbortSignal`. An already-aborted or later-aborted signal stops the child and transitions the parent step to `cancelled`. | | Dry run | The child receives `dryRun: parentContext.dryRun` and applies its own per-step policy. Child steps with `dryRun: "skip"` are skipped; ordinary child steps run normally. | | Selective execution | Selecting or targeting the opaque parent step runs the child according to its mapped controls. Parent selection is never forwarded into the child's ID namespace. | | Checkpoints | The adapter adds no checkpoint behavior. The child receives mapped options and the normal runtime context. | ## Guarantees Call sites infer required and optional dependency inputs, parent domain options, required child options, the default child result, and a transformed `mapResult` output without explicit generic arguments. Missing required child options fail at compile time. The parent plan contains only the opaque stage. The child receives the parent's working directory, logger, clock, sleep function, and abort signal. Parent hooks see no raw child lifecycle events. The internal bridge converts the child's canonical step statuses into monotonic progress on the opaque parent step. Parent dry-run precedence prevents a child side-effecting step from running. An unusable child result fails one opaque parent step and names the child pipeline, step, and error, including when `continueOnError` produced a best-effort final value. Later cancellation interrupts in-flight child work. An invalid child plan fails before the child starts. `fromPipeline` and `forEachPipeline` identity forms (no `mapResult`) use overloads so the parent-facing outputs stay the child result and child-result array, respectively. Their ordinary constructors remain non-optional; `.skippable` alone adds `undefined`. ## Out of scope Policy skip plus explicit `undefined` typing cover skip-without-run for opaque single and mapped child steps. Runner substitution and any richer skipped-output shape beyond policy skip are not supported. Hierarchical or versioned external events, flattened or namespaced child selection, and parallel DAG scheduling remain out of scope. Mapped children provide bounded parallelism inside one opaque parent step; they do not make the pipeline DAG executor parallel. ## Structured fan-out failures A failed or cancelled `forEachPipeline` step exposes `error.fanOut` on the parent run error and step report (also available to failure hooks, JSON error rendering, and recorded trace history). `failures` contains at most the first 32 failed started items in input order, with the original `index`, `key`, `keyTruncated`, `cancelled`, and a JSON-safe `error` cause snapshot. `failureCount` counts all failed started items; `omittedFailureCount` counts entries beyond the limit. Scheduler failures appear separately as `schedulerError`; unstarted items are not failures. Setup errors, such as duplicate keys or an exception in `items`, have no `fanOut` diagnostic. Keys and snapshot strings are capped at 1024 UTF-16 code units, and cause chains use the existing eight-level bound. `keyTruncated` explicitly marks shortened keys; use the original input index to recover those identities. Snapshots retain messages, names, source codes, and causes, without stacks, options, successful outputs, or nested run objects. Nested fan-outs do not recursively expand here. Use complete keys to select original inputs for a caller-directed rerun. Check `omittedFailureCount` before treating the list as exhaustive, and account for unstarted work after cancellation. Reruns are ordinary new pipeline runs; the caller owns retry policy and side-effect safety. See the helper in [`fan-out-progress.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/fan-out-progress.ts). This additive diagnostic does not change parent failure/cancellation precedence, primary causes, scheduling, downstream dependency skips, or `continueOnError`. Successful siblings still do not produce a partial parent step output. --- Source: https://tubeless.io/docs/remote-step-composition.md # Remote-step composition ## What ships Compose a remote engine as one typed parent step with `createSteps().fromRemote(...)`. The factory builds an ordinary step whose `run` maps input and calls `RemoteStepAdapter.invoke`. Overloads cover required `adapter`, `mapInput`, and `outputSchema`. Optional `fromRemote.skippable` widens dependents the same way as `fromPipeline.skippable`. One opaque parent step stays in the parent plan. `executePlannedRun` does not learn about engines; disposition, output validation, hooks, and reports apply unchanged. ## Two compositions | Need | Composition | Who drives the DAG | Who survives process death | | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ------------------------------------------------- | | The graph must outlive the process, sleep for days, or wait on humans | Host embedding: a Temporal workflow, Lambda handler, or queue worker calls `pipeline.runOrThrow(...)` and passes `runId` / `parentRunId` | The engine | The engine | | Some steps run elsewhere; the rest stay local | `fromRemote`: one opaque parent step per remote unit of work | The tubeless process | Only the remote job, not the parent `PipelineRun` | These are complementary. A host can persist job delivery and retry a whole invocation; calling `runOrThrow` does not checkpoint the graph. Put arbitrary pipeline I/O in an activity or worker handler, not a replayed workflow body. The host owns durable orchestration, idempotency, retries, and acknowledgements. Neither correlation IDs nor a remote invoker provide crash-resume. ## Adapter mapping `RemoteStepAdapter.invoke(payload, context)` is request/response. Temporal start-then-`result()`, Lambda invoke, HTTP RPC, and a queue that blocks until the job finishes all live inside the adapter. The adapter maps: - `context.signal` to cancel - `context.reportProgress` to heartbeats or status polls - `context.log` to remote output when the engine can stream or poll it - `context.dryRun` to the engine's side-effect gate - remote failures to a thrown `Error` with optional `cause` and `code` `PipelineStepContext` does not grow a `stepId` field. Job ids come from the `fromRemote("id", …)` call site, usually via `mapInput`. ## Inverse dry-run contract Omitting `dryRun` on `fromRemote` means the adapter is contacted during a pipeline dry run. That makes this adapter-side rule the only safety line: - When `context.dryRun === true`, `invoke` must not produce side effects. - The remote worker must treat the same flag the same way. - The kernel does not inject `dryRun` into the payload and does not inspect the engine. Authors prove the flag crossed the boundary by putting `dryRun: ctx.dryRun` (or the engine's equivalent) in `mapInput`. - There is no way to call `invoke` while lying about dry-run. `context.dryRun` is the pipeline's actual flag. | Author writes | Meaning | Pipeline dry run | | ---------------------------- | ------------------------------------------ | ---------------------------------------- | | omit | Work is safe, or the engine honors dry-run | Call `invoke`; `context.dryRun === true` | | `dryRun: "skip"` | This work is the side effect | Do not call `invoke` | | `dryRun: (inputs, ctx) => …` | A local preview is enough | Call the handler only | Authors do not write `dryRun: "run"`. That token is not part of the authoring surface. ## Local visibility Remote work stays one opaque parent step. Local studio, hooks, and the TTY already observe that step. **Logs are optional.** If the engine can stream or poll output while `invoke` is in flight, the adapter calls `context.log.log` / `warn` / `error` as lines arrive. If the engine has no stream, skip it. **Exceptions are required.** Do not swallow a remote failure. Rethrow an `Error`. Wrap the remote object as `cause`. Copy a machine `code` onto the thrown error when the engine has one. Cancellation still goes through `context.signal`. No `TUBELESS_REMOTE_*` codes and no `streamLogs` flag on `fromRemote`. **Progress is the latest heartbeat; logs are the append-only narrative.** Use `context.reportProgress` for status and a job-id detail row. Use `context.log` for lines. Both can run together. ## Non-goals - A placement driver that compiles the DAG into a Temporal workflow. - Official Temporal, AWS, or HTTP SDKs in the kernel. - Flattening remote activity, workflow, or request IDs into the parent DAG. - Injected runner overrides (same rejection as child pipelines). - `fromRemote` `mapResult`. Reshape in the adapter or a later local step. ## Executable HTTP boundary [`remote-steps.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/remote-steps.ts) uses native `fetch`, with no provider dependency. Call `runRemoteStepsExample("http://127.0.0.1:8080")` against a service implementing this application-owned protocol: - `POST /enrich`, JSON body `{ rows: string[], runId: string, dryRun: boolean }`. - Success: a 2xx JSON response `{ orderId: string, rows: string[] }`. - Failure: a non-2xx status. The adapter throws with code `HTTP_` and a status cause, without copying response bodies into diagnostics. The service must validate the request and gate all writes on `dryRun`. A pipeline dry run still sends this request. If the service cannot guarantee a side-effect-free preview, add `dryRun: "skip"` or a local preview handler instead. `outputSchema` checks the object and every row before the local summary step runs; malformed JSON and invalid output fail the step. Customize authentication and endpoint configuration in the application adapter. Cancellation passes the caller signal through to `fetch`, aborting the HTTP wait. It does not guarantee that the server stopped work or rolled back writes. A service that starts durable jobs needs its own cancellation protocol. Automatic retries are deliberately absent: the service must define idempotency before retrying a request that may already have committed. The integration tests in `src/testing/remote-steps.example.test.ts` use a real loopback server, covering live and dry-run requests, invalid output, malformed JSON, HTTP errors, and an aborted in-flight connection. The packed-artifact check also runs the recipe over HTTP without credentials or an external service. ## Separate host handler [`host-embedding.ts`](https://github.com/chetmancini/tubeless/blob/main/examples/host-embedding.ts) exports `handleHostJob` for a queue worker or activity handler. It passes host-owned `runId`, `parentRunId`, `dryRun`, and `signal` to `runOrThrow`. Failed or cancelled runs reject, allowing the host to withhold acknowledgement and apply its failure policy. Assign a unique run ID per invocation/attempt and a stable parent workflow ID. Validate the job envelope before invocation; keep host SDKs and persistence outside the pipeline module. --- Source: https://tubeless.io/docs/agent-evaluations.md # Agent evaluations The cases in [`../evals/agent-cases.json`](https://github.com/chetmancini/tubeless/blob/main/evals/agent-cases.json) test whether an agent can discover and apply the package's important patterns. Prompts and provider invocation remain provider-neutral. ## Generate a submission 1. Start a clean agent session with access to the repository. 2. Tell it to use the repository-local `tubeless` skill ([`skills/tubeless/SKILL.md`](https://github.com/chetmancini/tubeless/blob/main/skills/tubeless/SKILL.md)). 3. Give it exactly one case's `prompt`; do not reveal the expectations. 4. Have it write into a disposable submission directory outside production paths. The directory must contain a regular, non-symlink `solution.ts`. Additional `.ts`, `.tsx`, `.mts`, or `.cts` helper files are compiled with it. An optional `assessment.json` records operator scoring. ## Compile a submission ```sh bun run eval:agent -- \ --case sequential-import \ --submission .context/agent-eval/sequential-import \ --report .context/agent-eval/sequential-import-report.json ``` The evaluator builds and packs the current package, installs that artifact into a disposable project, compiles every submission source against its public declarations, emits a stable JSON report, and deletes the project. Pass `--package` to reuse an existing tarball. The evaluator never executes submission code. Run behavioral tests only in an external disposable sandbox with its own process, filesystem, and network containment. Exit status is `0` when compilation passes and no assessed expectation fails, `1` for a compile or assessed expectation failure, and `2` for invalid runner configuration. ## Score semantic expectations Compilation cannot decide whether a design materially demonstrates an expectation. An operator may add `assessment.json` without changing the generated solution: ```json { "schemaVersion": 1, "caseId": "sequential-import", "expectations": [ { "category": "mustDemonstrate", "expectation": "typed options", "status": "pass", "evidence": "ImportOptions types context.options.lines." } ] } ``` Categories and expectation text must exactly match the selected case. Partial assessments are valid; remaining entries stay `unscored`. The report contains normalized compiler diagnostics, installed package identity, submission file names, every semantic expectation in case order, `mechanicalStatus`, `assessmentStatus`, and the combined `ok` value. `make check` runs four representative fixtures: a valid assessed pipeline, compiler-rejected invented API usage, artifact-specific declarations, and a multi-file TSX submission. It also compiles every `learningSurfaceGate` case's assessed answer under [`evals/answers/`](https://github.com/chetmancini/tubeless/blob/main/evals/answers) and fails when compilation or any assessed expectation fails. The evaluator still does not execute submission code. ## Passing standard - The result typechecks using public package entrypoints. - Every `mustDemonstrate` item is materially present, not merely mentioned. - No `mustAvoid` behavior appears. - The solution uses the smallest fitting primitive and preserves dry-run and cancellation safety. - The agent finds the relevant recipe without reading executor implementation. The simple, safety-sensitive, fan-out, and CLI cases are `learningSurfaceGate` cases. Their answers run on every `make check`. Re-run those four against a fresh agent when changing the learning surface, and run all cases before a public release or declaration API redesign. --- Source: https://tubeless.io/docs/api-reference.md # `tubeless` API reference Generated from the package's emitted declaration files. Do not edit manually; run `bun run api:generate` from the package root. Package: `tubeless` ## Public entrypoints | Entrypoint | Declaration | Surface hash | Exported symbols | | ---------------------------- | ----------------------------------------- | ------------------------------------------------------------------ | ---------------: | | `tubeless` | `./dist/core/pipeline.d.ts` | `4c9eea9c3243f304493c43578af2c64d9406610054db5694519145854393ac15` | 75 | | `tubeless/batch` | `./dist/utilities/batch.d.ts` | `6b7fe7d6eec6532cc835505517c3481e4290a2a606c2fddf6c80367f5c76c5f5` | 7 | | `tubeless/cli` | `./dist/cli/cli.d.ts` | `afcd70f9ca23dac58af7a5256fb593cf3dd9b4f4718910ba30a6e0db838f3fcc` | 35 | | `tubeless/node` | `./dist/node/node.d.ts` | `2ded0b2084ecf401c09b414d3af7108daf149192f0cbd7efc03f1ca7020b320f` | 8 | | `tubeless/rate-limit` | `./dist/utilities/rate-limit.d.ts` | `5cd093bb780a19e44b087b01e2b38e06f07e7233c9920988ba399036e319c368` | 1 | | `tubeless/render` | `./dist/render/render.d.ts` | `5b3c6183d1df53eb5474cde672eb2614f6887381d65656544c30a9c5ce8d7b37` | 6 | | `tubeless/reporter` | `./dist/reporter/reporter-entry.d.ts` | `b745445b09b8122bd58c7c40eb910b9ccfd916c4664cd2e2d301a2b98961e59e` | 13 | | `tubeless/retry` | `./dist/utilities/retry.d.ts` | `f91f83f8e31e3e629ed90d1b04656110572ea970187a92369f0908cd75ae8f00` | 4 | | `tubeless/run-store` | `./dist/run-store/run-store.d.ts` | `db7d9e8c4f82bcb22d5ec7eecb47814f9fdfc961a8a11fc805397042a7040b7f` | 18 | | `tubeless/run-store/sqlite` | `./dist/run-store/run-store-sqlite.d.ts` | `a9363f5b1c72ebfd508bd66827e5a931d243ed2a804dbd99070e8abd6d0010d1` | 3 | | `tubeless/run-store/ndjson` | `./dist/run-store/run-store-ndjson.d.ts` | `011bef12c4c8bc4b1eedfb5dba72e6d1eccf4ebf04a7306837fef978a32fc53b` | 3 | | `tubeless/workbench/studio` | `./dist/workbench/workbench-studio.d.ts` | `6108782771ca4752fa95473cf5cbb1353e0bf2cda32554cfd20fcfefb21010b8` | 6 | | `tubeless/workbench/project` | `./dist/workbench/workbench-project.d.ts` | `eac0931b2968d98c1b7d41503c1ea61477a2685e8cfebd22863255fce67085a7` | 6 | | `tubeless/run-store/ui` | `./dist/studio/run-store-ui.d.ts` | `69e1f6c6508507b8cd346dc94c6a877b1e8b22d29e2c9f9212366677afc6d91b` | 9 | | `tubeless/testing` | `./dist/testing/testing.d.ts` | `150386a43659704068ce0219dbacb8f71dd9d83f35ac0859c197e59721ab96ea` | 7 | | `tubeless/tracing` | `./dist/tracing/tracing.d.ts` | `4c9eea9c3243f304493c43578af2c64d9406610054db5694519145854393ac15` | 9 | | `tubeless/tracing/json` | `./dist/tracing/tracing-json.d.ts` | `d19223de841ea24d9c6ab6342bbfdcbeb158f700e54b4072a7d334a309add820` | 2 | | `tubeless/tracing/otel` | `./dist/tracing/tracing-otel.d.ts` | `1ea929831eb01e14e7717bb13e12f1bc6c686411eed41849cbdd493490bfe06a` | 4 | ## Symbols ### `tubeless` - `AnyStep` - `createRunId` - `createSteps` - `defaultPipelineContext` - `definePipeline` - `formatMappedChildProgressMessage` - `FormatMappedChildProgressOptions` - `InferSchemaInput` - `InferSchemaOutput` - `MappedChildProgressDetail` - `mappedChildProgressDetails` - `MappedChildProgressOptions` - `MappedChildProgressSnapshot` - `mappedChildProgressUnits` - `MappedChildProgressUnits` - `Pipeline` - `PIPELINE_FINALIZE_STEP_ID` - `PIPELINE_MERMAID_DIRECTIONS` - `PipelineContext` - `PipelineDefinition` - `PipelineDefinitionError` - `PipelineError` - `PipelineErrorCause` - `PipelineErrorCode` - `PipelineErrorKind` - `PipelineErrorPhase` - `PipelineExecutionContext` - `PipelineExecutionError` - `PipelineFanOutDiagnostics` - `PipelineFanOutFailure` - `PipelineHooks` - `PipelineLogger` - `PipelineMermaidDirection` - `PipelineMermaidOptions` - `PipelinePlan` - `PipelinePlanStep` - `PipelineRun` - `PipelineRunControls` - `PipelineRunOptions` - `PipelineRunStatus` - `PipelineRuntime` - `PipelineStepCancelledEvent` - `PipelineStepCancelledReport` - `PipelineStepCompleteEvent` - `PipelineStepCompleteReport` - `PipelineStepContext` - `PipelineStepFailedEvent` - `PipelineStepFailedReport` - `PipelineStepLifecycleStatus` - `PipelineStepPlannedEvent` - `PipelineStepProgress` - `PipelineStepProgressDetail` - `PipelineStepProgressDetailStatus` - `PipelineStepProgressEvent` - `PipelineStepReport` - `PipelineStepReportStatus` - `PipelineStepSelectionReason` - `PipelineStepSkippedEvent` - `PipelineStepSkippedReport` - `PipelineStepSkipReason` - `PipelineStepStartEvent` - `PipelineStepStatus` - `PipelineValidationIssue` - `RemoteStepAdapter` - `requireOutputs` - `RUN_MODEL_VERSION` - `StandardSchemaV1` - `StandardSchemaV1Issue` - `StandardSchemaV1Props` - `StandardSchemaV1Result` - `Step` - `StepFactory` - `StepSkipDecision` - `toMappedChildStepProgress` - `ToMappedChildStepProgressOptions` ### `tubeless/batch` - `chunk` - `ConcurrentSettleResult` - `ConcurrentWorker` - `runBatched` - `runConcurrent` - `RunConcurrentOptions` - `runConcurrentSettled` ### `tubeless/cli` - `CliBooleanParam` - `CliCheckpointConfig` - `CliCommand` - `CliCommandConfig` - `CliCommandDescriptor` - `CliContext` - `CliHelpRequested` - `CliNumberParam` - `CliParam` - `CliParameterDescriptor` - `CliParams` - `CliParamsSchema` - `CliParamType` - `CliParseResult` - `CliPathParam` - `CliStringParam` - `CliValidationError` - `defineCommand` - `definePipelineCommand` - `DefinePipelineCommandConfig` - `MultiSelectChoice` - `MultiSelectResult` - `normalizeMultiSelectChoices` - `parseMultiSelectInput` - `ParseMultiSelectInputOptions` - `PipelineCliBuiltins` - `PipelineCliParseResult` - `PipelineCliValues` - `PipelineCommand` - `PipelineCommandHookConfig` - `PipelineCommandHookContext` - `PipelineCommandHookSets` - `promptMultiSelect` - `PromptMultiSelectOptions` - `TUBELESS_WORKBENCH_EXIT_CODE` ### `tubeless/node` - `CheckpointStore` - `definePaths` - `openCheckpoint` - `readJson` - `requireEnv` - `resetDir` - `withCheckpointedBatch` - `writeJson` ### `tubeless/rate-limit` - `RateLimiter` ### `tubeless/render` - `PipelineHumanRenderOptions` - `PipelineJsonRenderOptions` - `PipelinePlanRenderOptions` - `PipelineRenderOptions` - `renderPipelineError` - `renderPipelinePlan` ### `tubeless/reporter` - `createPipelineReporter` - `createRunReporter` - `PipelineReporterConfig` - `PipelineReporterController` - `PipelineReporterMode` - `PipelineReporterOptions` - `ReporterColorMode` - `ReporterOutput` - `ReporterSymbolMode` - `ReporterTerminalCapabilities` - `ResolvedPipelineReporterMode` - `RunReporterConfig` - `RunReporterOptions` ### `tubeless/retry` - `RetryAttemptContext` - `RetryOperation` - `RetryOptions` - `withRetry` ### `tubeless/run-store` - `createPipelineRunProjector` - `PipelineRunEventQuery` - `PipelineRunEventReader` - `PipelineRunEventStore` - `PipelineRunProjector` - `PipelineRunStoreSnapshot` - `projectPipelineRun` - `projectPipelineRunStore` - `StoredNestedPipeline` - `StoredPipelineAttempt` - `StoredPipelineDefinition` - `StoredPipelineDefinitionStep` - `StoredPipelineEvent` - `StoredPipelineLog` - `StoredPipelineRun` - `StoredPipelineRunStatus` - `StoredPipelineStep` - `StoredRemote` ### `tubeless/run-store/sqlite` - `openSqlitePipelineRunStore` - `OpenSqlitePipelineRunStoreOptions` - `SqlitePipelineRunStore` ### `tubeless/run-store/ndjson` - `NdjsonPipelineRunStore` - `openNdjsonPipelineRunStore` - `OpenNdjsonPipelineRunStoreOptions` ### `tubeless/workbench/studio` - `definePipelineStudio` - `isPipelineStudioConfig` - `PIPELINE_STUDIO_CONFIG_VERSION` - `PipelineStudioCommandModule` - `PipelineStudioConfig` - `PipelineStudioConfigInput` ### `tubeless/workbench/project` - `definePipelineProject` - `isPipelineProjectManifest` - `PIPELINE_PROJECT_MANIFEST_VERSION` - `PipelineProjectCommandModule` - `PipelineProjectManifest` - `PipelineProjectManifestInput` ### `tubeless/run-store/ui` - `PipelineRunStudioCancelResult` - `PipelineRunStudioCommand` - `PipelineRunStudioHistoryMaintenance` - `PipelineRunStudioLauncher` - `PipelineRunStudioLaunchRequest` - `PipelineRunStudioLaunchResult` - `PipelineRunStudioOptions` - `PipelineRunStudioServer` - `startPipelineRunStudio` ### `tubeless/testing` - `createPipelineTestRuntime` - `PipelineTestClock` - `PipelineTestLogEntry` - `PipelineTestLogLevel` - `PipelineTestRuntime` - `PipelineTestRuntimeOptions` - `PipelineTestSleep` ### `tubeless/tracing` - `composeTraceExporters` - `PipelineTraceAttributes` - `PipelineTraceAttributeValue` - `PipelineTraceContext` - `PipelineTraceError` - `PipelineTraceEvent` - `PipelineTraceEventName` - `PipelineTraceExporter` - `PipelineTracingOptions` ### `tubeless/tracing/json` - `createJsonTraceExporter` - `JsonTraceExporterOptions` ### `tubeless/tracing/otel` - `createOpenTelemetryTraceExporter` - `OpenTelemetryLikeSpan` - `OpenTelemetryLikeTracer` - `OpenTelemetryTraceExporterOptions`