Child-pipeline composition

What ships

Compose a child pipeline as one typed parent step with createSteps<TOptions>().fromPipeline(...). Overloads cover identity and mapped results. An async fromPipeline mapResult publishes its resolved value; dependents receive Awaited<TOut> and skippable adapters accept that resolved shape as their skip value. Child lifecycle hooks stay isolated, and child work reports through parent-step progress. Interactive reporters display selected child steps as an indented tree, retaining their terminal states and inner work counts.

Map a runtime item list onto the same child pipeline with createSteps<TOptions>().forEachPipeline(...). Item keys are stable, concurrency is bounded, and results keep input order even when children finish out of order. All running children settle before an aggregate failure is returned, so the parent cannot finalize while child side effects are still in flight.

Opaque parent steps pass outputs through typed dependencies, skip side effects in dry-run, and convert domain validation errors into failed parent steps.

Why not call the child from an ordinary step

Invoking a child pipeline from an ordinary parent step and passing the parent PipelineStepContext gives the child the parent's runtime seams, but it also gives the child the parent's raw hooks and parent-only reportProgress callback.

The resulting lifecycle stream is flat. A direct nested run emits the child pipeline, step, finalize, and completion events between the parent wrapper step's start and completion events. Most events have no pipeline path or run identity. The interactive reporter also owns one plan, one step map, and one result. A child start replaces its visible plan, and a child completion disposes the reporter before the parent completes.

The package needs a composition contract that preserves typing and runtime behavior without exposing two pipeline frames through hooks designed for one pipeline.

Goals

• Model a child pipeline as one typed step in the parent DAG.
• Infer parent dependency inputs, parent options, child options, the child result, and any mapped result.
• Preserve cancellation, logging, timing, working-directory, dry-run, progress, and failure behavior.
• Keep raw child lifecycle hooks inside the adapter while reporting useful progress through the opaque parent step.
• Keep the first slice bounded: one opaque child, or a mapped set of the same child.

Supported: policy skip on opaque steps

Policy skip (step.skippable / StepSkipDecision) is part of the shipped API and applies to ordinary steps, fromPipeline adapters, and mapped forEachPipeline adapters (the opaque parent step is a normal step under the hood):

• Return a non-empty string or { reason, value? } from skip to skip without calling run (or without running the child). Reporters show a yellow skip whose terminal report has status: "skipped" and reason: "policy".
• Policy skips unlock required dependents. A bare string (or { reason } without value) publishes undefined as the step output.
• Any step that declares skip is typed so dependents see TOut | undefined (including PipelineResultOf<Child> | undefined for fromPipeline and readonly PipelineResultOf<Child>[] | undefined for forEachPipeline, both without mapResult). Prefer { reason, value } on every skip path when dependents need a real output. With mapResult, skip value is the parent-facing mapped output — one TOut for fromPipeline, or the complete readonly TOut[] for forEachPipeline. Policy skip does not call mapResult.

Use fromPipeline.skippable or forEachPipeline.skippable to policy-skip optional child work and publish a concrete disabled result via { reason, value }. The ordinary constructors reject skip and retain their non-optional output types.

Non-goals

• Flatten child steps into the parent plan.
• Add hierarchical or versioned external lifecycle events.
• Support namespaced selection such as parent.child-step.
• Add parallel DAG execution, remote checkpoints, or subcommands.
• Injected runner overrides (substituting a child pipeline implementation at test time via parent options) — separate from policy skip.

Why this shape

Shared raw hooks are rejected because they already corrupt the interactive reporter's single frame and cannot distinguish nested runs. Immediate flattening is rejected because it needs stable identity, selection, dependency, failure, and finalization rules that the current event and plan models don't provide. The opaque adapter addresses the existing composition case without deciding those broader contracts.

Contract

The parent DAG contains one ordinary step. That step maps its typed parent inputs and context into typed child options, validates the child plan, runs the child with an isolated internal hook bridge, and returns either the child result or a mapped parent result.

The bridge owns child lifecycle events. Parent and custom hooks receive only the parent pipeline and opaque parent-step events. The bridge reports terminal child-step counts through the parent step's reportProgress, so existing reporters keep one live frame.

API

The parent-scoped step factory owns the adapter:

const seedIndexStage = step.fromPipeline("seed-index", {
  pipeline: IndexSeedPipeline,
  dependsOn: [seedCatalogStage],
  description: "Seed a precomputed search index",
  mapOptions: (_inputs, context) => ({
    indexDir: context.options.indexDir,
    syncSchema: false,
  }),
  mapResult: () => ({ ran: true, stageId: "seed-index" }),
});

StepFactory.fromPipeline and StepFactory.fromPipeline.skippable infer the child result when mapResult is absent, and infer the mapped result when it is present.

For runtime-selected sets (any domain — shards, files, jobs, catalog rows):

const processShards = step.forEachPipeline("process-shards", {
  pipeline: ShardPipeline,
  dependsOn: [resolveShards],
  items: ({ "resolve-shards": shards }) => shards,
  key: (shard) => shard.id,
  concurrency: (_inputs, context) => context.options.concurrency,
  // Optional presentation only. Defaults to the noun "items".
  progress: { itemNoun: "shards" },
  mapOptions: (shard, _index, _inputs, context) => ({
    shardPath: shard.path,
    outputRoot: context.options.outputRoot,
  }),
});

When the whole fan-out may be intentionally unnecessary, opt into the widened output explicitly:

const processShards = step.forEachPipeline.skippable("process-shards", {
  pipeline: ShardPipeline,
  dependsOn: [resolveShards],
  skip: ({ "resolve-shards": shards }) =>
    shards.length === 0 ? { reason: "no shards selected", value: [] } : false,
  items: ({ "resolve-shards": shards }) => shards,
  key: (shard) => shard.id,
  mapOptions: (shard) => ({ shardPath: shard.path }),
});

Only .skippable produces readonly ChildResult[] | undefined (or readonly TOut[] | undefined with mapResult). A valued skip supplies the complete parent-facing array and bypasses items, child execution, and mapResult.

The parent sees one opaque process-shards step. Progress is domain-neutral by default: a one-line item/concurrency summary plus structured details rows for each item and its selected child steps. Interactive reporters render these as an indented tree: parent step, item key, child steps, and any deeper composition. Pending, running, completed, failed, cancelled, and skipped rows retain their states; filtered steps are omitted. Completed item groups remain in the tree. fromPipeline provides the same child-step breakdown without an item-key level. Inner reportProgress counts and details survive both composition boundaries. The progress bar advances on terminal child steps so long fan-out work does not look hung at 0%. Override progress.itemNoun or progress.formatMessage for domain labels without changing scheduling. Duplicate keys fail before any child starts. Parent dry-run overrides the mapped child run object's dryRun value.

Live fan-out snapshots show up to 32 item groups by default, prioritizing active items and then failures while keeping displayed groups in input order. Descendants stay with their item; an overflow row reports omitted groups. All groups are retained and the complete default tree is emitted once after the fan-out settles. progress.detailLimit overrides the live cap and also caps the final snapshot; larger values increase per-event presentation work. Progress payloads are not constructed when neither hooks nor tracing observe them. The standalone mappedChildProgressDetails helper continues to format only the active entries supplied in its snapshot, with no default cap.

Progress details use preorder rows with depth: 0 (or omitted) for direct children. Each composition level adds one to descendant depths. IDs are stable within their containing group; name provides optional display text. completed and total describe inner work independently of the parent's terminal-step count. Reporters cap visual indentation at 32 levels. Trace snapshots preserve these fields within 128-row, 4096-character-per-field, and 256 KiB encoded-payload bounds, with detail_count recording the pre-truncation row count. The byte budget includes UTF-8 and escaping inside the enclosing JSON event, leaving room for event metadata below the default 1 MiB NDJSON event limit.

The interactive CLI keeps completed details after their parent settles. Trees larger than the terminal use a live window around active work, with omitted-row counts; the full retained tree prints at completion. Plain/non-TTY reporting continues to emit aggregate progress messages. Presentation does not change parent plans, child selection, hook isolation, scheduling, or result types.

Semantics matrix

Guarantees

Call sites infer required and optional dependency inputs, parent domain options, required child options, the default child result, and a transformed mapResult output without explicit generic arguments. Missing required child options fail at compile time.

The parent plan contains only the opaque stage. The child receives the parent's working directory, logger, clock, sleep function, and abort signal. Parent hooks see no raw child lifecycle events. The internal bridge converts the child's canonical step statuses into monotonic progress on the opaque parent step.

Parent dry-run precedence prevents a child side-effecting step from running. An unusable child result fails one opaque parent step and names the child pipeline, step, and error, including when continueOnError produced a best-effort final value. Later cancellation interrupts in-flight child work. An invalid child plan fails before the child starts.

fromPipeline and forEachPipeline identity forms (no mapResult) use overloads so the parent-facing outputs stay the child result and child-result array, respectively. Their ordinary constructors remain non-optional; .skippable alone adds undefined.

Out of scope

Policy skip plus explicit undefined typing cover skip-without-run for opaque single and mapped child steps. Runner substitution and any richer skipped-output shape beyond policy skip are not supported.

Hierarchical or versioned external events, flattened or namespaced child selection, and parallel DAG scheduling remain out of scope. Mapped children provide bounded parallelism inside one opaque parent step; they do not make the pipeline DAG executor parallel.

Structured fan-out failures

A failed or cancelled forEachPipeline step exposes error.fanOut on the parent run error and step report (also available to failure hooks, JSON error rendering, and recorded trace history). failures contains at most the first 32 failed started items in input order, with the original index, key, keyTruncated, cancelled, and a JSON-safe error cause snapshot. failureCount counts all failed started items; omittedFailureCount counts entries beyond the limit. Scheduler failures appear separately as schedulerError; unstarted items are not failures. Setup errors, such as duplicate keys or an exception in items, have no fanOut diagnostic.

Keys and snapshot strings are capped at 1024 UTF-16 code units, and cause chains use the existing eight-level bound. keyTruncated explicitly marks shortened keys; use the original input index to recover those identities. Snapshots retain messages, names, source codes, and causes, without stacks, options, successful outputs, or nested run objects. Nested fan-outs do not recursively expand here.

Use complete keys to select original inputs for a caller-directed rerun. Check omittedFailureCount before treating the list as exhaustive, and account for unstarted work after cancellation. Reruns are ordinary new pipeline runs; the caller owns retry policy and side-effect safety. See the helper in fan-out-progress.ts.

This additive diagnostic does not change parent failure/cancellation precedence, primary causes, scheduling, downstream dependency skips, or continueOnError. Successful siblings still do not produce a partial parent step output.