The zx architecture

This section helps to better understand the zx concepts and logic, and will be useful for those who want to become a project contributor, make tools based on it, or create something similar from scratch.

High-level modules

Module	Description
zurk	Execution engine for spawning and managing child processes.
./src/core.ts	`$` factory, presets, utilities, high-level APIs.
./src/goods.ts	Utilities for common tasks like fs ops, glob search, fetching, etc.
./src/cli.ts	CLI interface and scripts pre-processors.
./src/deps.ts	Dependency analyzing and installation.
./src/vendor.ts	Third-party libraries.
./src/utils.ts	Generic helpers.
./src/md.ts	Markdown scripts extractor.
./src/error.ts	Error handling and formatting.
./src/global.ts	Global injectors.

Core design

`Options`

A set of options for $ and ProcessPromise configuration. defaults holds the initial library preset. Snapshot captures the current Options context and attaches isolated subparts.

`$`

A piece of template literal magic.

interface Shell<
  S = false,
  R = S extends true ? ProcessOutput : ProcessPromise,
> {
  (pieces: TemplateStringsArray, ...args: any[]): R
  <O extends Partial<Options> = Partial<Options>, R = O extends { sync: true } ? Shell<true> : Shell>(opts: O): R
  sync: {
    (pieces: TemplateStringsArray, ...args: any[]): ProcessOutput
    (opts: Partial<Omit<Options, 'sync'>>): Shell<true>
  }
}

$`cmd ${arg}`             // ProcessPromise
$(opts)`cmd ${arg}`       // ProcessPromise
$.sync`cmd ${arg}`        // ProcessOutput
$.sync(opts)`cmd ${arg}`  // ProcessOutput

The $ factory creates ProcessPromise instances and bounds with snapshot-context via Proxy and AsyncLocalStorage. The trick:

const storage = new AsyncLocalStorage<Options>()

const getStore = () => storage.getStore() || defaults

function within<R>(callback: () => R): R {
  return storage.run({ ...getStore() }, callback)
}
// Inside $ factory ...
const opts = getStore()
if (!Array.isArray(pieces)) {
  return function (this: any, ...args: any) {
    return within(() => Object.assign($, opts, pieces).apply(this, args))
  }
}

`ProcessPromise`

A promise-inherited class represents and operates a child process, provides methods for piping, killing, response formatting.

Lifecycle

Stage	Description
`initial`	Blank instance
`halted`	Awaits running
`running`	Process in action
`fulfilled`	Successfully completed
`rejected`	Failed

Gear	Description
`build()`	Produces `cmd` from template and context, applies `quote` to arguments
`run()`	Spawns the process and captures its data via `zurk` events listeners
`finalize()`	Assigns the result to the instance: analyzes status code, invokes `_resolve()`, `_reject()`

Piping

The remarkable part is pipe() and _pipe() interactions: the first provides a facade, the second binds different streams with acceptors. We use initialization inside static scope to comply with TS method visibility restrictions and to avoid extra Proxy usage:

const p = $`cmd`
const crits = await p.pipe.stderr`grep critical`
const names = await p.pipe.stdout`grep name`

Another pipe() superpower is an internal recorder. It allows binding processes at any stage w/o data loss, even if settled.

const onData = (chunk: string | Buffer) => from.write(chunk)
const fill = () => {
  for (const chunk of source) from.write(chunk)
}

ee.once(source, () => {
  fill()                    // 1. Pulling previous records
  ee.on(source, onData)     // 2. Listening for new data
}).once('end', () => {
  ee.removeListener(source, onData)
  from.end()
})

Wayback machine in action:

const p = $`cmd`
await p
await p.pipe`grep name` // Still works, but `p` is settled

`ProcessOutput`

A class that represents the output of a ProcessPromise. It provides methods to access the process’s stdout, stderr, exit code and extra methods for formatting the output and checking the process’s success.

`Fail`

Consolidates error handling functionality across the zx library: errors codes mapping, formatting, stack parsing.

CLI

zx provides CLI with embedded script preprocessor to construct an execution context (apply presets, inject global vars) and to install the required deps. Then runs the specified script.

Helper	Description
`main()`	Initializes a preset from flags, env vars and pushes the reader.
`readScript()`	Fetches, parses and transforms the specified source into a runnable form. `stdin` reader, `https` loader and `md` transformer act right here. Deps analyzer internally relies on depseek and inherits its limitations
`runScript()`	Executes the script in the target context via async `import()`, handles temp assets after.

Building

In the early stages of the project, we had some difficulties with zx packaging. We couldn’t find a suitable tool for assembly, so we made our own toolkit based on esbuild and dts-bundle-generator. This process is divided into several scripts.

Script	Description
`./scripts/build-dts.mjs`	Extracts and merges 3rd-party types, generates `dts` files.
`./scripts/build-js.mjs`	Produces hybrid bundles for each package entry point
`./scripts/build-jsr.mjs`	Builds extra assets for JSR publishing
`./scripts/build-tests.mjs`	Generates autotests to verify exports consistency

Corresponding tasks are defined in the package.json.scripts:

{
  "prebuild":     "rm -rf build",
  "build":        "npm run build:js && npm run build:dts && npm run build:tests",
  "build:js":     "node scripts/build-js.mjs --format=cjs --hybrid --entry=src/*.ts:!src/error.ts:!src/repl.ts:!src/md.ts:!src/log.ts:!src/globals-jsr.ts:!src/goods.ts && npm run build:vendor",
  "build:vendor": "node scripts/build-js.mjs --format=cjs --entry=src/vendor-*.ts --bundle=all --external='./internals.ts'",
  "build:tests":  "node scripts/build-tests.mjs",
  "build:dts":    "tsc --project tsconfig.json && rm build/repl.d.ts build/globals-jsr.d.ts && node scripts/build-dts.mjs",
  "build:dcr":    "docker build -f ./dcr/Dockerfile . -t zx",
  "build:jsr":    "node scripts/build-jsr.mjs"
}

Testing

We understand the importance, impact and risks of the tool and invest significant efforts in comprehensive research of its quality, reliability and safety. Therefore, we use an extensive set of tools and testing scenarios.

First, we inspect not how the code was written, but how it actually works. Tests mainly focus on prod bundles, pretest ensures they are actual.

{
  "pretest": "npm run build"
}

A basic set of implementation correctness checks is provided by unit tests. We also evaluate coverage to ensure that areas of code are not missed.

{
  "test:unit": "node --experimental-transform-types ./test/all.test.js",
  "test:coverage": "c8 -c .nycrc --check-coverage npm run test:unit"
}

Next, we control the contents of all the artifacts and examine their functionality.

{
  "test:npm": "node ./test/it/build-npm.test.js",
  "test:jsr": "node ./test/it/build-jsr.test.js",
  "test:dcr": "node ./test/it/build-dcr.test.js"
}

Bundle code duplication issues are highlighted through size check.

{
  "test:size": "size-limit"
}

Static analyzers are responsible for code quality.

{
  "fmt:check": "prettier --check .",
  "test:circular": "madge --circular src/*"
}

Type checking examines declarations and compatibility with different loaders and transpilers.

{
  "test:types": "tsd",
  "test:smoke:strip-types": "node --experimental-strip-types test/smoke/ts.test.ts",
  "test:smoke:tsx": "tsx test/smoke/ts.test.ts",
  "test:smoke:tsc": "cd test/smoke && mkdir -p node_modules && ln -s ../../../  ./node_modules/zx; ../../node_modules/typescript/bin/tsc -v && ../../node_modules/typescript/bin/tsc --esModuleInterop --module node16 --rootDir . --outdir ./temp ts.test.ts && node ./temp/ts.test.js",
  "test:smoke:ts-node": "cd test/smoke && node --loader ts-node/esm ts.test.ts"
}

We also check compatibility with all the target runtimes x OS variants.

{
  "test:smoke:bun": "bun test ./test/smoke/bun.test.js && bun ./test/smoke/node.test.mjs",
  "test:smoke:win32": "node ./test/smoke/win32.test.js",
  "test:smoke:deno": "deno test ./test/smoke/deno.test.js --allow-read --allow-sys --allow-env --allow-run",
}

CJS and EMS exports are verified separately.

{
  "test:smoke:cjs": "node ./test/smoke/node.test.cjs",
  "test:smoke:mjs": "node ./test/smoke/node.test.mjs"
}

Finally, we check the license and supply chain security issues.

{
  "test:license": "node ./test/extra.test.js",
  "test:audit": "npm audit fix",
  "test:workflow": "zizmor .github/workflows -v -p --min-severity=medium"
}