15  Beyond the Basics: T in Depth

The earlier chapters took you from a blank Nix installation to a fully automated, reproducible pipeline running on GitHub Actions. Along the way you learned the core loop: declare nodes, build, inspect, test, and ship. But T is a larger language than that loop requires, and this chapter is where we go deeper.

The material here now focuses on more of T’s features. We cover ten topics, in the order you are likely to need them. Each section is self-contained, but it builds on the foundations from the earlier chapters, and we point back to them rather than repeating what they already teach.

15.1 The Interactive T: Magic Commands

Chapter 5 introduced the T REPL as a place to explore data and prototype node logic before committing it to a pipeline. The REPL has a small set of conveniences that make interactive work faster: the magic commands.

Magic commands are REPL-only shortcuts prefixed with %. They give you quick access to common operations without parentheses or string quotes. They are not part of the language proper (they only work at the REPL prompt) so they will never appear in a pipeline.t file.

15.1.1 Navigating your environment

The first group of commands helps you get your bearings. %pwd prints the current working directory, and %cd changes it (it expands ~ correctly, and reports an error if the target directory does not exist). %ls lists the contents of the current directory, and %env prints every environment variable, one per line.

%pwd
/home/user/projects
%cd data
%ls
raw.csv  clean.csv

These mirror the shell commands you already know, so the REPL feels less like a foreign object and more like a workspace you can move around in.

15.1.2 Inspecting what you have

When you have been working in the REPL for a while, it is easy to lose track of what is in scope. %objects (aliased %who) lists every user-defined variable, its type, and a compact summary:

x = 42
df = read_csv("data.csv")
%objects
--  name  type       summary
--  x     Int        42
--  df    DataFrame  150 rows x 5 cols

%history shows the entries from the REPL command history, stored in ~/.t_history, which is handy for re-running or copying something you did a few minutes ago. %reset removes all user-defined variables and returns you to a clean base environment.

15.1.3 Timing and capturing work

%time evaluates any T expression and prints both its result and how long it took. It is the quickest way to get a feel for whether a piece of logic is fast enough to keep:

%time df |> filter($x > 10) |> nrow()
42
Execution time: 0.1540 seconds

%save writes a transcript of the session, i.e.  every command and its result to a file, so you can turn an exploratory session into a starting point for a script. In compact mode it records a one-line summary of each result; in verbose mode it records the full pretty-printed output:

%save my_session.t
%save verbose my_session.t

A common workflow is to run %reset to clear the transcript, then do a clean run and %save it, giving you a tidy record of exactly what you tried. Typing %magic lists every magic command with a description, and if you mistype one, the REPL offers a fuzzy match: %objcts suggests Did you mean: %objects?.

15.2 Functions in Depth

You have already written functions in T: Chapter 6’s functional core gave you pure functions and composition, which is enough to build a pipeline. But T’s functions go further: they are first-class values you can build, transform, and pass around. This section covers the parts you will reach for as your pipelines grow.

15.2.1 Lambda syntax

T defines functions with a lambda. The preferred style is the R-like \(...) form, though a function keyword is also accepted:

square = \(x) x * x
square = function(x) x * x    -- equivalent

Multi-argument functions are just as natural:

add = \(a, b) a + b
add(3, 7)  -- 10

15.2.2 Closures

A function captures its enclosing environment, which lets you build parameterised families of functions:

make_adder = \(n) \(x) x + n
add5 = make_adder(5)
add5(10)  -- 15

15.2.3 Higher-order functions

Because functions are values, you can pass them to other functions. The standard library is full of them:

numbers = [1, 2, 3, 4, 5]
map(numbers, \(x) x * x)     -- [1, 4, 9, 16, 25]
filter(numbers, \(x) x > 3)  -- [4, 5]

15.2.4 Auto-quotation ($param)

A particularly useful feature when wrapping data verbs: prefix a parameter with $ and the caller can pass a bare name, a column name say, which is captured as a Symbol rather than evaluated. This is the same mechanism the data verbs themselves use (see the next section):

my_select = \(df, $col) select(df, col)
my_select(df, salary)    -- 'salary' is captured as a symbol

15.3 The Type System

Chapter 4 introduced annotations in passing: read them as a promise about what a function accepts and returns. Here is what is actually going on.

15.3.1 Lambda signatures

A lambda is either untyped or typed. The typed form annotates every parameter and the return type. The return type is written inside the parameter parentheses, which keeps the whole signature on one line:

add     = \(x, y) x + y                          -- untyped
add_int = \(x: Int, y: Int -> Int) (x + y)       -- typed

Generic lambdas declare their type variables explicitly with <...>:

id   = \<T>(x: T -> T) x
pair = \<A, B>(x: A, y: B -> Tuple[A, B]) [x, y]

The annotation grammar covers the base types (Int, Float, Bool, String, NA) and the composite forms List[T], Dict[K, V], Tuple[T1, T2, ...], and DataFrame[schema], where the schema names the columns the DataFrame is promised to carry.

15.3.2 Two modes: repl and strict

The same code can be checked differently depending on how you run it:

• t repl runs in repl mode, which is permissive: untyped top-level functions are fine, which is what you want while exploring.
• t run <file.t> runs in strict mode by default. For every top-level function assignment, strict mode requires all parameter types to be annotated, a return type to be present, and any type variables to be declared. Miss one and the script fails before it runs:

add = \(x, y) x + y

Error(TypeError): [add.t:L1:C7] Strict mode: top-level function 'add'
must annotate all parameter types.

--mode repl|strict overrides the default in either direction, so a script can opt into leniency and a REPL session into rigor.

15.3.3 How far it goes today

Be clear-eyed about what strict mode is: a signature validation layer, not yet a full static typechecker. It checks that top-level functions carry complete, well-formed signatures; it does not (yet) infer types across expressions, check function applications end to end, or enforce DataFrame[schema] column by column. That is planned, and the practical advice follows from it: put explicit signatures on your top-level and exported functions now, so the code is ready when the checker grows into the promises you wrote.

15.3.4 Semantic type names

Annotations accept several spellings for the same type, so signatures can read naturally: int/integer, string/text, bool/boolean/logical, float/double/number/numeric, and to_dataframe/table. The name any (synonyms value, all, mixed) is a wildcard meaning “no constraint.”

The most concrete payoff today is schema checking. t check --schema propagates column names and types across nodes and flags a reference that cannot be resolved, in milliseconds, before the pipeline ever runs. Chapter 5 covers the full tiered t check story; this is the tier that turns a DataFrame schema annotation into an actual guarantee.

15.4 Metaprogramming: Quotation and Quasiquotation

Chapter 6 introduced T’s functional core: pure functions and function composition. Most of the time you can stay in that world and never think about how expressions are evaluated. But sometimes you need to treat code itself as data: to build expressions programmatically, to forward a column name that is only known at runtime, or to write a function that behaves like the data verbs such as mutate and filter. That is what metaprogramming is for.

T’s metaprogramming is modelled on Lisp and on R’s rlang, and it rests on a small set of ideas:

• Quotation captures an expression without evaluating it.
• A quosure is a quoted expression paired with the environment in which it was written.
• Unquoting injects an already-evaluated value back into a quoted expression.
• Splicing expands a collection into the arguments of a call.

15.4.1 Capturing code: to_expr and quo

The two ways to capture code differ in whether they remember the surrounding environment. to_expr() captures a bare expression; quo() captures a quosure: the expression plus the environment at the call site.

x = 10
q = quo(1 + x)   -- captures x = 10
x = 99
eval(q)          -- 11, not 100: it runs in the captured environment

11

That distinction is the whole point of a quosure. When eval() runs a quosure, it evaluates it in the environment the quosure captured, not in whatever environment happens to be current. For a bare expression from to_expr(), eval() uses the current environment.

There are plural forms too: to_exprs(...) and quos(...) capture several expressions at once, returning a list of bare expressions or quosures respectively.

15.4.2 Unquoting: !! and !!!

Quasiquotation is how you fill in the blanks of a captured expression. The !! operator evaluates its operand and injects the result into the surrounding quoted expression. If the operand is a quosure, only the expression part is injected; the environment is stripped.

x = 10
e = to_expr(1 + !!x)
print(e)         -- to_expr(1 + 10)

to_expr(1 + 10) 

The !!! operator goes further: it evaluates its operand and splices the elements into the surrounding call. The operand must be a list, vector, or dict.

vals = [1, 2, 3]
e = to_expr(sum(!!!vals))
print(e)         -- to_expr(sum(1, 2, 3))

to_expr(1 + 10) 
to_expr(1 + 10) 

If you splice a named list, the names become argument names:

my_args = [x: 10, y: 20]
e = to_expr(f(!!!my_args, z: 30))
print(e)         -- to_expr(f(x = 10, y = 20, z = 30))

to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 

15.4.3 Dynamic names and symbols

A frequent need is to use a name that is only known at runtime, a column name stored in a string, say. to_symbol() turns a string into a symbol that !! can inject, and the !!name := value form lets a computed name become an argument or column name.

col = "age"
e = to_expr(mutate(df, !!col := 42))
print(e)         -- to_expr(mutate(df, age = 42))

to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 

get() is the companion for the reverse direction: it retrieves a variable’s value from a string or symbol name, which is how you look up a column dynamically.

15.4.4 Non-standard evaluation

The most common reason to reach for all of this is to write a function that accepts an unevaluated column name, the way the data verbs do. T gives you three tools for it, in order of increasing power:

1. Auto-quoted parameters. Prefix a parameter with $ and the caller can pass a bare column name, which you forward into an NSE-aware verb with !!.

my_mean = \(df, $col) {
  summarize(df, result = mean(!!col))
}

2. enquo(param) captures the caller’s full expression for a named parameter, as a quosure. Use it when you need more than a column name.

3. enquos(...) captures all variadic expressions as a list of quosures, which you can splice into a verb.

my_summarize = \(df: DataFrame, ... -> DataFrame) {
  cols = enquos(...)
  eval(to_expr(df |> summarize(!!!cols)))
}

to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 

The default advice is simple: use quo over to_expr when in doubt, so your code remembers its environment; use the $param form for the common “accept a column” case; and reach for enquo/enquos only when you genuinely need the caller’s expression.

One detail is worth knowing because it prevents a whole class of confusing bugs. Inside a data verb, expressions are resolved against a data mask: T looks up $column in the mask first, so a global variable or function with the same name as a column does not interfere. That is why df |> mutate(new = $score * 2) works even when a function called score exists in scope.

15.5 Data Access and Lenses

T provides a unified way to retrieve data from variables, collections, dicts, and pipelines through the get() primitive, and a Lens library for addressing specific parts of complex structures. Lenses matter because they are first-class, serializable values: you can pass them between pipeline nodes and apply them without rebuilding.

15.5.1 The get() built-in

get() is the primary entry point for retrieval. It adapts to what you give it:

salary = 50000
get("salary")              -- variable lookup, R-style      50000

lst = [10, 20, 30]
get(lst, 1)                -- 0-based collection indexing   20

d = [a: 1, b: 2]
get(d, "a")                -- dict key lookup               1
get(d, "missing", 99)      -- default when key is absent    99

Inside an NSE data verb (mutate, filter, summarize, …), get() checks the data mask first: a name that is a column in the current row or DataFrame is resolved there, and only if not found does it fall back to the global environment. That is what makes get("a") inside a mutate refer to the column a, not a global variable.

15.5.2 Lenses

Where get() takes an ad-hoc index, a lens is a structured, reusable “address” you can compose and pass around. The built-in lens creators are:

Creator	Target	Description
col_lens(name)	DataFrame, Dict, List	Targets a column or key by name.
idx_lens(i)	List, Vector	Targets an element by index.
row_lens(i)	DataFrame	Targets a specific row as a dictionary.
node_lens(name)	Pipeline	Targets a specific node within a pipeline.
env_var_lens(n, v)	Pipeline	Targets an environment variable v from node n.

You focus a lens with get() and compose lenses with compose() to reach deep into nested structures:

l = col_lens("mpg")
get(mtcars, l)            -- the 'mpg' column as a Vector

data = [users: [Alice: [id: 1], Bob: [id: 2]]]
l_deep = compose(col_lens("users"), col_lens("Alice"), col_lens("id"))
get(data, l_deep)         -- 1

To update data through a lens, use over():

data_updated = over(data, l_deep, \(x) 99)
get(data_updated, l_deep) -- 99

Pipeline lenses are serializable: you can pass them between nodes and apply them with mutate_node() or set_pipeline_global_options(), which is how you make cross-node data operations part of the reproducible build.

15.6 Introspection: explain() and intent

A language that can describe its own values is easier to debug, easier to test, and, increasingly, easier to work on alongside an agent. T has both halves of that story: explain(), which turns any value into a structured description, and intent, a block that records why a piece of code exists.

15.6.1 explain(): a value’s self-description

explain(x) works on any value and returns a Dict describing it. The first field is always kind, and the rest depend on what you handed it:

explain(42)
-- { `kind`: "value", `type`: "Int", `value`: 42 }

explain([1, 2, 3])
-- { `kind`: "value", `type`: "List", `length`: 3, `na_count`: 0, `examples`: [1, 2, 3] }

For a DataFrame, the description is a compact summary (row and column counts, the storage backend, the schema, and per-column missingness), and you can drill into individual fields with dot access:

df = to_dataframe([x: [1, 2, 3], y: ["a", "b", "c"]])
explain(df).nrow
-- 3
explain(df).schema
-- [{`name`: "x", `type`: "Int"}, {`name`: "y", `type`: "String"}]
explain(df).na_stats
-- { `x`: 0, `y`: 0 }

Errors are values too, and explain sees through them, surfacing the code, the message, and where it happened:

explain(1 / 0)
-- { `kind`: "value", `type`: "Error", `error_code`: "DivisionByZero",
--   `error_message`: "Division by zero.", ... }

The same function reaches the rest of the language. explain(y ~ x) reports a formula’s response and predictors; explain(add_int) reports a function’s argument names and types; explain(p) reports a pipeline’s node count and a per-node summary, while explain(p.b) inspects a single node: its name, runtime, dependencies, and configuration. And explain_json(x) returns the same description as a single string, which is handy when you want to log a value’s shape or hand it to another tool.

The point is that introspection is first-class: the same call that describes a scalar also describes a DataFrame, an error, a formula, and a whole pipeline. You do not need a separate debugging API per type.

15.6.2 intent: code says what, intent says why

T has a small, dedicated syntax for recording why a piece of code exists: a block of key-value pairs that travels with the code:

intent {
  description: "Analyze customer churn"
  goal: "Identify segments at risk of leaving"
}

An intent block is inert at runtime (it does not change what the code does), but you can read it back programmatically:

i = intent { description: "Analyze customer churn", goal: "Find at-risk segments" }
intent_get(i, "description")
-- "Analyze customer churn"
intent_fields(i)
-- { `description`: "Analyze customer churn", `goal`: "Find at-risk segments" }

The payoff is provenance. When a pipeline is a collaboration between a human and an agent, or between you today and you in six months, the why is the first thing to be lost. intent keeps it in the source, versioned in git and queryable by the tools that read it. It is the line-level companion to the AGENTS.md convention from Chapter 4: that file states the project’s intent, and an intent block states the intent of the code right next to it.

15.7 Escaping to the Shell: ?<{ }>

Chapter 8 introduced shn(), the node constructor that runs a shell command inside a pipeline’s sandbox. That is the right tool when the shell work is a first-class step in your pipeline. But there is a lighter way to reach the shell from T itself: the shell escape. The ?<{ }> syntax runs an arbitrary shell command directly from within T, at the REPL, in a script, or in a node’s T code, and is the quickest way to touch the filesystem, drive git, or call a CLI tool without standing up a whole shell node.

15.7.1 Running a command

As a standalone statement, the command is executed and its output is printed straight to stdout. The statement’s value is NA:

?<{ls -la}>
?<{git status}>

15.7.2 Capturing the output

When the escape is part of an expression, most often an assignment, T captures the command’s stdout and hands it back as a String, ready to be split, parsed, or passed on:

files = ?<{ls}>
current_user = ?<{whoami}>

15.7.3 Changing directory

cd is special-cased. Rather than changing the directory of a throwaway sub-shell, it changes the working directory of the T interpreter itself, so subsequent commands, and ?<{pwd}>, see the new location. ~ is expanded correctly:

?<{cd /tmp}>
?<{pwd}>       -- /tmp
?<{cd ~}>

15.7.4 Multiline commands

A shell escape can span several lines. The indentation common to all non-empty lines is stripped, so you can write a readable block of shell without backslash continuations:

?<{
  git add .
  git commit -m "update"
  git push
}>

15.7.5 Errors

If a command exits with a non-zero status, the escape yields a ShellError carrying the command’s stderr, so a failing command is easy to diagnose. You can test for it with is_error():

result = ?<{ls /nonexistent}>
is_error(result)   -- true

15.8 Serializers in Depth

Chapter 5 showed you that pipeline nodes pass data to one another through serializers, and that you can name a built-in one with the ^ prefix. This section goes deeper: what the built-ins are, how to choose between them, and how to write your own.

15.8.1 The built-in serializers

Every node’s output is written to disk in some format, and every node reads its inputs in the corresponding format. T ships a set of built-in serializers, identified by ^ symbols:

Identifier	Format	Best for
^ipc	Apache Arrow IPC	Fast live hand-off between nodes
^parquet	Apache Parquet	Durable, compressed storage
^csv	CSV	Simple tabular interchange
^json	JSON	Config, lists, dicts
^pmml	PMML	Predictive models
^onnx	ONNX	ML model interchange
^text	Plain text	Logs, shell output
^bin	Binary	Opaque blobs (default for fetchurl)

Most of these are symmetric across the T, R, Python, and Julia runtimes, which is what makes polyglot pipelines possible: a DataFrame written by an R node can be read by a Python node because both sides understand the same format.

The choice that comes up most often is between ^ipc and ^parquet. Both are columnar, type-preserving Arrow formats that work in every runtime. The difference is live hand-off versus durable artifact. ^ipc writes the in-memory Arrow layout straight to disk: the fastest possible round trip, but uncompressed. ^parquet is a compressed, storage-optimized layout: smaller files (often several times smaller for numeric data), column pruning on read, and first-class support in Spark, DuckDB, and pandas. The rule of thumb is to pass data between nodes while a pipeline runs with ^ipc, and to persist, ship, or store the final result with ^parquet. You can do both in one pipeline.

15.8.2 Symbols, variables, and the string trap

There is a subtle but important distinction in how you name a serializer. Built-in serializers are symbols with the ^ prefix. A custom serializer you have defined is a variable, and you pass its name with no ^.

node(command = read_csv("large.csv"), serializer = ^ipc) -- built-in
import "src/my_ser.t" [my_ser]
node(command = ..., serializer = my_ser)                 -- custom

to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 
node<T>(...)

And here is the trap: in a node constructor (node, rn, pyn, jln, shn, qn), a string literal is not allowed. Writing serializer = "ipc" raises a TypeError. You must use a ^ symbol for built-ins or a variable for custom serializers. (The mutate_node() and set_pipeline_global_options() functions are more permissive and accept both strings and symbols.)

If you do not specify a serializer at all, T uses the default, which is each runtime’s native binary format: saveRDS for R, pickle for Python, and so on. Shell nodes default to ^text.

15.8.3 Custom serializers

A serializer is a first-class value: a record with a format, a writer, and a reader.

type serializer = {
  format: string,
  writer: function(path: string, value: any) -> result[NA, string],
  reader: function(path: string) -> result[any, string]
}

To define one, you write a record that matches that shape. The format field should be a ^ symbol so it stays consistent with T’s symbol-based serialization.

my_log_serializer = {
  format: ^log,
  writer: \(path, val) {
    -- write val to path in your log format
    Ok(NA)
  },
  reader: \(path) {
    -- read the log back from path
    Ok("log content")
  }
}

node(command = ..., serializer = my_log_serializer)

For a serializer to work across other runtimes, you can add optional r_writer, r_reader, py_writer, and py_reader fields. These are code snippets, plain strings or foreign code blocks for readability, that T injects into the generated build script for that runtime.

my_custom_ser = [
  format: ^custom,
  writer: \(path, val) { Ok(NA) },
  reader: \(path) { Ok(42) },
  r_writer: <{ function(obj, path) { writeCustom(obj, path) } },
  r_reader: <{ function(path) { readCustom(path) } },
  py_writer: <{ def write_custom(obj, path): ... },
  py_reader: <{ def read_custom(path): ... }
]

T performs static coherence checks when you build a pipeline that uses a custom serializer: if a node is declared for the R runtime but the serializer has no r_writer or r_reader, you get an error at build time rather than a mystery failure at run time. And when T detects that your serializer references R or Python functions, it can add the corresponding [r-dependencies] or [py-dependencies] to the generated build automatically (controlled by TLANG_AUTO_ADD_PIPELINE_DEPS).

15.9 Plotting in Pipelines

Chapter 8 showed you how to structure a pipeline as data. This section covers what happens when a node’s output is not a DataFrame but a plot.

15.9.1 Returning a plot from a node

Any node can return a plot object; a Plot from CairoMakie, a ggplot2 object, a matplotlib figure, and so on. When it does, T serializes the plot and, crucially, records visualization metadata alongside the artifact. That metadata is what lets downstream tools know the output is something to be shown, not just read.

Let’s study this pipeline:

p = pipeline {
  iris_data = rn(
    command = <{
      # Load iris dataset
      iris <- datasets::iris
      
      # Return the data
      iris
    }>,
    serializer = ^csv
  )
  iris_ggplot = rn(
    command = <{
      # Load ggplot2
      library(ggplot2)
      
      ggplot(iris_data, aes(x = Petal.Length, y = Sepal.Length)) +
        geom_point() +
        facet_wrap(~ Species) +
        labs(title = "Sepal.Length ~ Petal.Length | Species")

    }>,
    deserializer = ^csv
  )
}

to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 
to_expr(1 + 10) 

15.9.2 Reading a plot back

Once built, it is possible to read the plot using read_node(), which on a plotting node returns not the raw artifact but a small metadata record describing it: the path to the serialized plot, its type, and the runtime that produced it.

read_node(p.iris_ggplot)

returns:

ggplot
├── backend: "R"
├── title: "Sepal.Length ~ Petal.Length | Species"
├── mapping
│   ├── x: "Petal.Length"
│   └── y: "Sepal.Length"
├── labels
│   └── title: "Sepal.Length ~ Petal.Length | Species"
└── layers
    └── geom_point: "Point"

and to actually see the plot in your browser, use show_plot():

T> show_plot(p.iris_plot)

This should open the plot in your browser, if not, the path is emitted and you can simply open the generated PNG file using whichever tool you prefer:

_pipeline/show_plot_render_20260821_132239_6808.png

This plot is generated in the actual R environment defined in your pipeline, but should something look off, try debug_node(p.iris_plot) and build it from the actual R session!

15.9.3 Plots in literate documents

The most common place this matters is a Quarto document. T’s Quarto integration has a dual behaviour that is worth understanding. A {t} chunk that produces a plot records the visualization metadata, and a companion {r} or {python} chunk can then call show_plot() to render it. This lets you compute the plot in the runtime that is best suited to it and display it in the document without copying data between runtimes.

The mechanism depends on a few underlying libraries: CairoMakie for T plots, kaleido for exporting ggplot2 objects to images, and cloudpickle for serializing Python plot objects that the standard pickle module cannot handle. If a plot does not render the way you expect in a document, the first thing to check is whether the runtime that produced it has the right export library available in its Nix environment.

15.10 Nix Build Options and Orchestration

Chapter 2 introduced Nix as the foundation that makes T reproducible. Chapter 5 showed you build_pipeline() and t run at the command line. This section covers the knobs in between: the nix_options dictionary that controls how a pipeline is actually built, and the ways to orchestrate that build.

15.10.1 The nix_options dictionary

The build functions, populate_pipeline(), build_pipeline(), pipeline_run(), and the t_make() helper, all accept a nix_options argument. It is a dictionary of build-time settings that override the defaults. The most useful keys are:

• max_jobs and max_cores: how much of the machine a build may use.
• dry_run: evaluate the build plan without running any node.
• force: rebuild nodes even if their outputs already exist.
• targets: build only a subset of the pipeline’s nodes.
• cache: which Nix cache to use for the build.
• builders: which machines to build on (local or remote).
• keep_env and sandbox: how much of the host environment a node sees.
• env_vars: per-node environment variables, set when the node is defined.

Validation is strict: pass a key that is not a valid option, or a value of the wrong type, and you get a TypeError or ValueError before the build starts.

build_pipeline(p, nix_options = { max_jobs = 4, dry_run = true })

15.10.2 Dry runs and plans

Setting dry_run = true is one of the most useful habits to form. Instead of executing the pipeline, the build returns a plan as a DataFrame: every node that would run, in dependency order, with its inputs and outputs. You can inspect it, filter it, or assert on it in a test. It is the cheapest way to check that your pipeline is wired the way you think it is.

15.10.3 Where environment comes from

It is worth separating two related ideas. keep_env controls whether a node sees the host’s full environment or a clean one; this is a build-time isolation choice. env_vars, by contrast, is how you declare specific variables a node needs, and it is set when the node is defined, not at build time. The first is about isolation; the second is about configuration.

15.10.4 Building on other machines

The builders option is what turns a local build into a distributed one. You can point a build at a remote Nix builder, a faster machine or a machine with the right architecture, and Nix will ship the build steps there. This is particularly useful when your pipeline has heavy nodes that would take minutes locally but seconds on a larger box. The pipeline_to_drv(p) function returns the underlying Nix derivation, which is the escape hatch for when you need to hand the build to a tool that understands derivations directly.

The cache option is the companion to builders. Where builders decides where a build runs, cache decides where pre-built artifacts come from. Like builders, it is a key in the nix_options dictionary. Passing a Cachix cache name registers it as an extra Nix binary substituter:

build_pipeline(p, nix_options = { cache = "my-team" })

Nix then pulls any matching store paths from my-team.cachix.org before falling back to building them, and can push freshly built artifacts back to the cache. This is how a team shares build results across machines and CI runners without each one recompiling the same R and Python packages from source.

15.11 Building T Packages

Chapter 12 covered packaging T pipelines for distribution as R and Python packages. But T can also be packaged in T. This section is about that: turning a collection of T functions into a reusable package that others can import.

15.11.1 Initialising a package

t init --package scaffolds a T package. The result is a directory with a DESCRIPTION.toml that declares the package’s metadata, its dependencies, and its build inputs, and a flake.nix that makes the package itself a Nix input.

[package]
name = "mylib"
version = "0.1.0"

[dependencies]
otherlib = { git = "https://github.com/me/otherlib", tag = "0.3.0" }

[additional-tools]
# extra tools the package needs at build time

Dependencies are declared by git URL and tag, which keeps them reproducible in the same way the rest of T is.

15.11.2 Public and private API

By default every top-level function in a package is part of its public API. Mark a function @private to exclude it from the documented surface. This matters because the public API is what t doctor checks for and what the generated documentation advertises.

15.11.3 Testing a package

t test runs the package’s tests, and it honours the same flags as the pipeline tests from Chapter 7: you can select a subset, run in verbose mode, and so on. A test marked noop is skipped, useful for a test that depends on an optional dependency that may not be present.

15.11.4 T-Doc documentation

T packages use a lightweight documentation format, T-Doc. A function is documented with a --# block that supports directives:

--# @param x The input value.
--# @return A transformed value.
--# @example
--#   myfunc(1)
--# @seealso myotherfunc
--# @family transforms
myfunc = \(x) { x * 2 }

t doc --parse --generate parses these blocks and generates the package’s documentation. t doctor is the linter: it checks that public functions are documented, that @param names match the signature, and that cross-references resolve.

15.11.5 Publishing

t publish publishes the package by tagging the git repository. There is no central T package registry; distribution is by git, which is deliberate: it means a package version is pinned to an exact commit, and consumers depend on it the same way they depend on any other Nix input.

15.11.6 Importing

Consumers import a T package the same way they import any T module, by name or by path, with optional selective or aliased imports:

import "mylib"
import "mylib" [myfunc]
import "mylib" [myfunc as transform]

15.12 Property-Based Testing with Propcraft

Chapter 7 introduced unit testing with Testcraft: you write concrete examples and assert on their results. That is the right tool for most functions, but it has a blind spot. A function can pass every example you thought of and still be wrong for the input you did not think of. Property-based testing is how you close that gap, and in T it is provided by Propcraft.

15.12.1 The idea

Instead of testing specific inputs, you describe a property that should hold for all inputs in some domain, and Propcraft generates hundreds of random inputs to try to break it. If it finds one that fails, it shrinks the failing input down to a minimal, easy-to-read counterexample.

The entry point is prop_for_all, which takes a generator for each argument and a property function:

prop_for_all(
  prop_gen_int_range(0, 100),
  prop_gen_int_range(0, 100),
  \(a, b) { add(a, b) == add(b, a) }
)

15.12.2 Generators

A generator describes a domain of values. Propcraft ships generators for the common types:

• prop_gen_int, prop_gen_int_range(lo, hi)
• prop_gen_float_range(lo, hi)
• prop_gen_bool
• prop_gen_string_from(alphabet)
• prop_gen_factor(levels)
• prop_gen_date_range(lo, hi)
• prop_gen_df(...) and prop_gen_df_from(template, na_prob = 0.1)

The DataFrame generators are the ones you will use most. prop_gen_df builds a random DataFrame with the column types you specify, and na_prob controls how many missing values to sprinkle in, which is exactly the kind of input that exposes bugs in real pipelines.

You can also build generators out of other generators with combinators: prop_map_gen transforms a generator’s output, prop_such_that filters it to values satisfying a predicate, and prop_resize scales the size of a collection-valued generator.

15.12.3 The property contract

There is one rule that catches people out: a property that returns NA fails. A property must return a boolean, and NA is not a true. This is deliberate: a property that cannot decide is not a valid property. The Expect type is understood by Propcraft, so you can write properties in terms of the same assertions you use elsewhere. A property that raises an Error also fails.

15.12.4 A concrete example

Here is the kind of bug property-based testing finds. Suppose a function is supposed to drop rows with missing values in a given column:

prop_for_all(
  prop_gen_df(na_prob = 0.3),
  \(df) { nrow(drop_na(df, $x)) <= nrow(df) }
)

The property, that the result has no more rows than the input, is obviously true, yet it fails. Propcraft shrinks the failing input to a tiny DataFrame and shows you the exact rows that broke it, instead of leaving you to guess.

15.12.5 Seeding and sample size

Generated inputs are random, so tests can be flaky unless you control the randomness. set_seed(n) and with_seed(n, ...) make a property test deterministic for a given seed. The n argument of prop_for_all sets how many cases to try; it defaults to 100.

15.12.6 Running with t test

Propcraft properties are just tests, so t test runs them alongside your Testcraft unit tests. A failing property shows the shrunk counterexample and the seed that reproduces it, so you can pin the seed and turn the counterexample into a permanent regression test.

One caveat: property-based testing is a package-hardening tool. It shines when you are writing a library that others will call with arbitrary inputs. For a one-off analysis pipeline, the concrete examples from Chapter 7 are usually enough, and the cost of generating and shrinking random DataFrames is not worth it.

15.13 Where to go from here

You now have the full toolkit: the REPL conveniences, the metaprogramming primitives, the shell escape, the serializer and plotting machinery, the build orchestration knobs, the package toolchain, and property-based testing. None of these are required for a simple pipeline, but together they are what let a T project grow from a script into a library, a service, or a platform without changing its foundations. The next chapter pulls the threads together.