6  Pipelines as Function Composition

6.1 Introduction

In the previous chapter, we set up a T project and wrote our first pipeline. You have seen the syntax: a pipeline { } block, nodes defined with node(), rn(), pyn(), and so on, a call to build_pipeline(), and then read_node() to inspect results. Now we take a step back and ask: why is T designed this way? Why mandatory pipelines? Why the restriction that each node produces exactly one artifact? Why does the order of node declarations not matter?

The answer lies in a body of ideas called functional programming. You do not need to become a functional programming theorist to use T effectively. But understanding the core concepts—purity, composition, immutability—will make you a more effective user of T, a better author of the R and Python code that lives inside your nodes, and a better collaborator with AI agents. This chapter is short by design: it makes one argument, carefully.

6.2 The Problem with State

Consider a typical data analysis script:

# Step 1
data <- read.csv("sales.csv")

# Step 2
data <- data[data$revenue > 0, ]

# Step 3
data$margin <- (data$revenue - data$cost) / data$revenue

# Step 4
model <- lm(margin ~ region, data = data)

This script works. But it has a hidden, dangerous property: state. Each step depends on the variable data existing in the global environment at exactly the right moment, with exactly the right content. The script can only be understood by reading it from top to bottom, in order, in its entirety. Rename a variable, reorder two steps, or run a subset of the code, and things break—often silently, producing wrong numbers rather than errors.

Now imagine this at scale: dozens of scripts, each hundreds of lines long, maintained by multiple people over months. Variables are overwritten, reused, and passed around in an unspoken contract of “run these files in this order.” Debugging becomes archaeology. Testing is nearly impossible because there is no clear boundary between inputs and outputs. And the entire thing is irreproducible by construction: you cannot re-execute step 4 without re-running steps 1 through 3 first, and you cannot re-run step 3 without step 2 having already modified data.

This is the root problem that functional programming addresses.

6.3 Pure Functions: The Mathematical Ideal

In mathematics, a function has a beautiful property: for a given input, it always returns the same output. The function \(f(x) = x^2\) does not depend on what you calculated before. It does not modify anything outside itself. It takes an input and returns a value, every time, guaranteed.

A pure function in programming is the same idea:

  1. It depends only on its explicit inputs. No global variables, no shared state, no hidden preconditions.
  2. It produces only its return value. No side effects: no modifying variables outside its scope, no writing files as a byproduct, no changing the world in ways the caller cannot see.

Compare these two R functions:

# Impure: secretly depends on a global variable
threshold <- 0

filter_positive <- function(data) {
  data[data$value > threshold, ]  # What is threshold? Who set it? When?
}

# Pure: all inputs are explicit
filter_above <- function(data, threshold) {
  data[data$value > threshold, ]  # Everything this function needs is right here
}

The pure version is predictable. You can read it in isolation, test it in isolation, and reason about it without understanding any surrounding context. For a given data and threshold, it will always return the same result.

6.4 T Nodes Are Pure Functions

This is not a metaphor. A T pipeline node is, by design, a pure function.

  • It takes its inputs explicitly: upstream node outputs are passed into the node’s sandbox as named values. There is no global environment to accidentally read from.
  • It returns exactly one thing: the artifact written to the Nix store by its serializer. There is no way for a node to modify another node’s output as a side effect.
  • Given the same inputs, it produces the same output—always. This is guaranteed by Nix’s hermetic sandbox. The node has no access to the outside world beyond what is explicitly declared.

Look at a concrete example:

p = pipeline {
  -- Node 1: produce data. Inputs: none. Output: a DataFrame (serialized as ^ipc).
  data_node = node(
    command = <{
      data.frame(
        x = c(1.0, 2.0, 3.0, 4.0, 5.0),
        y = c(0.0, 0.0, 1.0, 1.0, 1.0)
      )
    }>,
    runtime = R,
    serializer = ^ipc
  )

  -- Node 2: produce a model. Inputs: data_node (^ipc). Output: a model (^pmml).
  model_node = node(
    command = <{
      data_node$y <- as.factor(data_node$y)
      glm(y ~ x, data = data_node, family = binomial(link = "logit"))
    }>,
    runtime = R,
    serializer = ^pmml,
    deserializer = ^ipc
  )
}

build_pipeline(p)
T execution failed:
Error running t (error code 1): <no output>

data_node takes no inputs and returns one DataFrame. model_node takes one DataFrame (data_node, via ^ipc) and returns one model (^pmml). Neither can reach outside its sandbox. Neither modifies anything shared. Each is a pure function. And because they are pure, T can cache them: if data_node’s inputs have not changed, T does not rebuild it. Content-addressed outputs make this exact.

This also explains something that often surprises new T users: the order of node declarations inside a pipeline { } block does not matter. T infers the execution order from the data dependencies between nodes, not from the order you wrote them. This is only possible because nodes are pure: if a node had side effects, order would matter. Because it does not, T is free to determine the correct order itself, in the same way a compiler can reorder pure expressions.

6.5 A Pipeline Is Function Composition

When you write a pipeline in T, you are describing a composition of pure functions. You are saying: apply data_node, take its output, apply model_node to it. The pipeline is the composition graph.

This idea will be familiar if you have used pipes in R or Python:

mtcars |>
  filter(am == 1) |>
  select(mpg)

Each step takes the output of the previous step as its only input. The whole expression is a composition of three pure-ish functions (ignoring that they implicitly read column names). T makes this structure explicit, named, and cached at the node level.

Here is another example which does exactly this:

p = pipeline {
  -- Load: no inputs, output is a DataFrame
  mtcars = read_csv("data/mtcars.csv", separator = "|")

  -- Filter: one input (mtcars), one output (filtered DataFrame)
  filtered_mtcars = mtcars |> filter($am == 1)

  -- Select: one input (filtered_mtcars), one output (single-column DataFrame)
  mtcars_mpg = filtered_mtcars |> select($mpg)
}

build_pipeline(p)
T execution failed:
Error running t (error code 1): <no output>

Three nodes (because this is using pure T code, there is no need to use the node() constructor), three pure functions, composed in sequence. T resolves the dependency graph (mtcars_mpg depends on filtered_mtcars, which depends on mtcars), builds them in the correct order, and caches each result. If you change only the filter predicate, only filtered_mtcars and mtcars_mpg are rebuilt—mtcars is unchanged, so T does not touch it.

6.6 Polyglot Composition: Passing Values Across Languages

One of the most important consequences of treating nodes as pure functions with explicit inputs and outputs is that the runtime inside the node becomes an implementation detail. The serializer/deserializer pair is the function’s type signature: it says what the node accepts and what it returns. T uses this to move data between R, Python, Julia, and T nodes without any manual glue code:

-- Node 1: R produces a DataFrame, serialized as Arrow IPC
df_r = node(
  command = <{
    data.frame(
      id    = 1:5,
      val   = c(1.1, 2.2, 3.3, 4.4, 5.5),
      label = c("A", "B", "C", "D", "E")
    )
  }>,
  runtime = R,
  serializer = ^ipc
)

-- Node 2: Python receives the same DataFrame, doubles val, returns it
df_py = node(
  command = <{
    res = df_r.copy()
    res['val'] = res['val'] * 2
    df_py = res
  }>,
  runtime = Python,
  deserializer = ^ipc,
  serializer = ^ipc
)

p = pipeline { df_r = df_r; df_py = df_py }
build_pipeline(p)
T execution failed:
Error running t (error code 1): <no output>

df_r is a pure function: it takes nothing, returns a DataFrame. df_py is a pure function: it takes one DataFrame (^ipc), returns a modified DataFrame (^ipc). The fact that one runs in R and the other in Python is invisible to the composition. Purity is what makes this safe: because neither node has side effects, T can route the output of one into the input of the other through the Nix store without any risk of interference.

6.7 Errors as Values, Not Exceptions

One more consequence of the pure function model is worth understanding before we move to testing.

In a stateful script, errors are typically thrown as exceptions that interrupt the normal flow of execution, which is a side effect in itself. A pure function cannot throw an exception and still be pure: throwing is a side effect that transfers control to some unknown handler elsewhere.

T handles this by making errors first-class values. If a node produces an error—because a file is missing, a computation fails, or you explicitly call error()—the error is the node’s output, not an interruption of the pipeline. Downstream nodes can inspect it, recover from it, or propagate it, using the maybe-pipe ?|>:

p = pipeline {
  -- This node intentionally fails
  risky_node = node(
    command = error("DATA_MISSING", "Expected file was not found."),
    serializer = ^json
  )

  -- This node recovers: ?|> always forwards, even errors
  handled_node = node(
    command = risky_node ?|> \(input) {
      if (is_error(input)) {
        "Fallback Data"
      } else {
        input
      }
    }
  )
}

build_pipeline(p)
T execution failed:
Error running t (error code 1): <no output>

risky_node returns an error value. handled_node receives it, checks is_error(), and returns a fallback. The pipeline completes successfully. No try/catch scattered through the code, no implicit exception handling—the error path is as explicit as the success path.

This matters for reproducibility: if part of a pipeline fails, you know exactly which node failed, what it returned, and which downstream nodes were affected. The error is in the content-addressed store alongside every other output.

6.8 What This Means for Your R and Python Code

The pure function model applies not just to T nodes as a whole, but to the R and Python code you write inside those nodes.

A node is only as pure as the code it contains. If your R code inside <{ ... }> reads from a file path that is hardcoded and only exists on your laptop, or depends on a global R environment variable that is not declared as a node input, you have introduced a hidden dependency—and the node is no longer reproducible in practice, even if T tries its best to make it so.

The practical rule is simple: inside a node, treat everything from outside the node’s declared inputs as if it does not exist. Do not read files unless they are passed as explicit inputs (via fetchurl, read_csv, or another node’s output). Do not use global R options or environment variables unless they are declared in the node’s env block. Do not install packages at runtime—declare them in tproject.toml. If you follow these rules, your nodes will be genuinely pure, and T’s reproducibility guarantees will hold end-to-end.

6.9 Nodes Can Point to External Scripts

So far, we have written the R and Python code for nodes inline, inside <{ ... }> blocks. T also lets a node point to an external script file using the script argument:

p = pipeline {
  -- runtime auto-detected from .R extension
  model    = rn(script = "src/train_model.R", serializer = ^pmml)

  -- runtime auto-detected from .py extension
  preds    = pyn(script = "src/predict.py",   deserializer = ^pmml)

  -- node() auto-detects from any supported extension
  summary  = node(script = "src/summarise.R", serializer = ^json)
}

script and command are mutually exclusive: you either inline your code or point to a file. T reads the file to extract identifier references, so the dependency graph is still built correctly. The runtime is inferred from the extension (.R → R, .py → Python, .jl → Julia, .sh → Shell, .qmd → Quarto) unless you set it explicitly.

This has an important consequence. When your analysis logic lives in a plain .R or .py file, that file carries no T-specific syntax. It is a normal script. It can be:

  • Sourced directly in an interactive R or Python session
  • Picked up by another orchestrator (a Makefile, a targets plan, a Snakemake rule, a CI pipeline)
  • Run standalone by a colleague who has never heard of T
  • Tested with standard language tools independently of any pipeline

T becomes the orchestration layer on top of code that is already self-contained and portable. This is the right separation of concerns: your domain logic lives in plain language files; T handles scheduling, sandboxing, caching, and cross-language data routing.

You can push this further with the functions parameter, which sources utility files into the node sandbox before the main script runs:

p = pipeline {
  cleaned = rn(
    script    = "src/clean.R",
    functions = ["src/utils.R"],    -- sourced before clean.R runs
    serializer = ^ipc
  )

  model = rn(
    script       = "src/model.R",
    functions    = ["src/utils.R"],  -- same utility file, reused
    deserializer = ^ipc,
    serializer   = ^pmml
  )
}
T execution failed:
Error running t (error code 1): <no output>

utils.R might define clean_colnames(), log_transform(), or whatever helpers your analysis needs. Those functions are written once, tested once, and shared across any node that needs them—inside T or outside it.

The principle is the same one that runs through this whole chapter: write your analysis code as if T did not exist. Pure functions in plain files, with explicit inputs and explicit outputs. T wires them together. If T were replaced tomorrow by a different orchestrator, your functions would survive unchanged.

6.10 Summary

The key ideas in this chapter:

  • Stateful scripts make reproducibility hard because dependencies are implicit and execution order is load-bearing
  • Pure functions — same inputs, same outputs, no side effects — are the antidote: they can be tested, composed, cached, and reasoned about in isolation
  • Every T node is a pure function: explicit inputs via deserializers, exactly one output via the serializer, hermetically sandboxed by Nix
  • A T pipeline is function composition: the DAG expresses which functions compose into which, and T infers the correct execution order from the data dependencies, not from the declaration order
  • Errors are values: T propagates errors through the composition graph without breaking the pipeline, making failure as explicit and inspectable as success
  • Nodes can point to external scripts: use script = "file.R" instead of inlining code; plain R/Python files are orchestrator-agnostic and can be sourced, tested, or reused outside T entirely
  • functions shares utility code: source helper files into any node sandbox without coupling them to T’s syntax

With this model in mind, the next chapter on testing becomes natural. Testing a pure function is easy: call it with known inputs, assert on the output, done. Testing a T pipeline node is the same idea—and T provides the tooling to do it systematically.