Preface
In 2023, I wrote a book with a straightforward premise: by borrowing a few key ideas from software engineering, people who analyse data could save themselves a great deal of frustration. The response exceeded anything I had hoped for and confirmed a suspicion I’d had for some time: as a community, we are hungry for better ways to work.
That first book focused exclusively on the R ecosystem. One question came up again and again: “This is great, but what about Python?” It was a fair question. Data science is not a monologue; it’s a conversation between languages. Writing a Python edition seemed like the obvious next step.
I mapped out the chapters and identified the equivalent tools: pipenv for dependency management, ploomber for pipelines, and started writing. But the deeper I went, the stronger a nagging feeling became. I was solving the same problems all over again, just with a different set of tools. The rapid churn within the Python ecosystem only reinforced that feeling. How many package managers have been created to manage virtual environments? As I write this, uv is all the rage. It may well be here to stay, but history suggests that a new contender is always waiting just around the corner.
This pointed to a larger issue. I became convinced that the future of data science is polyglot. An R user and a Python user could both follow the advice from my original book and end up with reproducible projects, yet their workflows would remain fundamentally incompatible. They couldn’t easily share an environment or build a single pipeline that leveraged the strengths of both languages. Companies such as Posit have made excellent progress in making it easier to call Python from R, but building a truly integrated development environment remains challenging. And what if you also want to bring Julia into the mix? It may not be as popular as Python or R, but it has its own strengths and deserves a place in the conversation.
I realised I was treating the symptoms, not the disease.
The root problem wasn’t “How do I make R reproducible?” or “How do I make Python reproducible?”. The real challenge was the lack of a universal foundation that could handle any language, any tool, and any system dependency with absolute, bit-for-bit precision.
That’s when I stopped writing the Python book.
The solution wasn’t to create another language-specific guide but to find a tool that operated at a more fundamental level. That tool, I am now convinced, is Nix.
Nix is not just another package manager; it is a powerful, declarative system for building and managing software environments. It allows us to define the entire computational environment (from the operating system libraries up to the specific versions of our R and Python packages) in a single, simple text file. When you use Nix, the phrase “it works on my machine” becomes obsolete. It is replaced by the guarantee: “it builds identically, everywhere, every time,” with a few caveats that we will explore, of course.
Since discovering Nix, I’ve built several packages around it: {rix}, {rixpress}, and its Python counterpart, ryxpress. {rix} has been particularly successful and has received contributions from many people. To me, that validated my original intuition: people want better tools, but they often need help getting started.
Nix has a steep learning curve for many reasons, so a package like {rix}, which provides a friendlier interface for R users, lowers the barrier considerably.
{rixpress} has had a more modest reception. I suspect that’s because it asks users to fundamentally change the way they structure their work. {rix} simply generates a Nix file describing an environment. {rixpress}, by contrast, asks users to adopt reproducible pipelines. That is a much bigger behavioural shift. It draws heavily on {targets} (Landau 2021), and while {targets} is an outstanding package, pipeline-based workflows remain something of a niche. Most people still write long, sequential “spaghetti” scripts, and changing that habit takes more than good tooling.
But there is now a new opportunity which I think will truly change this: large language models (LLMs). I believe that LLMs will be a game changer and help democratize the adoption of tools like {rix}, {rixpress} and more broadly Nix. You, as the data scientist, can now 100% focus on the actual data science and leave setting up the reproducible environment and writing of the pipeline code plumbing to LLMs. These tasks are purely mechanical, low-value added (from a scientific or business perspective) and can also be easily verified: if the pipeline doesn’t run and it’s not because you made a mistake in the scientific or business logic code, then the issue comes from the pipeline code!
The idea, then, is that you shouldn’t have to write that code anymore. LLMs can write most of it for you. That said, as capable as today’s models are, they are far from perfect. They make mistakes, and I also believe there is value in giving them tools that reduce the surface area for errors in the first place.
This is why I decided to build T.
T is a new programming language that I designed in OCaml and implemented entirely with the help of LLMs. I was the architect; they wrote the code.
T is a domain-specific language, just as R is a language specialised for statistics. T is specialised for pipelines. Its only purpose is to make writing, inspecting, debugging, and running pipelines as simple as possible.
More importantly, you’re not really expected to learn T. In fact, you’re not even expected to write it. T is designed to be used both by humans who want full control and by LLMs acting on their behalf.
Everything you can do with T, like running and inspecting pipelines visually in its command prompt, can also be executed by an LLM which will get the output in structured JSON, so it’s easily actionable by the LLM. When starting a new project with T, it comes with an AGENTS.md and a language reference in markdown to help get LLMs started quickly.
T also makes it easy to coordinate R, Python and Julia, so you can use the best language for any task. You don’t have to think much about how R will communicate with Python or Julia (or vice-versa), as T handles that for you.
Finally, the last thing that makes T quite unique, is that it is built on top of Nix: this means that any T project is actually a Nix project, and that Nix is used for both setting up the reproducible environment but also for running the pipeline. This will be explained in detail in the next chapters.
This book is the result of several years of work. I first started as a user with a simple need: I should be able to re-install the packages I used for an analysis easily. So I started using {renv}, then continued through Docker and Nix, led to packages for both R and Python, and ultimately culminated in an entirely new programming language.
In the chapters that follow, you’ll learn how to build pipelines in which R, Python, and Julia are not merely neighbours but collaborators, working together in a single reproducible environment.
The core message from three years ago remains unchanged. You, as someone who writes code to analyse data, are a developer. Your work is important, and it deserves to be reliable. Using LLMs, you can now spend more of your energy on what actually matters. This book aims to give you the tools and the mindset to achieve that. The journey is more ambitious this time, but the payoff is far greater.
I hope you’ll join me.
You can read this book for free online at https://b-rodrigues.github.io/reproducible-data-science/.
You can submit issues, suggest improvements, and ask questions on the book’s GitHub repository1.
https://github.com/b-rodrigues/reproducible-data-science↩︎