Deterministic guardrails and documentation that becomes obsolete: what I took away from Santander’s open source project

_{Estimated reading time: 5 minutes}

Santander has released its AI work

A few weeks ago, Banco Santander published part of its open-source AI work, licensed under Apache 2.0, on GitHub. What’s interesting—and Stefan Bergsten explains this very well in his article “Santander’s Open-Source AI Move: The Control Loop Banks and Portfolio Managers Need to Understand”—is not that it’s sharing “banking intelligence,” but rather that it’s making the infrastructure for governing AI available: guardrails, evaluation harnesses, vendor-agnostic access to models, and synthetic data. The key takeaway for me is that, in regulated AI, the model is an input; the control process surrounding it is the true asset.

The Most Valuable Aspect of Open Source

While reviewing the repository, I found two projects that, over the course of a summer weekend afternoon, improved systems we already had up and running. It’s the pattern that travels, not just the code.

Project 1 — mech-gov-framework: From a Rule in the Prompt to a Deterministic Guarantee

One of our internal agents is a chatbot that uses your data in BigQuery: a two-step pipeline (one agent generates SQL, the other crafts the response in natural language), deployed on Cloud Run behind IAP.

What did we already have, aside from the prompt:

• Read-only identity. The service account for this service has only the roles bigquery.dataViewer and bigquery.jobUser. Even if the model were to generate a DELETE or a DROP, BigQuery would reject it: “read-only” access was guaranteed at the IAM level, not at the prompt level.
• Robust execution. A structural check that rules out truncated SQL (unbalanced parentheses, dangling clauses) and a retry loop that, in the event of a BigQuery error, feeds the error back to the model so it can correct it.
• Endpoint authentication via IAP.

Where was the real loophole? In the separation between clients and in the cost. The SA could read the entire warehouse (dataViewer at the project level); the only thing preventing one client’s agent from accessing another client’s table was the prompt asking for permission. And there was no limit on the number of billed bytes.

Since it’s a project for 100% internal use with limited access, it’s no big deal, but there’s always room for improvement. And when I saw the mech-gov project in the Santander repo, I thought to myself, “This is going to be useful to me.”

The mech-gov pattern: deterministic gates that surround the model and cannot be bypassed, both before and after generation. The idea that emerges: a prompt is a request, not a guarantee.

I’ve implemented it as a deterministic validator at runtime (a hook that runs before the query is executed). First-match-wins rules: a single read-only statement (no DDL/DML or chained statements), fully qualified tables, and, the icing on the cake, a strict per-client allow list: each agent can only access its own mart. Plus non-blocking warnings (SELECT *, division without SAFE_DIVIDE, absence of a date filter) and a hard cap on `maximum_bytes_billed`. If a query fails the filter, the hook throws an exception that reuses the existing retry loop (it is regenerated); if it fails again, nothing is executed (fail-closed).

The improvement, in a nutshell: it turns the only rule that lacked a deterministic basis—isolation between clients—into a guarantee, and adds a cost cap. Three-layer defense-in-depth: identity (IAM), query format (the gateway), and scale (the byte limit).

Project 2 — ralph-vault-skill: Documentation with Source and Expiration Dates

As a true Karpathy fanboy, I’ve been working with an Obsidian vault (Markdown with wiki links and YAML frontmatter) for a long time. As a complement, I use Pinecone for semantic search on those notes (the classic “find me the document about X”); and, since embeddings don’t handle questions like “What is the current state of X?” very well, I use a hand-written bitemporal “state ledger” (YAML) that’s mirrored to BigQuery so that the agents themselves can perform JOINs against the current state.

This stack is excellent for understanding (the why, the decisions, the status), but it has a blind spot: it doesn’t document the code itself in a structured way, and the prose about the code quietly becomes outdated—Pinecone will happily return an obsolete note because it’s the closest match, not the most up-to-date one.

It’s an issue I’ve been mulling over for a while, and I hadn’t found a perfect solution. I’ve explored options like Graphiti, but they haven’t quite convinced me for my specific use case. And it turns out I came across Ralph in the repo.

ralph-vault generates a multi-level deepwiki from the source code, and there are two things that won me over:

– Origin + expiration. Each document lists the commit and the source globs it covers; a check compares this against the diff since the last sync and tells you which sections have become obsolete (staleness by path, not by the repo’s HEAD) and which files exist that no document covers (omission audit).

– A validation check on the wiki itself (frontmatter, wikilinks, and a “no copied source code” gate: summarize, don’t paste).

We built one for our agents’ code and their data layer (the dbt project that powers them), generating it with a single run of documentation per repo. It fits in with what I already had: the vault stores the rationale, Pinecone retrieves it, the ledger provides the current status, and now the deepwiki provides the structured map of the code—and, unlike prose, it knows when it’s lying.

The technical insight I’ve taken away: when it comes to “Is this still true about the code?”, documentation that includes sources and expiration detection outperforms embeddings based on prose. Vector search gives you the most similar snippet—not the most up-to-date one.

What I took with me

Interestingly, my two adoptions fit into two stages of the control loop described by Bergsten: the filter/guardrail (the gate that validates each query) and the evaluation/governance (the knowledge base that monitors its own obsolescence). Without intending to, I have incorporated two elements of the very same control loop that drives this initiative.

A note: not all of the repository consists of production code at this point—there are research prototypes and READMEs that have been heavily “polished” by AI—but the ideas are very valuable, and knowing how to distinguish one from the other is part of that value. And that, to me, is the real headline: a media consulting firm improving its internal agents using a bank’s AI security repository is, exactly, the democratization the article is talking about.

If you’re building with LLMs, check it out. And congratulations to Santander for releasing it under a permissive license: more of this, please.

Alex Masip Data & MadTech Director MIO One