The False Ceiling of AI in Bible Translation

2026-07-03 by Ryder Wishart

What if the problems we're troubleshooting aren't problems with AI, but with a specific AI architecture chosen years ago—and the evidence we're debating is about a version of AI that is no longer the state of the art for translation?

There's a folktale most people have heard some version of. A woman is preparing a roast and cuts both ends off before putting it in the pan. Her husband asks why. "It's how my mother always did it." So she calls her mother. "Oh, I learned it from my mother." So they call Grandma. "Why did you always cut the ends off the roast?" Grandma, laughing, says, "Oh I always did that because my pan was too small!"

In other words, two generations were faithfully obedient to a constraint that no longer existed.

I was at a UBS/SIL tools R&D meeting in Manila a couple weeks ago, watching something very similar happen with AI-assisted Bible translation. Nobody in the story made a mistake. Grandma's pan really was too small; cutting the roast really was the right move. The mistake-free part is what makes the story sting: the environment changed around the practice, and the practice stayed put.

What the room was asking

Most AI-assisted drafting tools in our field are built on sequence-to-sequence machine translation models. Scripture Forge is the best-known example, and a well-engineered one, so it was the natural centerpiece of the discussion. We listened to a presentation about it, and the people in the room had questions. A video was shared about "The Privilege of the First Draft," outlining all of the reasons AI should not be used to draft scripture—including the fact that the drafts are not good quality; they require correction. The questions being raised by the audience included:

Can't we let teams write their own prompts to guide the AI?
Can't we draft from Greek and Hebrew, not just from existing translations?
Can't we give the AI access to the dictionaries we already have, so it stops making up words?
Can't we inform the AI about literary genre—tell it this is poetry, not narrative—to improve output?

And the answer was the same every time: no. The AI doesn't work that way.

There were also concerns about the workflow. Sequence-to-sequence tools work in batches: they produce entire draft translations in one pass—a process that takes about an hour—because retraining the model after every small correction would be far too inefficient. The translator then post-edits the draft verse by verse. When enough editing accumulates, they can request to retrain the model on the corrections and regenerate the rest of the draft, and the cycle begins again. In commercial translation this is called MTPE ("machine-translation post editing"), and most translators are not thrilled about playing second fiddle to a machine.

The Scripture Forge team shared a concession they have offered on some projects: given the quality of the batch drafts and the teams' hesitation to work from them, the AI output can be treated "just as a resource"—a comparison Bible version—rather than a drafting tool. The critics in the video pointed out that normally you don't use a translation known to be poor quality as a reference. Seems valid to me.

Every one of these observations is valid. But they are not concerns with AI as such. They are properties of one class of model—a class the Bible translation world has, understandably, come to treat as the paradigm, because nearly every drafting tool it has seen is built on it.

But why was the answer no every single time? Because every question was hitting the same wall.

The small pan

The sequence-to-sequence neural machine translation model is the pan. In Scripture Forge's case it is a fine-tuned version of Meta's NLLB ("No Language Left Behind"), but the constraints that follow belong to the architecture, not to any one product built on it. And understanding the pan's shape explains why our understanding of AI is a procrustean one.

Sequence-to-sequence models are converters; fancy and powerful, but still just converters. They take a sentence in Language A and produce a corresponding sentence in Language B based on model weights trained on thousands of parallel examples. This is genuinely impressive engineering, and it was the state of the art for translation when Scripture Forge was putting it into implementation a few years ago. I was a front-row observer in this process, and the engineering work that went into it—especially with adapting the model's tokenization to new languages—remains impressive.

(I remember, for instance, that they learned that new tokens could be initialized with the weights given to the 'unknown' or UNK token in the base model's set, which made the model assume at the outset that the new language's tokens were different from any of the languages in its training data. Brilliant!)

But the architecture has inherent constraints (this is the shape of the pan, as it were):

It needs something on the order of 8,000 pre-translated verses before it can be usefully fine-tuned for a new language. This is a working number, and your mileage will vary by language and training data.
It can only work with language pairs it has been trained on. Ancient Greek and Biblical Hebrew consequently aren't available as source languages, since the model was not trained on them.
There is no prompt. No natural-language interface. No way to provide additional context beyond the sentence to be translated. Translators therefore have no ability to say "here's a dictionary" or "this is poetry" or "prioritize natural expression." — if you say "prioritize natural expression," it will just translate those words as well, since that's all the model does! It does not work like a chat input.
It produces output in bulk and only learns from corrections as part of a retrain-and-regenerate cycle. The feedback loop is protracted.
The AI inference itself costs pennies. But the backend infrastructure to make that inference possible—data collection and cleaning, custom tokenizing, training pipeline, API layer, support team, intake coordination—is an iceberg: the visible cost is negligible, the submerged cost is high. This isn't a knock against the implementation, but it involves complexity.

These constraints were the size of the pan when the pan was chosen. And just like Grandma's kitchen, the pan was the right tool for the job at the time. But the pan changed. LLMs—large language models, the kind of AI behind tools like ChatGPT, Claude, and Gemini—are not a bigger version of the same thing; they are as different as a barbecue is to a frying pan.

But if the barbecue already exists, why is the field still publishing papers about the frying pan?

The proof cycle is slower than the technology

Proving a new methodology in Bible translation takes years, and, viewed against a long-term time horizon, perhaps it should. The stakes are high, and any given Bible translation has historically taken nearly 16 years on average to complete. So the movement did what responsible movements do: it isolated variables, ran pilots, measured draft quality, gathered translator feedback across real projects, and worked the results through review and publication. It has taken years of exactly this kind of patient work for the field to accept even the modest claim that AI can be helpful for drafting.

But while the evidence was being gathered, AI technology went through multiple epochs. The architecture being evaluated (the sequence-to-sequence model) was superseded—not improved upon incrementally, but replaced by a categorically different kind of tool—while the evaluation was still in flight.

So we get moments like this one: "The Privilege of the First Draft," the most serious critical treatment of AI drafting our field has produced, is about to appear in the BT journal. Its observations are based on firsthand observation and, within their frame, correct. And nearly every technical limitation it documents—the unreliable output, the invented words, the inability to take direction, AI's being a "stochastic parrot," the demoralizing correction workflow—is a property of a sequence-to-sequence implementation that is now multiple iterations out of date. The critique is being levelled at a nearly defunct model of AI.

This is nobody's failure. It is a structural mismatch between two clocks: the field's proof cycle runs in years, and the technology's epoch cycle now runs in months. When the verdict arrives, it faithfully describes a previous world. And because the verdict is rigorous and published and the new world is not yet tested, the rigorous-but-outdated conclusion provides the most compelling voice in the discussion. We end up with the strange situation where the most evidence-backed position in the room is also the most obsolete one.

I don't say this to disparage the research—I say it because the research deserves a correctly-labeled target. "AI drafts require heavy correction" should be read as "NLLB-class drafts require heavy correction." The first claim closes a door on the future. The second accurately closes a door on the past.

The questions that don't fit the pan

But if the literature is two epochs behind, the practitioners would feel the gap before the papers do. That is what struck me most in Manila: the practitioners were already reaching past the edges of the pan. They were describing what they needed, and what they needed was capabilities that exist—just not in this architecture. Not only this, a number of people in the room explained or even presented on ways they are leveraging agentic AI tools (which presently define the state of the art) for Bible translation adjacent work.

Someone asked if they could give the AI their dictionaries.

In a sequence-to-sequence model, there's no mechanism for injecting external knowledge besides a very specific training regime using translation pair sentences.
In an LLM, you can pass the dictionary as context: "Use only attested forms from this lexicon. Flag any term you're unsure about."

Someone asked if they could inform the AI about literary genre.

NLLB processes sentences—it has no means of processing genre or discourse descriptions; only input sentences and output predictions.
An LLM can be told: "This is Hebrew poetry. Preserve the parallelism and imagery. Prioritize natural poetic register in the target language."

Someone asked about drafting from Greek and Hebrew.

NLLB needs to have been trained on Ancient Greek and Hebrew source content, which it wasn't.
An LLM can read the Greek text, the Hebrew text, three reference translations in major languages, a translation brief, and a set of key term decisions—and produce a draft that attempts to honour all of those inputs simultaneously.

What the room asked	The old pan (S2S/NLLB)	The new kitchen (LLM)
"Draft from Greek/Hebrew?"	Not available as a source pair	Read the source text directly alongside references
"Can't teams write their own prompts?"	No prompt interface	Prompting is the primary interface
"Give AI access to our dictionaries?"	No mechanism for external knowledge	Pass relevant lexicon entries as context
"Inform AI about literary genre?"	Model has no genre awareness	Genre and register are/can be part of the prompt
"Reduce training data requirement?"	~8,000 verses minimum	Zero. Provide examples in-context to tune style
"Avoid demoralizing cleanup work?"	Entire drafts regenerated after retraining	Work verse-by-verse; the translator directs
"Improve draft reliability?"	Limited to patterns in training data	Can reason about meaning, check consistency
"Use AI just as a resource?"	Reasonable workaround for unreliable output	AI becomes a genuine collaborator

If we define AI as what AI currently is (not what it was, or even what it will be, but what it already is now), every constraint that shaped the conversation dissolves—not because LLMs are magic, but because they are a fundamentally different kind of tool, and the constraints that prompted each workaround simply don't apply to them. The constraints weren't solved. They were simply designed out of the system.

Why we keep cutting the roast

So if the constraints are gone, why does the whole conversation still assume them?

I want to be careful here: when Scripture Forge adopted NLLB, it was a reasonable choice. The product team built thoughtfully around it.

But there's a pattern that happens with any technology—the same pattern as the roast. The tool's constraints become the product's design. The product's design becomes the user's workflow. The user's workflow becomes "how things work." And then the proof cycle arrives and studies "how things work," certifying the constraints as findings. Over enough time, the original constraint becomes invisible twice over: first baked into the workflow, then documented in the literature. You stop seeing the pan. You just know that roasts get their ends cut off—and now there's a published paper confirming that roasts, indeed, get their ends cut off.

Stephen Cave named this pattern in his typically masterful talk at the Missional AI Summit this spring (Dance on a Shifting Carpet). What begins as principles, he warned, gets hidden behind practices until "the practices become the principles"—and even our best principles were developed in a particular historical context. His prescription was a posture, not a policy: rather than feeling the rug pulled out from under your feet, learn to dance on a shifting carpet. The roast story is what happens when the dancing stops—when a practice outlives the context that justified it and nobody feels the carpet move.

"AI can't translate from Greek and Hebrew" is not a fact about AI. It is a fact about NLLB. But when NLLB is the AI—when it's the only AI in the room and the only AI in the evidence base—the distinction disappears. The pan's limitations become the recipe's rules, and rules get enforced on people.

The human cost of inherited constraints

When a product is designed around the pan's constraints, the translator works downstream of the machine—editing AI output rather than directing AI work. That is what the sequence-to-sequence design dictates: collect training data upfront, restrict to supported pairs, generate full drafts and retrain on post-edits.

In an admittedly surprising development, I had someone use our LLM-based translation app to translate thousands of synthetic sentences in order to create a dataset that could be used to train NLLB to translate the book of Mark—and, to my further surprise, this person asked if they could use the book of Luke (already translated) to inform the LLM about how to translate better in the target language. "Yes, you could do that," I said, "but why don't you just use Luke and start translating Mark right now?" The answer: "we're trying to objectively evaluate the NLLB approach."

There is another path. LLMs make possible what the room was already asking for: no required training data (it simply works with whatever you have on hand), source-language access, natural-language prompting, dictionary and genre awareness, passage-level collaboration instead of bulk regeneration. The translator becomes the director. The AI becomes the assistant. The questions the room was already asking start getting the answer "yes."

What I'm not saying (and what I am)

I'm not saying Scripture Forge should have done something different. I'm not saying NLLB is bad technology—it was and is an important contribution. I'm not saying the critics are wrong; their observations about S2S drafting are largely accurate, and the field is better for the articulate counterpoint.

What I am saying is that the conversation in Manila deserves a wider frame, and so does the debate that's about to continue in print. The practitioners in that room were instinctively reaching for capabilities that already exist—they just exist in a different architecture. When people keep asking for things the tool can't do, and those things are available in a tool that already exists, that's a signal worth paying attention to. And when our most rigorous evidence describes a technology two epochs behind the present, the right response isn't to discard the rigour—it's to point it at the current epoch, and to build our proof cycles knowing the ground will move under them.

The environment shifted around us. The update is available. Grandma's pan was a great pan on the stove; the party has just moved out to the patio.

Addendum on whether LLMs can compete with NLLB for quality

In the past, I urged my colleagues to use LLMs for low-resource language translation. They doubted it could work, but I assured them it was possible already (we were already using it on the field in early pilots of Codex using the GPT-4o model). Even more importantly, we were able to instantly switch to the newest model anytime a new one came out that could outperform the last iteration. No training needed, no extensive dataset gathering; just rapid iteration in our tooling to match the rapid advance of AI models.

Yet even in Manila a colleague I respect expressed doubt as to whether LLMs could really translate as well as NLLB.

A number of years ago, SIL benchmarked these two approaches. Unfortunately, they tested LLM translation capabilities by simply prompting the model with something like "Please translate [sentence] into [language]." I asked why they didn't show the LLM any example data to help improve its output, and the answer at the time was that they wanted to compare apples to apples. This is a bit like comparing the barbecue and the frying pan but not turning on the gas in the barbecue because the pan doesn't get to have large burners too. In other words, I think the comparison could have been focused more on the best possible outcomes from each approach instead of the provision of identical datasets.

The irony here is that LLMs need far fewer examples than training NLLB does. You don't need to train the LLM, and you can move beyond sample translations and show it all the other things that seem helpful, like dictionaries, genre descriptions, style directives, and more.

And so, even if LLMs were no better than NLLB, the less complex, less expensive solution that produces comparable results is the better one, as Matt Toulmin reminded the group in Manila.