Capturing the Quiet Gremlin

I have learned from years of ERP, finance systems and data projects that problems rarely arrive wearing a name badge. Most often they stay quiet, keep their disguise, and ambush the process at a critical time. AI workflows are no different. In fact, they can be worse, because the error often arrives looking polished.

The quiet gremlin.

Error Whiteboarding

Just a minute or two of focused error whiteboarding can land real outcomes that have to be accounted for and handled.

Here is what I got. No doubt you can think of others.

• A missing field.
• A bad assumption.
• A connector timeout.
• A duplicated record.
• An old loader spreadsheet.
• A slightly wrong mapping.
• A file in the wrong format.
• A supplier record that looks right but isn’t.
• An interface that “usually works”.
• A report that balances until someone asks a different question of it.

Or, what seems to be classic nowadays, a prompt that was interpreted slightly differently from the last time and differently from what the user intended.

In normal systems delivery, we mitigate errors via error handling techniques. ERP, finance, reporting and integration processes have failure points. We design controls around them: validation checks, exception reports, approval steps, reconciliations, audit logs and support procedures.

Consistent with my previous paper, Finance AI: 5 Control Questions, AI errors are less visible. They can look very polished and not be obvious.

A failed lookup can still produce a confident answer. A missing source document can still generate a report. An ambiguous instruction can still create a beautiful but wrong output. A workflow can run successfully in technical terms while still producing business nonsense.

The job is not simply to build an AI workflow that works when everything is perfect.

The job is to build one that knows what to do when something is missing, wrong, uncertain or incomplete.

It is not always dramatic. It is often small, boring and easy to miss. But left alone, it can travel downstream and eventually become a much bigger problem.

A workflow can fail to read the right document and still produce a well-written answer. It can use an old source and still sound current. It can miss a field and still complete the task. It can make an assumption and present it as if it were fact.

This is where we need to stop treating AI workflow design as a clever prompt exercise and start treating it as a proper business process.

Because that is what it is.

The Actual Process

Not the ideal process. Not the slide-ware version. The real one.

• Who starts it?
• What do they submit?
• Where does the file come from?
• Which system holds the source data?
• What does the AI need to read?
• What should it ignore?
• What output is expected?
• Who checks it?
• What happens after that?

Then ask:

What could go wrong, and is it mitigated?

• Wrong file.
• Incomplete data.
• Vague prompt.
• Out-of-date collateral.
• Connector failure.
• Miss-branching workflow.
• Plausible but fabricated answers.
• Erroneous auto-approval.

None are exotic errors, and so they should be mitigated for.

Nested Pre-validation

Basic but important: nested pre-validation captures errors before AI even gets involved.

• If the workflow expects a spreadsheet, check that a spreadsheet has been provided.
• If it needs specific columns, check they exist.
• If it needs specific values, check the values.
• If it needs a contract, check that the uploaded file is actually the contract and not the email that came with it.
• If it needs current data, check the date.
• If a field is mandatory, do not let the process continue without it.
• If the instruction is too vague, stop and ask for clarification.

OK, but what do I mean by nested?

For a 2,000-line spreadsheet, make sure not to hard stop the process simply because an error has been found.

Check through all 2,000 lines and list the errors.

Otherwise, you face fixing the same file multiple times as you cycle through each line.

Serious Prompting Check

A prompt should not just say:

“Summarise this document.”

It should say something closer to:

“Use only the attached document. Do not invent missing information. Identify any missing sections. Flag uncertainty. Separate facts from assumptions. If the document does not support the answer, say so.”

So check that it does.

Pass the prompt through a validation analysis and check whether it meets the criteria. If it does not, determine whether that creates a hard or soft error threshold and act accordingly.

So, how?

A practical approach is to use a hyper-guardrail validation prompt, including the JSON version of the submitted prompt, and pass it into your favourite GPT.

Then ask a binary Boolean question:

Do the required guardrails exist in the subject prompt or instructions?

Yes? Fine, continue.

No? Chuck it into error handling.

That may sound blunt, but it is better than allowing a weak prompt to quietly drive a business process.

Build the Unhappy Paths

The happy path exists, but this is on a probability outcome spectrum where the unhappy path also lives.

To be clear, the happy path is:

• User submits request.
• AI does task.
• Output is produced.
• Everyone is impressed.

Fine.

But what happens when the source is missing?

What happens when the API times out?

What happens when the output confidence is low?

What happens when the AI finds contradictory information?

What happens when the workflow is about to send something externally?

These need to be handled.

In practical terms, the workflow needs a few basic routes:

• Retry the step if it is a temporary technical failure.
• Stop the process if a required input is missing.
• Route the issue to a person if judgement is needed.
• Hard stop the process if defined hard-stop criteria are met.
• Log the event so someone can see what happened later.

This is not complicated in practical terms. Most workflow tools can do this. Power Automate, n8n, Make, Zapier, Azure Logic Apps and custom Python workflows can all support branching, conditions, retries and notifications.

The important thing is to build the exception path deliberately.

Do not leave the gremlin to wander around the process unsupervised.

Test the process with multiple unhappy path examples to make sure the gremlin has fewer places to hide.

Do Not Waste the Errors When They Emerge

The point is to automate the right things, with the right controls, and with a clear way of handling the parts that go wrong.

More importantly, the workflow should help make things go right by surfacing the errors and possible fixes.

Errors are valuable. They can contribute immensely to the integrity of the workflow.

• If people keep uploading the wrong document, fix the intake form.
• If the AI keeps missing a field, change the extraction logic.
• If the reviewer keeps rejecting the same type of output, improve the prompt or the template.
• If the source data is regularly weak, do not blame the AI. Fix the data.
• If the workflow often needs judgement, maybe that step should not be fully automated.

Final Thought

The real test of AI workflow design is not whether it works on the happy path.It is whether it fails safely, visibly and can be recovered when reality gets messy.

That is how you capture the quiet gremlin.