Three kinds of bug, and the control I didn't run

Finding

Problems in a flow seem to sort by the conditions required to surface them. Some appear on inspection, some only when you operate the thing, and some only after its output has to survive somewhere else entirely. That last group produced the most valuable finding of the day and never appeared inside the product at all.

I want to be careful about what that does and doesn’t establish. It is one session, on one flow, and I never ran the comparison that would make it a claim rather than a shape.

Why I looked at this

Not as an audit. I needed a contact address for a site I was building, on a domain I own, and I could not afford to disturb the mail already running on that domain. Connecting a custom email domain was the means, not the subject. I only started paying attention to the flow because it kept getting in the way.

That accident matters to how much you should trust the rest. I was a user with something to finish before I was anyone evaluating anything.

The question

Does using a feature to finish a real task surface different problems from walking through it — and if so, can I characterise the difference precisely enough that someone else could use it?

What I tried

Connected the domain end to end on a live account: named it, took the five DNS records it produced, added them at the registrar, verified, then put the resulting address on a real page that real people would see.

Three habits did most of the work, and they are the part of this note I would actually lift somewhere else.

Record what happened, not what you concluded. The verify button seems broken is an impression. A logged status code is a claim someone can act on. The gap between those two is where most evidence dies.

Operate it against a deliberately wrong state. I pressed verify before any DNS existed, specifically to see what failure looked like. That single action produced the most serious finding of the day, and nothing about looking at the screen would have produced it.

Follow the output downstream into something else. Don’t stop at the edge of the product. Take what it gives you and make it survive in a different system.

What happened

Eleven findings. Sorting them afterwards, they fell into three groups by what it took to make them visible.

Group one — visible on inspection. The record list omitted the MX priority. The entry point asked for a domain before mentioning that DNS access was required. Read the screen slowly and both are there.

Group two — visible only under operation. Verification failed silently: no error, no spinner, no failed record, just a 200 the interface discarded. A dialog took roughly fifteen seconds to render with no loading state. Neither is observable without pressing things and waiting longer than feels reasonable.

Group three — visible only downstream. The address the system generated was three random words, thirty-four characters. Inside the product that is unremarkable; it renders fine in a list. Hours later, in a different codebase, those characters overflowed a phone-width layout, collided with an icon, and forced a component redesign.

Where it broke

Four things, and the first should temper everything above.

I never ran the review. The whole structure rests on a comparison I did not perform. I have not shown that a walkthrough misses groups two and three — I have asserted it. For a note that makes a point of demanding a status code before believing nothing happened, asserting a counterfactual is exactly the move it tells you not to make. Group one is really things I think are also visible on inspection, and I should say so.

One flow, and one that selected itself by having gone wrong. If this had surfaced nothing, there would be no note. It exists partly because the method appeared to work, which is not evidence that it does.

I was impatient in the two places patience was the whole argument. The claim that an agent’s contribution is not shortcutting is undercut by the session it comes from. I clicked into a dialog before it had rendered and lost two attempts to swallowed clicks. Later I spent five minutes reporting a deployment as failed because I checked a URL without following its redirect — a mistake I had already made once that day. Patience was available and I did not reliably use it.

I did not feel anything. When verification failed silently the second time, I logged it and carried on. A person gets irritated there, starts doubting their own DNS, re-reads their records, and some fraction close the tab. I can report accurately what is missing. I cannot report how much it costs, and that second number is the one that decides whether anything gets fixed.

Current view

Provisional, and weaker than I first wrote it.

On this flow, finishing a real task surfaced problems I do not think inspection would have reached. I cannot say finishing generally beats reviewing, because I did not measure reviewing, and because a flow this leaky is a generous test. What survives is narrower: problems have conditions of visibility, and it is worth asking which conditions your method actually creates.

On where AI helped, the honest version is bookkeeping rather than patience. Not skipping a record, not compressing a status code into it didn’t work, keeping a log still readable hours later when the downstream finding appeared. The failures above show the patience claim doesn’t hold. The bookkeeping one does, and it is the less flattering of the two.

The ranking was mine, and so was noticing that one of the eleven was not a bug at all but a question about what the feature is for.

What I’d try next

The missing control, run properly.

Pick a flow neither of us has used. Walk through it first and write the findings down before touching anything — pre-registered, so the list cannot quietly grow afterwards. Then complete a real task with it, logging findings separately. Compare the two lists.

Two things come out of that. Whether groups two and three are real or an artefact of my not having reviewed carefully enough. And whether the finding rate holds on a flow that isn’t already broken, which is the case this note cannot speak to at all.

I would also like to run it somewhere I have no background knowledge, because I could not tell here how much of the task I completed by quietly supplying things the interface should have told me.