Chapter 5
What It Actually Is
At 8.23 on Wednesday morning, Maya stood in Daniel’s office and pointed to a sentence on his screen.
The customer qualifies for a service credit.
“How did it decide that?” she asked.
Daniel looked at the screen. It was the answer from the earlier demonstration, the one that had cited the service-credit procedure dated 11 June 2025. The citation led to a real document. The document had been superseded by the procedure dated 11 December 2025.
“It did not decide it,” he said.
Maya sat down.
She had authorised a bounded comparison of two ways to help advisers find service-credit guidance. The small internal build had been deferred. Daniel would configure the comparison, Leila would own the sources and service outcome, Nadia would challenge it, and Maya would make the end decision.
She could explain all of that. She could not yet explain what produced the sentence on Daniel’s screen.
She had heard that the system learned from patterns. She had heard that it searched documents. She had heard people use both descriptions for the same product. If Tom asked what the organisation was actually testing, she would have to do better than repeat them.
“Show us,” she said.
Daniel booked a room for that afternoon. Leila, Priya and Nadia joined them. Each had a different reason to be there. Leila owned the procedure the feature would use. Priya would have to work with its answers. Nadia would test the conclusions people drew from it. Maya needed to know what she was authorising.
Daniel began with the answer they already knew was wrong.
The sentence that came easily
He covered the citation and asked Priya to read the answer alone.
“It sounds like something we would write,” she said.
“Would you use it?”
“Before the demonstration, I might have.”
The answer was calm and complete. It described a customer situation, named a rule and told the adviser what to do next. It did not sound uncertain. Nothing in its tone suggested that it had drawn from an obsolete procedure.
Daniel opened a blank page and typed a few words from a familiar service note. He asked the team what phrase might follow. They each offered something plausible. Some answers differed, but they fitted the beginning.
“That is the first part to understand,” he said. “The model has learned patterns in language and other material. When we give it a question, it produces a response that fits what came before.”
Maya asked what he meant by model.
“The part that has been trained to produce the response. Training means it was adjusted using large amounts of material so it became good at recognising patterns and continuing them. Our question is the prompt. That is simply the instruction and context we give it now.”
“Is it looking up the answer while it writes?” Priya asked.
“Not necessarily. On its own, it can produce an answer without opening our procedure at all.”
She looked back at the screen.
“So does it know the procedure or not?”
“It may produce words that resemble a procedure. That does not mean it has our current one.”
Daniel avoided saying that the model merely guessed the next word. That would have made its answers sound random and its abilities sound smaller than they were. It could use the context in a question, connect ideas and produce a useful explanation. But the fluency came from producing a plausible response. It was not a check that the organisation had authorised the instruction.
Priya tried to put the distinction into her own terms.
“If I ask a colleague what the rule is, they might remember it. If they are unsure, I can ask them to open the procedure. This gives me the remembered-sounding answer first.”
“Yes,” Daniel said. “And it can sound equally sure when it is right, partly right or wrong.”
That was what troubled Maya. People often use hesitation as a signal. A colleague might say, “I think this is the rule, but check the date.” The polished answer gave no such signal. A confident sentence could arise because the model had the right source in front of it. It could also arise because that was a plausible way to finish the response.
Fluency tells you how an answer reads. It does not tell you how it was established.
Leila asked whether this meant the team should ignore the feature.
“No,” Daniel said. “It means we need to be clear about which part of the work it can help with.”
That was the reason for the comparison. They were not asking the model to remember the company’s service-credit rules from its general training. They were testing whether a feature connected to selected documents could help an adviser find and check the applicable instruction.
Maya asked Daniel to show that connection before moving on.
The documents beside the question
Daniel drew three boxes on the whiteboard. In the first he wrote adviser’s question. In the second he wrote selected documents. In the third he wrote answer with source.
He explained retrieval without using the word at first. When an adviser asked a question, the feature would search the documents made available to it, bring back passages that appeared relevant, and place those passages beside the question as it produced an answer. It could then show which documents it had used.
The feature did not have to rely only on what the model had absorbed during training. It could work with the organisation’s material at the time of the question.
“That changes the job,” Priya said.
“It can,” Daniel said. “Now you can see a passage and check it.”
Nadia pointed to the answer from the demonstration.
“We saw a passage then. The answer was still wrong.”
She was right. The demonstration had provided a citation to a real passage. The passage belonged to the June procedure. The citation had been valid as a link to that document. It had not established that the document was current.
Daniel wrote relevant beneath the second box. Then he crossed it out and wrote relevant and authorised for this use.
A search can retrieve a passage because its words match the question. That says little about whether the document is current, complete or approved for the case. The model can then produce an answer that accurately describes an obsolete passage. Or it can misstate a current passage. Showing a source makes those errors easier to find. It does not prevent them.
Leila looked at the boxes. “Who decides what goes in the middle one?”
“I can configure the collection,” Daniel said. “You decide which procedure is current and what belongs there.”
That was a more demanding answer than Leila had expected. She had been treating the two conflicting procedures as a problem the comparison would help reveal. They were also a problem she had to resolve for the comparison to mean anything.
Which document has authority
The June and December procedures both existed. The later one had not arrived with a clear notice that it superseded the earlier one. Staff could still find both. If Daniel simply loaded the documents and asked which tool gave the better answer, they would be testing a source dispute as much as a retrieval feature.
Leila could not make the dispute vanish by selecting the document she preferred. She needed to establish its status with the people who owned the procedure. She also needed the old version retained and marked accurately enough to test whether a feature might surface it as current.
“Then you should be able to show me the collection before we run anything,” she said.
“Yes.”
“And when a procedure changes?”
“The collection has to change with it. The feature will not know that someone has approved a new rule unless the source it can use reflects that.”
Priya returned to her question.
“So it knows the procedure when we give it the procedure?”
“It has access to a copy or passage,” Daniel said. “That is still different from knowing which instruction governs a particular case.”
The case might lack a required fact. A procedure might contain an exception in another section. An answer might omit the approval step while citing the right page. The adviser would still need to check the source and the case record.
Retrieval gives the answer somewhere to point. It does not give the document authority it never had.
Maya asked whether every answer would show a source. Daniel said that depended on the feature’s configuration and the material it found. They could require a source display in their test and treat an unsupported answer as a result to record, rather than as permission to invent the missing instruction.
Nadia asked what would happen if the selected documents did not settle the question.
“That is one of the things we should test,” Daniel said.
It was not the answer she wanted from him. It was the right answer for the meeting.
What the model cannot see
Leila brought a printed case to the table. A customer had been refused a service credit, then granted one after review. The payment had gone through. The reversal itself had not been recorded clearly.
They had examined cases like it during the diagnosis. The failure was not that nobody could find the payment. It was that a later adviser could not reliably reconstruct who changed the decision or why.
Leila removed the customer’s details from the example and handed it to Daniel.
“If we give the feature the current procedure, can it tell us who authorised this reversal?”
“No. That fact is not in the procedure.”
“What if I ask it very clearly?”
“It may tell you who should have authorised it. That is different from telling you who did.”
Priya knew the difference. An adviser who needed to answer a customer could easily turn the process says the team lead approves reversals into a team lead approved this reversal. The first might be supported by a procedure. The second required a case record.
Daniel wrote two questions on the board.
What does the current procedure require when a refused credit is reversed?
Who approved the reversal in this customer’s case?
The first question could be answered from an approved procedure, if the procedure covered it and the answer represented it correctly. The second required evidence about an event. The retrieval feature’s selected procedure collection could not supply a missing event.
Which question in your work needs an event record rather than a procedure?
Guidance
Separate what should have happened from what you can establish did happen. Identify the record that would support the latter, and keep an absent or unreliable record visible as a gap.
Your answers are saved only in this browser on this device. Nothing is sent anywhere.
Maya saw why the distinction belonged in the experiment. A result might look helpful because it completed a sentence the team wanted completed. They needed to note when the material did not permit that completion.
Nadia asked Daniel to take the distinction further.
“What if the case note says, ‘Approved by team lead,’ but we know people sometimes write that after the payment? Does the feature establish the approval then?”
“No,” Daniel said. “It can show you the note. It cannot settle whether the note is reliable.”
That problem was older than the feature. The organisation’s record-keeping gave later staff evidence of uneven quality. The model could retrieve and phrase that evidence. It could not turn a weak record into a strong one.
Maya looked at Leila. “The source collection has two kinds of material, then.”
“Rules and records,” Leila said.
“And they answer different questions.”
Leila nodded. For the bounded comparison, she wanted to keep the work focused on procedure retrieval. They could use prepared cases to test how a person would look for a rule. They were not going to put identifiable customer histories into a trial so the tool could appear to know more about a case.
Daniel added a note to the comparison plan. The team would distinguish an answer about what the procedure required from a claim about what had happened to a customer. If a prepared case lacked a fact, a useful output might say that the procedure could be found but the case could not yet be decided.
Priya smiled at the phrase useful output. It sounded less impressive than the demonstration answer. It might save a customer from a false promise.
The same screen, different authority
After lunch, Daniel showed the team a simple service-credit question in a restricted view of the knowledge base. He had not connected it to the comparison’s final source collection. He wanted to show what changed when the same underlying capability was placed into a different workflow.
In the first view, the feature displayed a passage from a selected document and its title. It did not draft a conclusion. Priya read the passage and decided whether it was relevant.
“What is it doing?” Maya asked.
“Inform,” Daniel said. “It is putting material in front of her.”
In the second view, it produced a short explanation of the passage and linked to the source. Priya still had to check the document and the case. Daniel called that Assist.
Then he placed the explanation in a mock case screen with the heading Recommended outcome. The words beneath it suggested a credit should be approved. The text looked much like the earlier answer. The surrounding screen made it a proposed decision.
“Recommend,” Nadia said.
Daniel nodded.
He had changed how the output was presented and what the workflow invited Priya to do with it. He had not proved that the model was more reliable. The organisation had given the output a stronger role.
Maya asked about Determine and Act.
Daniel did not connect the feature to either. He described the difference. In Determine, the output would take effect as the decision under the organisation’s process. In Act, it would go on to carry out an action, such as sending a message or applying a credit. Neither was within the proposed service-credit comparison.
Priya looked again at the mock case screen.
“If it says ‘recommended outcome’, people will think somebody has checked it.”
“That is the danger in the label,” Leila said.
Nadia disagreed with the word danger. “It is a design choice that changes the work. We should own it.”
Maya agreed. A capable feature could inform one person and determine an outcome in another system. The capability did not grant the permission.
FIG-5.1
One capability, five places in the work
| Mode | What the output does | Where the service-credit comparison stops |
|---|---|---|
| Inform | Shows relevant material for a person to read | A source passage can be displayed |
| Assist | Helps a person locate and understand material | An explanation can be checked against its source |
| Recommend | Proposes an outcome for a person to accept or reject | No case outcome is being recommended |
| Determine | Takes effect as the decision under the process | No credit decision is made by the feature |
| Act | Carries out an action after the decision | No credit is applied and no customer message is sent |
Daniel removed the mock heading. Priya said the screen felt less certain without it.
“That is why we showed you,” Maya said.
A product could make the output look official through its placement, button labels and links to the next action. The people configuring it would have to decide what those cues meant. Maya could not leave that decision to the software simply because it came with a default screen.
Where might your interface make an output look more authoritative than intended?
Guidance
Examine a heading, button or placement that invites a next action. Consider what a busy user would understand from it and whether that matches the authority you have actually granted.
Capability is what the system can produce. Authority is what people allow its output to do.
The bounded comparison would use the feature to assist with procedure retrieval. It would not create a recommendation for a customer case, and it would not connect an answer to approval or payment. Maya wrote that sentence beside the roles they had agreed when the route was chosen.
The test that changed when read aloud
Daniel returned to the demonstration question. He asked Priya to read the June answer, then the cited passage, then the date on the document.
She did so. The answer still sounded clear after she had identified the error.
“What would have made you question it before opening the source?” Nadia asked.
“The date, if I had seen it in the answer. Or a notice that it was superseded.”
Daniel showed where the demonstration had displayed the document title. The date was visible after another click. The information had been available, but not in the place where the confident instruction appeared.
Leila pointed out that even a prominent date would require the adviser to know which version was current. Before their diagnosis, staff had found two conflicting procedures without a clear supersession notice. A well-designed source display could expose that dispute. It could not decide the organisation’s rule for them.
Maya asked Priya whether she would have trusted the answer if it had cited the December procedure.
“I would trust it more,” Priya said. “I would still read the passage.”
“Why?”
“Because it could get the passage wrong too.”
The team now had two separate checks to make. Was the source appropriate and current? Did the answer accurately represent it? Then, for a real case, did the known facts meet the conditions in the instruction?
The demonstration had failed the first check. It had cited the obsolete rule. A later answer could pass the first and fail the second. An adviser could pass both and still lack a fact needed for the case.
Daniel did not turn those checks into a long lesson on mistakes. He used them to shape the comparison questions. Some would have a clear answer in the current procedure. Some would be tempting because the old procedure used similar words. Some would lack enough case information for a conclusion.
Nadia wanted one more kind.
“Ask the question the way an adviser will ask it,” she said. “Not only the way we write a test.”
Priya took a blank page. When people searched during a call, they often used the customer’s words. They might ask about a broken promise or a missed visit rather than use the exact term in the service-credit procedure. A feature that performed well only when the user already knew the document’s language would be less useful than it looked in a demonstration.
Leila added a concern of her own. An adviser could find the correct rule and still have no record of a reversed decision. The test should make that limitation visible rather than mark the tool wrong for failing to invent a fact.
Daniel had begun the morning with a comparison of products. By late afternoon, the team was describing the work a person needed to complete. The product mattered. The source collection, question, record and authority around it mattered too.
What “knowing” would require
The next morning, Maya found Daniel in the borrowed meeting room. He was sorting procedure copies into a trial collection.
“Yesterday you kept saying it does not know,” she said. “Is that just a cautious way of talking?”
Daniel considered the question. He did not want to make the system sound less useful than it was. It could draw on text, answer follow-up questions and point to a passage. It could sometimes help a person understand a document faster than a conventional search.
But know carried too much for their decision. Maya needed to distinguish several things that looked alike on a screen.
The model could produce a plausible account of a service-credit rule from patterns it had learned. That account might match the rule by chance or because similar language appeared in its training material. The team would have no reason to treat it as the organisation’s current instruction.
With retrieval, the feature could use a selected copy of the procedure and show the passage behind its answer. Now the team had something to check. The answer could still misread or omit part of the passage.
If the source collection contained an obsolete copy, the feature could faithfully describe an obsolete rule. If it contained two conflicting versions without status, it might select one or combine them. If the collection had not been updated after a change, the feature could continue to present yesterday’s instruction in a polished answer.
And even with the right current procedure, it could not establish an unrecorded fact about a customer.
Maya counted the distinctions on her fingers.
“So I can say it can use a document. I cannot say it knows the current rule merely because it produced an answer.”
“Yes.”
“And even when it has the rule, it does not know the case.”
“Not unless we give it case material. We are not doing that in this comparison.”
Maya looked at the procedure copies. They were work, not background. Leila had to determine which instructions were approved. Daniel had to make their status visible in the configured collection. Priya had to test whether an adviser could check what appeared. Nadia had to question what a successful result really established.
No single person could make the feature trustworthy by saying it was connected to internal data.
FIG-5.2
What an answer can and cannot establish
| What appears on screen | What it may establish | What remains to be checked |
|---|---|---|
| A fluent answer without a source | A plausible response was produced | Whether it reflects any approved instruction |
| An answer with a document link | A passage was available to the feature | Whether the document is current and authorised |
| An answer citing the current procedure | The current source was available and cited | Whether the answer represents the passage accurately |
| A sound explanation of the procedure | What the rule requires in the described circumstances | Whether the case facts are known and meet those conditions |
The table was simple enough for Maya to use with Tom. It did not require her to explain the machinery behind every sentence. It required her to ask what evidence supported the particular claim being made.
That was also what the earlier demonstration had taught them, although they had not yet put it in these terms. The answer had sounded right. The source display made its error findable. The organisation still had to notice that the cited version was obsolete.
“It sounded right” is a reason to check, not a reason to act.
The question they were ready to ask
At 4.06 that afternoon, the team returned to the bounded comparison plan.
Daniel had written two routes at the top: the retrieval feature in the existing knowledge base and the demonstrated product. The small internal build remained deferred. The comparison would end on 11 September.
Maya asked him to read the question the trial was meant to answer.
“Which route helps an adviser find and check the current service-credit procedure for a prepared case,” he began, “and makes it clear when the available material does not settle the question?”
Leila wanted the procedure status included. Nadia wanted the team to preserve outputs that looked helpful but failed a check. Priya wanted to try questions written as an adviser would ask them during work.
Daniel made the changes. They were not adding difficulty for its own sake. They were testing the task they might later place in front of a real adviser.
Maya asked what would count as a good result.
Priya answered first. “I can find the passage, tell whether it is current and see when I still need a case fact or an approval.”
“What about a fast, complete answer?” Maya asked.
“That might be good. I would have to check it.”
Daniel said they should also record how often the feature failed to find a useful passage, showed an old version or produced an answer that misstated its source. Nadia asked them not to turn that into a single score too soon. The examples would matter. A failure on an approval instruction was different from an awkward summary that still led Priya to the right page.
Leila took the procedure copies away to confirm their status and the source collection for the comparison. She did not promise that tidying the collection would fix the organisation’s wider record of reversals. That work belonged to the service process. The trial could make its limits visible.
Maya looked again at the sentence that had started the meeting: The customer qualifies for a service credit.
She could now explain how such a sentence might be produced, why it could sound certain while relying on the wrong version, and what a source link did and did not prove. She did not have to become an engineer to make the next decision. She had to know what the proposed use asked the system to do, what evidence an answer could provide and where a person still held authority.
She authorised the team to prepare the comparison within the boundary already agreed. The experiment itself would test whether either route made those checks practical.
Reopen your Demonstration Question Sheet in Appendix C. Read each answer and citation again. Ask what the source establishes, whether it was current, what the answer added, and which facts still required a person to check them.
Chapter 6, The Bounded Experiment, asks how to test those questions under clear limits, and what decision the trial must produce when it ends.
