Chapter 6
The Bounded Experiment
At 8.16 on Thursday morning, Priya found the same service-credit question answered two different ways.
She had not asked two different questions. She had changed one word in the description of the case. The first answer said the customer qualified for a credit. The second said the adviser should seek approval before offering one.
Both answers displayed a source.
Priya put the two screens beside each other and called Leila.
“Which one would you use?” Leila asked.
“Neither yet.”
That was the answer Leila wanted to hear.
The team had until 11 September to compare the retrieval feature already in their knowledge base with the product they had seen demonstrated. They were trying to find out whether either could help an adviser locate and check the right procedure for a service-credit case. They were not giving either system permission to decide whether a customer received a credit.
It was a small distinction on paper. At a busy service desk, it could disappear in a sentence.
Priya copied the question, the two answers and the displayed sources into the test record. She then opened the current procedure herself. The difference was not a matter of wording. One answer had drawn on the procedure dated 11 June 2025. The other had drawn on the version dated 11 December 2025.
They already knew those two procedures conflicted. They had found that while diagnosing the service-credit process. What they did not yet know was whether the retrieval feature could reliably keep the old instruction out of an adviser’s path.
Leila looked at the screen.
“The source makes the mistake visible,” she said. “But only if Priya has time to inspect it.”
Priya had spent four minutes finding the problem. A real adviser might have four calls waiting.
That observation went into the test record as well.
The question before the trial
Maya had asked Daniel to prepare the comparison after the route decision. He brought a draft plan to a short meeting with Leila, Priya and Nadia.
At the top he had written:
Which tool gives the best answers to service-credit questions?
Nadia read it twice.
“That question is too large,” she said. “What does ‘best’ mean? And are we testing a tool or a repaired procedure?”
Daniel had expected the objection. He had not yet found a better sentence.
Leila tried one.
Can an adviser find the current service-credit procedure quickly enough to use it, see where an answer came from, and recognise when the material does not settle the case?
Priya added another condition. The tool must not quietly present the June procedure as current. Daniel wrote that beneath the question.
Maya crossed out “best answers”.
The change mattered. A trial can produce a favourable result while failing to answer the decision that justified it. A fluent answer might impress a room. A fast answer might reduce a measured task time. Neither would help if an adviser could not tell whether the procedure was current, or if the system treated a gap in the rules as permission to guess.
Test the decision you need to make, not the performance that is easiest to display.
The question also kept the two routes in their proper place. The existing knowledge-base feature and the demonstrated product were candidates. The team was not committed to buying either. The small internal build remained deferred because support for it was unresolved.
If both candidates failed, Maya could choose neither.
That was written into the plan before anyone ran a test.
Daniel had used the Pilot Plan Template to put the question, limits and decision on one page. It took longer than he expected. The difficult part was not describing what the products could do. It was describing what the organisation would allow them to do for this comparison.
No customer was to be contacted because of an output. No service-credit decision was to be made from an output. No customer record was to be entered into a product merely to make the test feel realistic. Priya and Leila would use prepared cases drawn from the kinds of problems they had already examined, with identifying details removed or replaced. The source documents would be copies selected for the trial, with their status clearly marked for the test team.
Daniel would configure each route and record what he changed. He could disable access if a boundary was crossed. Leila would decide whether the results were useful for customer operations and whether the source set represented the work. Nadia would challenge the interpretation of the results. Maya would make the end decision.
The roles were not a courtesy list. Each person could stop a different kind of mistake.
What goes inside the boundary
The first version of Daniel’s plan said internal service-credit cases only.
Maya sent it back.
“Internal to whom?” she asked.
The answer was less obvious than Daniel had assumed. The cases began in customer conversations. Notes about them sat inside the organisation. Some included names, account details and explanations of hardship. Calling them internal did not make those details suitable for a comparison with a new product.
They sat down with the Data Readiness Checklist. Daniel wanted to know whether the documents were available and could be searched. Leila wanted to know whether they were correct. Maya wanted to know where a copy would go, who could see it, and what would happen to it when the trial ended.
Those were different questions. All of them belonged in the same experiment.
Leila selected the procedures and the supporting material. The June and December versions both remained in the test collection. Removing the old version would have made the trial easier, but it would have hidden the problem they needed to test. The June procedure was marked as superseded in the prepared source set. The team needed to see whether each route respected that status and showed it clearly enough to an adviser.
Priya helped turn past work into test cases. She removed names and account numbers. She changed incidental details that might identify a customer. She kept the features that made the decision difficult: which credit had been requested, what had already been promised, and whether an earlier decision had been reversed.
She did not copy a customer’s whole history into the test merely because it was available.
The prepared cases were useful, but they were not the same as live work. People in a real call leave facts out. They remember the sequence differently. An adviser may have to resolve the procedure and calm a customer at once. The plan acknowledged that the comparison would test retrieval against prepared cases. It would not prove how the tool behaved across a working day.
Nadia underlined that sentence.
“We will be tempted to forget it if the result is tidy,” she said.
FIG-6.1
The service-credit comparison boundary
| Inside the comparison | Outside the comparison |
|---|---|
| Prepared, de-identified service-credit cases | Live customer conversations and identifiable records |
| Selected procedure copies with their status recorded | Unreviewed documents added by convenience |
| Adviser questions about locating and checking a procedure | Automated credit decisions or customer messages |
| Recorded outputs reviewed by the test team | Outputs treated as approved advice |
| A decision by 11 September on what to do next | An automatic move to live use |
FIG-6.2
The bounded experiment loop
1
Question
One question the decision actually needs answered.
2
Boundary
What may be used, who may use it, and what is off limits.
3
Trial
Recorded observations inside the boundary, including failures.
4
End decision
A named decision-maker decides: continue, change or stop.
Then: the end decision sets up the next question - or closes the work.
The boundary was specific enough for Daniel to configure access. It was also specific enough for Priya to say no if somebody offered a “more realistic” case that contained a customer’s details.
A boundary that exists only in a meeting depends on everyone remembering the meeting. Daniel put the authorised source set and access list beside the test plan. Leila checked the source copies. Priya checked the cases. Maya approved the comparison on that basis.
When a sales director asked whether the team could add a tender question while the product was available, Daniel declined. The tender trial was still closed. Its documents, users and decision would require their own authority.
The comparison stayed about service credits.
A screen is not a result
On Monday, Daniel set up the first test session. He had made the same selected source material available to both routes as far as their different configurations allowed. He recorded where they differed, including how each handled document dates and status.
Priya sat at a desk with a list of prepared questions. She was not asked to score which answer she liked. For each route, she had to try to complete the adviser’s task: find the applicable instruction, inspect the source, and say what she would still need to establish before a credit decision.
Leila watched without helping. She knew where the answer was in the procedure, which made staying quiet difficult.
The first case was straightforward. Both routes found the December procedure. Both showed a usable passage. Priya could verify the instruction without much effort.
The second case described a reversed decision. The customer had first been refused a credit and then granted one after review. That history mattered because a later adviser needed to know what had happened, not merely whether a payment had been made.
The knowledge-base feature found a passage about recording the outcome. The demonstrated product produced a fuller answer about the credit. Priya initially preferred the fuller answer.
Then she looked for the reversal in its cited material. It was not there.
The product had given a plausible account of what an adviser might do. It had not established what the organisation had done in that case. Neither route could repair the missing record.
Priya changed her note from “useful answer” to “useful lead; record gap remains”.
Leila saw the distinction immediately. Their earlier diagnosis had shown that the company’s memory failed after a payment succeeded. A retrieval tool could expose that gap. It could also cover it with fluent prose.
An answer cannot recover a fact the organisation failed to keep.
They continued. Some cases had a clear current instruction. Others turned on whether a fact in the case was known at all. Nadia had deliberately included questions that sounded complete but lacked the detail needed to decide them.
When a route said it could not establish the answer from the selected material, Priya marked that as a useful result. Daniel had initially left no place for it on the test sheet. He added one.
The purpose was not to reward refusal. A tool that declined every hard question would not help the team. The purpose was to distinguish a supported answer from a confident completion of a gap.
By the end of the session, the record contained examples of both.
Who gets to challenge it
Nadia was not assigned to find fault with every output. Her job was to challenge what the team concluded from them.
She began with the cases.
“Who chose the wording?” she asked.
Priya had, with Leila’s review.
“Then we should say so. You know the procedure. You may be asking in a way that helps the tools.”
They took several cases back to advisers who had not helped write the questions. The advisers described what they would actually ask while searching for an instruction. Some used the customer’s language. Some searched for the exception. One asked about a promise made by another adviser without naming service credits at all.
The changed wording affected what the routes retrieved. It did not reverse every result, but it made the comparison less comfortable.
Nadia then challenged the source set. Leila had selected it because she knew the work. That was necessary. It also meant she might have left out documents that advisers often found through the ordinary knowledge base.
They checked the material advisers commonly opened for service-credit work and noted where it sat outside the selected collection. Daniel did not quietly add everything. A larger collection could change the results and create new problems with document status. The team recorded the missing material and decided which additions belonged in the comparison.
The challenge was aimed at the experiment, not just the software.
An experiment can fail because a route cannot do the job. It can also fail because the test has made the job unrealistically easy.
What might make your test easier than the work it is meant to represent?
Guidance
Look at who wrote the questions and selected the source material. Consider a variation from someone unfamiliar with the test, while keeping any change within the agreed boundary.
Your answers are saved only in this browser on this device. Nothing is sent anywhere.
Maya asked Nadia what would count as enough evidence.
“I don’t want a number that pretends to settle this,” Nadia said. “I want to see the cases where it worked, where it failed, and what changed when we made the question less helpful.”
Maya agreed. The end decision would use recorded observations, including failures. It would not be a vote on the better demonstration.
The old procedure returns
The Thursday result on Priya’s screen was not the first sign of trouble. It was the clearest.
One route had surfaced the June procedure as though its instruction could still be used. The source display gave Priya enough information to find the date. But the answer itself did not warn her that the instruction had been superseded.
Daniel checked the configuration. The selected copy carried a status marker. The route had retrieved the passage without making that marker useful in the answer. When Priya changed one word, it retrieved the December procedure instead.
Daniel could adjust the way the documents were presented. He asked for time to do that and rerun the affected cases.
Nadia asked him to leave the first result in the record.
“I’m not deleting it,” Daniel said.
“I know. But once the fix works, people will remember the fix.”
They recorded the original question, the altered question, both outputs, the source displays, the configuration and the date of the change. Daniel then adjusted the document handling and repeated the tests. The route was more consistent in that set of cases.
That was progress. It did not erase the first failure.
Leila also asked a harder question. Why was the June procedure still available in a place where it could look authoritative? The team had known for some time that the two versions conflicted and that no supersession notice had been issued when the December version appeared. A better retrieval setting might prevent the old text from appearing in this trial. It would not by itself make the organisation’s procedure history clear to every adviser.
The comparison was showing two kinds of work. One belonged to the route they might choose. The other belonged to the organisation that supplied the route with material.
Maya wrote both on the board.
Document control.
Retrieval behaviour.
“We need both,” she said.
Daniel looked at the revised output. It was now easier to see which procedure applied. He could have presented that as a successful fix. Instead, he added a condition to the end decision: Leila’s team would need an owned method for marking a procedure as current or superseded before any route reached live advisers.
Do not ask a tool to settle an authority dispute its owners have left unresolved.
That condition was not an excuse to abandon the comparison. It was part of learning from it.
The end is part of the beginning
As 11 September approached, Daniel prepared a summary of the results. He began with a table of cases and outputs. Maya asked him to put the decision choices above it.
The plan had named three possible end decisions: stop; extend the bounded comparison for a stated question and period; or authorise preparation for a limited live use, subject to a separate decision. There was no fourth choice called “the pilot went well, so leave it running”.
What decision will end your trial?
Guidance
Write the available choices before looking at the results. Name who can make that decision and distinguish preparation for a next stage from permission to begin it.
That mattered because access had already been configured. People had spent time learning the routes. A product representative had asked when more users could be invited. None of those things changed the authority granted for the comparison.
The Use-Case Screening Sheet helped Maya test whether the proposed next use was still the same kind of use. In the comparison, prepared questions were answered for reviewers. If real advisers began using a route during customer work, the data, pace and consequences would change. The system would still be assisting, but its output would sit much closer to a customer decision. A trial result alone could not authorise that change.
Four readings of one result
Maya asked each person for a view before she read Daniel’s recommendation.
Priya favoured the route in the existing knowledge base. She found it easier to move from the answer to the source she needed. She also thought advisers would recognise the screen. She was clear that she had tested prepared cases at a desk, not live calls.
Daniel thought the existing route was supportable with the team they had. The demonstrated product had useful features, but those advantages had not answered enough of the service-credit question to justify the extra work of introducing it. He still needed to confirm the knowledge base’s behaviour when the source collection changed.
Leila wanted the procedure status fixed before any adviser relied on the route. She also wanted the reversed-decision record repaired in the underlying process. Otherwise the assistant would become a convenient place for people to look for an event nobody had recorded.
Nadia agreed that one route had emerged as the smaller candidate. She did not agree that the comparison had proved it was safe in live service work. Several questions from other advisers had produced weaker results than the prepared questions. The team needed training and a way for staff to report a doubtful answer without having to settle it themselves.
Maya listened. Nobody had asked her to choose a product for the whole organisation. She had to decide what this evidence allowed next.
She authorised preparation for a limited live use of the existing knowledge-base retrieval feature. That was not permission to turn it on for advisers that afternoon. It was permission to complete the source work, specify the safeguards and bring a go-live decision back to her. The demonstrated product would not proceed for this service-credit use. The internal build remained deferred.
The decision record stated why. It also stated what the comparison had not established.
The team had evidence that an adviser could sometimes find and check the current procedure faster with the existing feature. They had seen failures when a superseded source was presented without enough warning. They had found that question wording and missing records changed the result. They had not measured performance in live calls, established a reliable time saving, or tested every kind of service-credit case.
The bounded comparison ended on 11 September as planned.
It ended with a decision, not a claim of certainty.
The copies left behind
At 4.42 that afternoon, Daniel opened the list of trial material.
The result meeting had finished. The access and copied data still existed.
He had recorded which prepared cases had been placed in each route, which procedure copies had been uploaded or indexed, who could see the trial workspace, and where outputs had been saved for review. The plan said what to do at the end. Now he had to do it.
Daniel removed the trial access that was no longer required. He followed the recorded deletion steps for the prepared case copies in the demonstrated product and checked what evidence of that action he could retain. He separated the test observations needed for the decision from unnecessary copies of case material. The organisation kept the decision record and the specific failure examples required to explain it, under its own access controls.
Maya asked whether deletion meant there was no trace of the comparison.
“No,” Daniel said. “It means we keep the evidence we need and stop leaving working copies wherever we tested.”
The distinction had been in the Pilot Plan Template from the start. During a busy week, it was easy to treat disposal as administration to be completed later. But the experiment had put material into particular systems and given people access for a particular purpose. When that purpose ended, those permissions needed attention.
Leila checked that the source corrections would continue in the organisation’s normal knowledge base work. Removing a trial copy would not remove the obligation to repair the original procedure status. She took ownership of the supersession notice and the record of reversed decisions.
Priya kept one short list for her team: cases where the tool helped, cases where it looked helpful but the source was wrong or incomplete, and questions advisers would need to ask before relying on an answer. She had begun the trial wanting to know which screen would be quickest. She now wanted to know whether an adviser could slow down at the right moment.
That was a more useful question for the next stage.
What the comparison earned
The trial had not transformed service-credit work. It had narrowed a decision.
It showed that a route already available to the organisation might help advisers locate a procedure. It exposed a source-status failure that a polished answer could have concealed. It showed where the organisation’s own records were too weak for retrieval to help. It gave Maya a reason to set aside the demonstrated product for this use without claiming that the product was poor in every setting.
It also left work unfinished.
A source needed an owner and a clear status. An adviser needed to know how to check an output. Someone needed to decide what to do when the tool could not establish an answer. The organisation needed a way to notice if a source change altered the route’s behaviour after go-live.
Those were conditions for the next decision. They were not reasons to pretend the experiment had failed. A bounded experiment earns its place when it changes what the organisation can responsibly decide.
Maya closed the comparison record and looked again at the authorisation for preparation. The scope was still narrow: service-credit procedure retrieval, advisers retaining the decision, no automatic customer action. She could explain both why they were proceeding and why they were stopping short.
That was enough for Friday.
Plan your own bounded experiment before a trial begins. Use the Pilot Plan Template to state the question, boundary, challenger, end decision and treatment of the data. Use the Use-Case Screening Sheet to check whether the next proposed use changes the authority or consequence of the output. Use the Data Readiness Checklist before moving source material or cases into a test.
Chapter 7, What Could Go Wrong, asks what must be protected before the retrieval feature reaches real advisers.
