Chapter 10
When the Assumptions Break
At 5.02 on Friday afternoon, Daniel moved his hand away from the disable control and called Maya.
The service-credit procedure had changed that morning. The edit was marked as minor. It altered a sentence about when an adviser should consult a team lead. Nobody had notified Daniel or Leila before the changed document became available to the retrieval feature.
One saved question still produced the answer they expected. Another produced an answer that could lead an adviser to skip a referral.
Daniel could disable the feature for the small live group. He could not tell from those two questions whether anyone had seen the changed answer during the day.
Maya listened, then asked Leila what she knew about the edit.
“Not enough. I have the earlier version and the new one. I don’t yet know who approved the wording.”
“Can advisers still reach the procedure without the feature?”
“Yes. I can direct them to the last approved working copy and the team lead.”
“Then pause retrieval for this use,” Maya said. “Keep the ordinary service route open.”
Daniel clicked the control. He verified with an ordinary adviser account that the feature was unavailable to the group. He recorded the time. Leila told the team leads which procedure copy to use and asked them to hold any case whose approval route was unclear.
The feature had been live that morning. By 5.09, it was not.
They still did not know whether the edit was wrong.
Stop the uncertain part
Maya joined Daniel and Leila in the service meeting room. Nadia came in with the monitoring review from earlier that day. Priya was finishing a call.
Daniel showed them the two versions of the sentence. The earlier version told an adviser to consult a team lead before offering a credit in a stated circumstance. The new version could be read as requiring consultation only after the credit was offered.
“The difference is small on the page,” Daniel said.
“Not in a call,” Leila replied.
Nadia asked whether the new sentence was intended to change the rule or merely clarify its timing. Leila could not yet say. The document had been edited by someone with permission to update the knowledge base. Permission to edit a page did not establish authority to change an approval rule.
They used the Rescue Guide to separate the immediate response from the investigation. Daniel had contained the retrieval route. Leila had provided a service fallback. Next they needed to preserve the old and new source versions, the outputs produced from them, the access and change history, and the affected case window. Then they could establish what had happened and decide what could resume.
Which part of your workflow could you pause while keeping an approved route open?
Guidance
Choose a specific failure that would make the current output doubtful. Identify the affected route and the working alternative people would need immediately. Consider which source versions and outputs must be preserved before anyone attempts a repair.
Your answers are saved only in this browser on this device. Nothing is sent anywhere.
Daniel was relieved that Maya had not asked him to switch off the whole knowledge base. Advisers still needed approved procedures. The problem was the unreviewed change flowing into a feature that could turn a sentence into a direct answer. They had stopped that route while the source status was uncertain.
Stop the path you cannot defend. Keep the sound path available.
Leila sent a short instruction to the limited group. The service-credit procedure search was paused while a source update was checked. Advisers should use the identified approved working copy and the existing team lead referral. They should not rely on saved assistant answers or screen captures for current guidance. If a customer was waiting on an approval question, they should give the customer a clear next step and time for a reply.
She did not say that the system had made wrong decisions. They had no evidence of that. She did not call the interruption routine maintenance. Staff needed a truthful instruction they could use.
Priya read it before Leila sent it.
“Add where to report a case already handled today,” she said.
Leila added that line. An adviser who had used the feature on an approval question should tell a team lead and provide the case reference through the normal service route. They did not need to decide whether the output had caused an error before raising it.
What had actually changed
At 8.34 on Monday morning, Leila found the person who had edited the procedure.
The editor was a service manager outside her immediate team. She had been asked to make a passage easier to read after an adviser complained that the old wording was awkward. She believed she had clarified the sequence, not changed the approval requirement. The knowledge base had allowed her to publish the edit as minor. She had not known that the retrieval feature would use the revised text as soon as it appeared.
Leila showed her the two sentences without the assistant’s answer.
The manager read them and stopped.
“I can see the other reading now,” she said.
There was no need to make her the cause of everything that followed. Her edit exposed a gap in the organisation’s change route. A person could publish a procedural clarification that affected an approval instruction without the procedure owner checking it. The retrieval feature then made that wording easier to encounter and easier to mistake for a settled instruction.
Leila asked who had approved the original rule. She traced the procedure’s owner and the approval record. The earlier sentence reflected the approved requirement. No approval existed for changing the point at which the team lead became involved. The new sentence was therefore not an authorised change to the rule.
Daniel put the findings in the Diagnosis Record. He separated observations from interpretations. The edit time and document versions were observations. The absence of approval for a changed rule was established from the procedure record. Whether a particular adviser had relied on an affected answer was still open.
He also tested the feature against both versions in a restricted view. With the earlier source, the saved question directed an adviser to consult the team lead before offering the credit. With the edited source, one wording of the question produced an answer that placed consultation later. Other questions still returned the correct sequence.
The behaviour explained why a quick check on Friday had not settled the matter. The feature did not fail consistently. It responded to the wording of the question and the passage it retrieved. A single sound output could sit beside an unsafe one.
Nadia asked Daniel to leave the Friday outputs in the record, including the answer that looked reasonable.
“If we show only the bad one, it looks as though the problem was obvious whenever anyone used it,” she said. “It wasn’t.”
The record now showed the condition under which the answer changed. That mattered for identifying affected work and for testing a repair.
The cases inside the window
Daniel could identify when the edited procedure entered the feature’s source collection and when he disabled access. He could also identify questions asked during that interval under the access and logging settings already explained to staff.
He did not send Leila a list of every adviser question and ask her to read it as a performance record. They limited the review to questions that might involve the changed approval passage. Daniel first identified the relevant outputs and source links. Leila and the team leads then matched those interactions to cases through the authorised service process.
Two cases needed a closer look.
In the first, an adviser had opened the source passage after receiving an answer. The case note showed that she referred the proposed credit to a team lead before speaking to the customer. The approval route had held. Leila confirmed that with the team lead rather than assuming the source-opening record proved it.
In the second, the adviser had asked whether she could tell a customer that a credit would be applied. The feature’s answer used the new wording. The case note said the adviser would “confirm with the team lead”, but the customer had already been told the credit was likely. There was no record of a credit being applied or a final promise. The adviser was away that morning.
Leila asked the team lead to review the call and speak to the adviser when she returned. The customer would receive a clear answer through the normal route. Until then, the team could not describe the case as unaffected.
How far to look
Maya asked whether they should contact every customer whose case had involved a service credit that day.
“Not every case was near this instruction,” Leila said. “We should check the relevant cases first and contact anyone whose information or outcome needs correcting.”
Nadia agreed with the scope but wanted the limit recorded. They could identify the interactions visible in the feature’s records and the service cases they matched. That did not prove no one had read a colleague’s screen or used a saved answer. The team leads would ask the limited group about those possibilities. They would not claim perfect visibility.
The second case went into the AI Incident Log with its status open. The source edit, affected output, possible customer communication, containment and case review were linked in the entry. The earlier live quality reports remained separate. This event had a different cause and a possible effect on a customer.
By late afternoon, the adviser confirmed what had happened. She had read the answer, checked the source briefly and told the customer the credit appeared likely, subject to confirmation. She then referred the case to the team lead. She had not skipped approval, but she had given the customer an expectation before the required check.
Leila listened to the call. The adviser had not said the credit was approved. The distinction mattered. The team lead completed the review and contacted the customer with the decision and its basis. Leila recorded the correction to the conversation without describing it as a completed wrong credit.
The incident was real even though no unauthorised payment had occurred. An unapproved wording change had reached the feature. The feature had produced a misleading answer. An adviser had partly relied on it when speaking to a customer. The approval control had still prevented a credit from being applied without review.
The absence of the worst outcome does not erase the failure that nearly reached it.
Fix, narrow, retrain or stop
On Tuesday, Maya called a decision meeting.
Daniel had a technical proposal. He could restore the approved wording, remove the edited version from the source collection and test the saved questions before switching the feature back on. He also wanted a change notice whenever a service-credit procedure was updated.
Leila agreed with restoring the source. She did not think the previous boundary was enough on its own. The feature had been allowed to answer questions about approval steps. Those answers could sound like instructions to the adviser, even though the feature held only Assist authority.
Priya thought training needed attention. The adviser in the second case had opened the source, as instructed, but had not held the customer communication until the team lead confirmed the decision. A reminder to “check the source” would not solve that part.
Nadia wanted them to consider stopping the feature altogether. Not because one source change proved the route unusable, but because the proposed repair depended on a change process they had not demonstrated. If someone could publish another approval edit tomorrow without Leila’s review, a corrected answer today would have a short life.
Maya wrote four choices on the board: fix, narrow, retrain, stop.
They discussed each as a real decision, not a progression toward the answer Daniel preferred.
A fix to the source and configuration could address the known sentence and question. It would not by itself close the publication gap.
Narrowing could keep the feature useful for locating a procedure while withholding generated explanations of approval and eligibility instructions. An adviser could open the source directly for those questions. That would reduce the value of the feature in some cases and make the boundary easier to defend.
Retraining staff could reinforce when to pause a customer conversation and refer a case. It could not make an unapproved procedure current or guarantee that a plausible answer would be checked every time.
Stopping the use would remove the feature from service-credit work and return advisers to ordinary search and escalation. That route remained available. The team would lose a possible aid to finding procedures, but they would not lose the service process.
What Leila could own
Maya asked Leila which use she could own that week.
“I can own a narrower one,” Leila said. “Finding the procedure and showing its status. I am not ready to let it explain approval steps until the source change route is working.”
Daniel said he could configure that. For questions near approval and eligibility, the feature could direct advisers to the source rather than compose a case-facing instruction. He would test whether that behaviour held across the questions Nadia and Priya had collected.
Nadia did not call the proposal safe. She asked what would prove the narrower use worked, who would check the next source update, and how long the restriction would last before review.
What would you need to see before allowing a narrower use to resume?
Guidance
Start with the failure that prompted the pause. Describe an observable test of the proposed boundary and identify who can judge it. Include the next ordinary change that would show whether the repair holds beyond the saved test questions.
They set conditions. Leila would restore and approve the procedure text and establish who could publish changes to that source. Daniel would show that the feature did not turn approval passages into an instruction under the narrowed configuration. Priya and a team lead would test questions in the language advisers used. Nadia would review the failures as well as the successes. Maya would make a fresh decision before access resumed.
Until then, the pause stayed in place.
FIG-10.1
The choices after the source change
| Choice | What it would address | What would remain |
|---|---|---|
| Fix the source and retrieval setup | The known wording and affected answers | Future unreviewed source changes |
| Narrow the use | The chance of a generated approval instruction being treated as guidance | The need for advisers to check source and case facts |
| Retrain advisers | The point at which staff pause, verify and refer | An incorrect or unauthorised source |
| Stop the use | Exposure through this feature | The underlying procedure and recording work |
The table made one thing clear. No choice repaired every part of the failure. The organisation had to make changes to the source process as well as decide what the feature could do.
When help is useful
Daniel could configure and test the existing knowledge-base feature. Leila could establish the service procedure and approval route. Nadia could challenge the decision. Maya could authorise the use.
There were questions they might not be able to answer themselves.
Daniel could see the outputs and source versions available to their team, but he was less certain about how quickly a published source change became available across every part of the feature. He could test observed behaviour. He could not infer the entire update mechanism from a few examples.
Maya asked him to use When to Seek External Help to frame a narrow question for the system provider’s support team. They needed to know how source changes were taken up, whether earlier versions could remain available for retrieval, what controls existed for holding an update until review, and what evidence Daniel could retain when an answer changed. They did not send customer case material merely to ask those questions.
The response would inform the recovery decision. It would not replace it. Even a clear account of the product’s update process would not decide who within the organisation could approve a service-credit rule.
Leila also considered whether the approval wording itself needed specialist review. She could establish the internal authorisation for the current service procedure. If the proposed change had raised a question beyond the team’s authority or affected obligations to customers, she would seek appropriate advice before publishing it. The immediate task was simpler: restore the approved instruction and stop treating an unapproved edit as current.
Nadia asked why they had not raised the source issue with support during the original experiment.
“Because the problem then was visible and bounded,” Daniel said. “We knew which copies we had selected. This time the source changed under a live feature before our check.”
That answer went into the Diagnosis Record. The conditions had changed. The team’s knowledge of the system had to change with them.
A smaller return
By Thursday morning, Leila had restored the approved wording and the procedure owner had signed it off. The edit route now required her review before a change to eligibility, approval or escalation instructions could be published into the retrieval collection. Daniel tested the notice by working through a proposed change in a restricted copy. Leila received it, reviewed it and either approved or held it before it reached the live source set.
It was a modest repair. It depended on people following the route. Daniel and Leila would test it again when an ordinary update arose.
Daniel’s narrowed configuration showed source passages and status for approval questions without drafting an outcome. On other service-credit procedure questions, it could still provide a short explanation with a source link. Priya and the team lead tried the saved questions, the wording from the affected case, and several ordinary adviser phrasings. Nadia added a question that joined a reversal with a missed appointment.
One answer was awkward. It displayed two sources without saying which governed the case. Priya could see both and use the procedure directly. Nadia asked whether that was acceptable under the narrowed use.
“For this question, perhaps,” Priya said. “It does not pretend to settle it.”
Leila wanted the screen to make the need for a person to establish the case facts clear. Daniel adjusted the display and repeated the test. They kept the awkward first result in the record.
Maya reviewed the evidence on Friday. She authorised the feature to resume for the same small group under the narrower conditions. Approval and eligibility questions would lead to source material, not a generated case outcome. Existing team lead approval remained unchanged. Leila would check the next source update before Daniel let it reach the feature. Any repeat of an unreviewed approval change would bring the release back to Maya immediately.
She did not call the incident closed. The customer case had been resolved. The source had been corrected. The feature was operating within a narrower boundary. The investigation and decision were recorded. The team would still watch whether the change route held.
Daniel re-enabled access and checked it with an ordinary adviser account. Leila told the group exactly what had changed and what had not. They could use the feature to find the procedure. If it showed an approval passage, they needed to read the source and follow the team lead route. Saved answers from before the pause were not current guidance.
Priya took two questions from advisers in the first hour. One wanted to know whether they had done something wrong by using the feature on Friday. The other wanted to know whether it was worth using now that approval explanations had been narrowed.
“No, the edit reached you before we checked it,” Priya told the first. “If an answer affected a case, tell us so we can review it.”
To the second, she said, “Try it where it helps you find the source. If direct search is clearer, use that. Tell us which happens.”
A smaller return was still a return. The feature had to earn its place within the new boundary.
The work left after recovery
On Monday, Nadia read the incident entry and the Diagnosis Record together.
The immediate cause was an unapproved wording change made available to the retrieval feature. The contributing conditions were the ability to publish it as a minor edit, the lack of notice to Leila and Daniel, and the feature’s ability to turn a changed passage into a plausible instruction. The adviser had checked the source, but the source itself carried the disputed wording. The ordinary team lead approval had prevented an unauthorised credit.
The response had contained the use, examined the relevant cases, restored the approved source, narrowed the feature and changed the publication route. Those were observations. Whether the new route would hold through routine workload remained to be seen.
Nadia asked Maya whether she would have made the same resumption decision if the team could not identify the source versions used during the day.
“No,” Maya said. “We would not know what we were recovering from.”
That was the lesson Maya wanted recorded. Recovery was not just a switch back to the last configuration that seemed to work. They had to know what failed, which work might have been affected, which authority remained valid and what evidence supported the new boundary.
Leila looked at the time spent on the incident. Daniel had investigated the feature. Team leads had reviewed cases. Priya had handled questions and helped test the narrowed use. Nadia had challenged the conclusions. Maya had made two decisions, to pause and to resume.
None of that appeared on the knowledge-base subscription line.
Maya knew what the CFO would ask when they brought the service-credit use forward for a wider decision. It would be reasonable to ask. The recovery had preserved a limited use, but it had also revealed what maintaining that use required.
Use the Rescue Guide to contain the uncertain path and keep an approved fallback working. Create a Diagnosis Record that separates what happened from what you infer. Record live effects and near misses in the AI Incident Log. Use When to Seek External Help when a question about the system or the governing decision exceeds the team’s evidence or authority.
Chapter 11, The Whole Cost, asks what the feature costs when the people who prepare, check, correct and recover it are included in the number.
