Contents

Chapter 9

Watching It Work

At 8.47 on Wednesday morning, Leila put two sheets of paper in front of Maya.

One showed the first doubtful answer reported by an adviser. The other showed no reports at all from the rest of the group.

“Which one worries you?” Leila asked.

Maya looked at the empty sheet.

The reported answer was still being investigated. It had joined an instruction about a missed appointment to a sentence from a different service procedure. The adviser had checked the source, used the correct passage and reported what she saw. No customer decision had been affected.

The empty sheet told them less. It might mean the other advisers had found nothing wrong. It might mean they had scarcely used the feature. They might have seen doubtful answers and dealt with them without reporting. Leila could not distinguish those possibilities from the report count.

Maya had asked her to bring a pattern, not just incidents. Leila had come to ask what would count as one.

They opened the Monitoring Plan.

What they meant by watching

Daniel had prepared a first page showing whether the feature was available and how often people opened it. Maya read it and set it beside the doubtful answer.

“Can you tell whether anyone followed this answer?” she asked.

“No.”

“Can you tell whether they opened the source?”

“For some interactions, yes. Not whether they read it.”

“Can you tell whether the case went to a team lead?”

“That is in the case system.”

The information they needed sat in different places. An activity report from the knowledge base could show questions and source openings within the access and logging settings they had confirmed. It could not show that an adviser understood a passage or made the right customer decision. Case records and team lead reviews could show parts of the service outcome. They could not always show whether the retrieval feature helped.

Leila suggested that they watch four things. The quality of answers and sources. The time and effort spent using and supporting the feature. The way advisers used it, including uses outside the agreed boundary. And customer complaints or staff reports that might reveal a problem.

Daniel added availability. If the feature was down, usage would fall for an obvious reason.

Nadia asked who would look at each part. A plan that said monitor quality without assigning a person would produce a report and no decision.

Priya could review a small sample of live questions with Leila and the team leads. Daniel could check availability, access and changes to the source collection. Leila could examine case outcomes, reports and complaints. Nadia could challenge the conclusions they drew from those views. Maya would receive a short review and decide whether to continue, narrow, pause or widen the authorised use.

The review would be frequent while the group was small and the feature new. A team lead would look at flagged cases as they arrived. Daniel and Leila would check the first controlled source change before leaving it in place. Leila would bring a weekly view to Maya, including what they could not yet establish.

Maya did not ask for a dashboard. She asked for a way to notice when the conditions of her release decision no longer held.

Watch the conditions of the decision, not only the activity of the tool.

The feature had been authorised to assist with service-credit procedure retrieval. That meant a rise in questions would not automatically be good. A fall would not automatically be bad. What mattered was whether advisers could find and check current guidance while keeping customer decisions and approvals with the right people.

The change that was waiting

Daniel had prepared a controlled change for the missed-appointment answer. He and Leila had traced the sentence that did not belong. Two procedures used similar language for different service failures. The source collection showed both as current, but the feature sometimes brought their passages together when a question was broad.

Leila did not want to remove a valid procedure simply because one answer had joined them. Daniel adjusted how the two documents were identified and separated for retrieval. He kept the original question and output. Nadia added variations of the question that an adviser might ask during a call.

They tested the change in the restricted view first. The original question now returned the relevant missed-appointment procedure without the stray sentence. Several variations did too. One broad question still brought up both procedures, but displayed them as separate sources rather than making a single instruction from them.

“Is that a pass?” Daniel asked.

“It is an improvement we can describe,” Nadia said.

She wanted the record to say exactly which questions they had tried and what remained open. The change had corrected a known failure. It had not established that every future question would separate the procedures.

Leila signed off the source status. Daniel applied the change for the small live group at a time when a team lead and Priya could check it. He confirmed the feature still showed the current service-credit procedure, the supersession status for the old June procedure, and a clear failure message if retrieval was unavailable.

For the rest of the week, their quality review included questions near the one that had failed. Priya asked advisers to flag an answer that combined procedures even if they caught the problem before taking any action. A caught error was evidence about the feature. It was also evidence that the check had worked.

Daniel kept the pre-change and post-change outputs together. If a later review showed another mixed answer, the team would need to know whether it was the same failure returning or a different one.

A count with the wrong name

At 3.12 on Thursday, Daniel showed Leila a usage report.

Two advisers accounted for most of the questions submitted since release. Several others had opened the feature only once. One had not opened it after training.

Daniel thought the uneven use was normal. Priya and one other experienced adviser had been involved in preparing the test cases. They knew what to ask. The others would catch up.

Leila was less sure. If the feature was useful only to people who already knew the procedures well, it might be helping the wrong part of the team.

They asked Priya to speak to advisers during the next shift. She did not ask why they were failing to use the feature. She asked how they had found procedures in recent service-credit cases.

One adviser said the ordinary search took her directly to the document. Reading an answer first felt like an extra step. Another had tried a question during a busy call, received a long response and returned to her saved procedure link. A third had been uncertain about whether a reversed-decision question contained too much case detail, so she had avoided the box.

The adviser who had not opened it after training had handled no service-credit cases that week.

The usage report had put them all in the same group: low users. Their reasons were different.

What might your usage figures be grouping together?

Guidance

If your report groups people as low users, choose one such group. Consider whether they had work suited to the tool, found another route clearer, or were unsure what they could enter. Note what you would need to ask before deciding that more use would help.

Your answers are saved only in this browser on this device. Nothing is sent anywhere.

Priya took those findings to Leila. The first adviser did not need to be persuaded to add a step. The second might benefit from practice asking a narrower procedure question, but the tool also needed to earn its place against ordinary search. The third needed the input rule explained through examples. The last needed no intervention.

Nadia challenged Daniel’s earlier reading.

“Some will catch up. Some have made a sensible choice. The count cannot tell us which.”

Daniel agreed. He changed the review so it showed use alongside the opportunities to use it, where the team could establish those opportunities. They would not infer that every adviser should ask a question each day. They would look at service-credit work and ask whether the feature had helped in the cases where someone chose it.

A usage count describes a behaviour. It does not explain it.

Leila also wanted to know whether the boundary was holding. Daniel could see questions about service credits and a few that were harder to classify. He did not want to inspect staff questions as a general performance exercise. The purpose of the review was to check the authorised use and find problems with the feature.

They agreed on a limited review with the advisers informed of what was being checked. If a question appeared to be outside service-credit procedure retrieval, Daniel would bring it to Leila for context. The first response might be clearer guidance. A repeated or consequential use outside the boundary would go to Maya before it became an accepted practice.

Minutes gained, minutes moved

On Friday, Priya brought Leila the Time-Savings Tracker.

An adviser had recorded that the feature saved her time on a difficult reversal case. She had found the current procedure quickly. The entry looked encouraging until Priya checked what happened after the search. The adviser had spent additional time reading the cited passage and confirming the missing approval in the case record. The customer had been called back later.

“Was time saved?” Leila asked.

“On finding the procedure, yes,” Priya said. “On finishing the case, I don’t know.”

They kept the entry. They changed its description.

A second adviser had recorded no saving because she already knew where to find the procedure. She had still used the feature to check that she had the current version. That check had value, but the tracker could not turn it into minutes without pretending to know how long an error might otherwise have taken to find.

A third entry showed a longer search with the feature than with the ordinary knowledge base. Priya had watched it happen. The adviser read a broad answer, opened two sources, then searched directly for the procedure title. The direct search settled the question.

Leila did not want the tracker used to prove a benefit the team had assumed at launch. She wanted it to show where time was spent and what work the feature displaced or added.

The subscription cost was already part of the organisation’s knowledge base. Daniel had spent time configuring the feature and investigating the doubtful answer. Priya had trained advisers and walked the floor. Leila and the team leads were reviewing sources and cases. Advisers were reading answers and checking passages. Those were costs even when no new invoice arrived.

Some of that work would reduce after the initial release. Some would continue as long as the feature remained in use. They could not know the steady amount yet.

The Time-Savings Tracker showed individual tasks, not a financial return. Leila told Maya that plainly.

“I can show where it shortened a search and where it added a check,” she said. “I cannot give Finance a defensible saving from this week.”

Maya accepted that. The CFO would want a number later. An early number that ignored training, correction and review would make that later conversation harder.

The team also watched customer effort. A quick procedure search might still lead to another contact if the case record lacked a required fact. A slower first call might avoid a wrong promise. They had too few cases to make a claim about either pattern, but they could record the sequence when it occurred.

Count the work that moves, not just the minutes that disappear from one step.

What earns an incident entry

The first live doubtful answer went into the AI Incident Log. The adviser had found a sentence from the wrong procedure, checked it before acting and reported it. The entry recorded the question, output, displayed sources, time, people involved, customer effect, immediate response and later controlled change. Its customer effect was recorded as none known, rather than left blank.

The incorrect approval answer found before release remained in the pre-release test record. It had informed training and the risk work. It was not rewritten as a live incident to make the log look fuller.

The routine access problem on launch day went to Daniel’s support record. It had prevented one authorised adviser from seeing the feature, but it had not produced an unreliable answer or affected a case. If access failures became frequent or exposed the wrong documents, the classification would change.

The missing reversal approval found during Monday’s customer call belonged in the service case record and Leila’s process work. The assistant had helped locate the procedure. The missing fact predated its use. The team linked the case to the monitoring review because the process weakness mattered to the outcome. They did not label every problem near the assistant an AI incident.

Priya asked about an answer that was awkward but correct. Leila would use it as feedback for improving the feature, with an incident entry if it created a material risk of misuse or affected a customer decision. They needed room to record ordinary friction without treating each inconvenience as a serious event.

That distinction cut both ways. An answer caught before harm could still earn an incident entry if it exposed a failure that might recur. The first live doubtful answer did. It mattered that the adviser had caught it. It also mattered that the tool had produced it.

FIG-9.1

Where the first live observations went

ObservationRecord and reasonPerson who follows it
Missed-appointment answer joined two proceduresAI Incident Log: unreliable output in live use, caught before a decisionDaniel investigates; Leila confirms sources
Adviser could not see authorised access on day oneSupport record: access setup faultDaniel corrects and checks permissions
Reversal approval absent from a customer caseCase record and process review: required fact was missingTeam lead resolves case; Leila repairs recording practice
Adviser finds ordinary search fasterTime-Savings Tracker and staff feedback: comparison of working routesPriya and Leila review usefulness
Question appears outside service-credit scopeMonitoring review: possible boundary driftLeila establishes context; Maya decides if authority must change
Source: author’s method and fictional case.

Nadia reviewed the distinctions. She agreed with most, then pointed to the access problem. If the same fault had shown an adviser material they should not see, it would need an incident entry, even if nobody acted on the material. Daniel added that condition to the Monitoring Plan.

Maya asked for the log to preserve the first report’s original output after the controlled change. Without it, a later reviewer might see only the corrected answer and wonder what the concern had been.

Silence after the announcement

By the following Tuesday, no second doubtful-output report had arrived.

Daniel thought the change might have solved the problem. Priya was pleased, but cautious. She had heard an adviser say, “It looked odd, so I just searched the old way.” The adviser had not filed a report.

Priya asked her what looked odd. The answer had brought up two valid procedures without making clear which one covered the case. The adviser had done the right thing by checking elsewhere. She had also seen an output that could help the team understand a remaining weakness.

“Would you report that next time?” Priya asked.

“I thought reports were for wrong answers.”

The announcement had asked staff to flag a doubtful answer, but the adviser had interpreted doubtful as something she could prove was wrong. Priya changed the next team reminder: report an answer that mixes procedures, hides a source, leaves you unsure which rule applies or appears to tell you to skip a check. You do not have to establish the cause.

Leila entered the example as feedback for review. She and Daniel examined the output before deciding whether it warranted an incident entry. It had not instructed the adviser to take a wrong action. It had left a material distinction unclear. Because the same confusion could affect another case, they recorded it in the AI Incident Log as a live quality issue, with no known customer effect.

Nadia asked whether the increase from one entry to two meant quality was falling.

“No,” Priya said. “We made it easier to tell us.”

That was the answer Nadia wanted on the record. Reports could rise because the feature was worse, because staff were using it more, or because they trusted the route for raising concerns. Reports could fall because quality improved, because use fell, or because staff stopped speaking up.

If doubtful-answer reports fell next week, what would you check?

Guidance

Begin with the explanations this passage leaves open. Look for evidence about how often people used the feature and whether they still felt able to report uncertainty. Decide whose account could help you interpret the count.

The count needed company.

They decided to pair the report view with selected output reviews, adviser conversations, team lead observations and usage context. It would still be incomplete. At least it would be less likely to mistake silence for assurance.

The review Maya could use

At the end of the second week, Leila brought Maya a short monitoring review.

It described the controlled source change and the questions used to test it. It included the two live quality entries, both caught before customer action. It showed that use was concentrated among a few advisers and why some others had chosen ordinary search. It separated time spent locating a procedure from time spent checking it and finishing the case. It noted the continued weakness in recording reversed decisions.

There was no single score.

Maya read the review and asked what Leila wanted to change.

“Keep the small group for now,” Leila said. “We need more ordinary cases and less help from Priya standing nearby. I want the team leads to review source checking in their normal case discussions. I also want Daniel to test each source update before it reaches this feature.”

Daniel agreed with the source test. He asked that Leila’s team notify him when an owned procedure was replaced or its status changed. He could check the retrieval behaviour, but he could not know that a procedure owner had quietly edited a document unless the change reached him.

Nadia suggested a trigger rather than a general promise to be careful. A source change to approval, eligibility or escalation instructions would require Leila’s review and Daniel’s retrieval test before it became available through the feature. A live output that misrepresented one of those instructions would prompt an immediate assessment of whether to pause the feature for the group. A customer decision made from an unchecked output would bring Maya back to the authorisation.

They wrote those triggers into the Monitoring Plan. Other signals would call for a review rather than an automatic pause: advisers avoiding the feature because checking cost more time than direct search; repeated questions outside service credits; reports clustering around one procedure; or complaints suggesting customers had received inconsistent explanations.

Maya asked whether they should widen the group.

“Not on this evidence,” Nadia said.

Daniel thought a few more advisers would help them learn. Leila was not opposed in principle. But the small group had already shown how much review each source change and doubtful answer required. They had not yet established a working rhythm that would hold with more people.

Maya kept the authorised group as it was. She set a further review after the team had worked without Priya beside them for a period and after the next ordinary procedure update had passed through the change check. She did not need a polished improvement curve to continue a bounded use. She did need to know that the controls could survive normal work.

The customer panel remained available for a later discussion of how the organisation explained service-credit decisions. Nothing in the monitoring review yet called for a panel question. The immediate work was inside the service team.

The notice Daniel did not get

At 4.54 that afternoon, Daniel compared the source list he had tested with the knowledge base’s change history.

A service-credit procedure had been edited that morning. The change was marked as minor. It altered a sentence about when a team lead should be consulted.

Daniel had received no notice. Leila had not approved that wording for the retrieval collection. The feature was still available to advisers.

He ran one of the saved questions. The answer looked reasonable. It cited the edited procedure.

He ran another. The answer used the new sentence in a way that might change what an adviser did next.

Daniel did not yet know whether the edit clarified the approved rule or changed it. That was Leila’s question. He could see that the source had changed before their agreed check.

He called her and showed her both outputs.

Leila read the new sentence. “I need to find out who approved this.”

Daniel opened the control that could disable the feature for the group. He did not click it yet. The two of them had enough to trigger a review, but they still needed to establish the effect of the change and whether anyone had relied on the new wording.

Maya’s Monitoring Plan had told them when to look. It had not yet told them what the morning’s edit meant for the cases already handled.

Use the Monitoring Plan to name what you will watch, who will review it and which findings bring a decision back to you. Use the Time-Savings Tracker to record the work added as well as the work shortened. Use the AI Incident Log for live failures and near misses, preserving the original output and the response.

Chapter 10, When the Assumptions Break, asks what the team does when a source changes before its controls catch up, and how it decides whether to fix, narrow or stop the use.