Before an AI assistant touches a work task, define the decision around it

Start with one bounded decision, not a general promise

A demonstration can make an AI assistant look ready for almost anything. Give it a neat prompt and it may draft an email, summarize a document or sort a list in moments. But a good-looking response is only one part of a work process. It says little about the information that was supplied, the exceptions it missed, the people affected by the decision or what happens when the tool is confidently wrong.

Before a team switches on a new assistant, write a short task card for one narrow use. State the input it may receive, the output it may produce, who will use that output and what decision remains for a person. ‘Help support staff turn approved knowledge-base articles into a first-draft reply’ is bounded. ‘Handle customer support’ is not. The first can be evaluated; the second quietly combines policy, privacy, tone, escalation and accountability into a slogan.

NIST’s AI Risk Management Framework treats context, intended purpose, users, impacts, assumptions and limitations as things to understand and document. That is a helpful counterweight to feature-led adoption. A task card is not bureaucracy for its own sake. It makes it possible to tell whether the trial succeeded at its actual job—and to spot when people begin using the tool for a different one.

Draw the boundary around the input before writing the prompt

The prompt is not the only input. Attachments, retrieved documents, pasted conversations, spreadsheet rows, system instructions and tool results can all influence an AI application. Make an explicit list of what the trial may receive and what it must never receive. Include customer records, employee information, credentials, unpublished plans, regulated data and material subject to confidentiality obligations. A short, approved example set is a better starting point than a live shared drive.

Also separate content that an AI may summarize from content that is allowed to direct it. A document being summarized can contain a sentence that looks like an instruction; that does not make it a trustworthy instruction for the system. OWASP identifies prompt injection as a leading risk for applications built around large language models. In plain terms, untrusted text may try to steer the model away from the task it was given. A user-facing warning alone is not a control.

For a small pilot, keep the design boring. Restrict the data sources, give the assistant the minimum permissions it needs, and prevent it from taking an irreversible external action. If it can draft a ticket, a person can approve and send it. If it can search a policy library, keep that search separate from the authority to change a customer record. The boundary is valuable even when the tool is accurate: it limits the impact of an error, an unexpected instruction or a changed integration.

• Name approved sources and explicitly excluded data types.
• Treat retrieved or uploaded text as content to assess, not instructions to obey.
• Give the pilot read-only or draft-only access where practical.
• Record the model, application version, connected tools and settings used in the test.

Test the work people will actually hand it

A trial built only from easy examples tests presentation, not reliability. Build a compact test set from realistic work: ordinary cases, incomplete inputs, ambiguous requests, out-of-date source material, conflicting records and cases that should be escalated. Remove sensitive details or use approved synthetic examples. For every case, specify what a good answer looks like, what must not happen and when the right result is ‘I do not have enough information.’

Do not reduce the result to a single percentage. A draft that is useful 90 percent of the time can still be unsuitable if the remaining 10 percent includes an unsafe instruction, a disclosure of private information or an invented policy. Review error types as well as frequency. Did the system omit a condition, cite a source it did not use, overstate uncertainty, follow text embedded in a document, or apply a rule outside its scope? Those findings tell you whether to change the task boundary, the source set, the review step or the tool itself.

NIST’s framework calls for testing before deployment and monitoring in operation, with measures documented for conditions similar to the intended setting. That does not require a giant lab. It does require enough evidence that the team is not mistaking a polished demo for an operational result. Keep the prompts, inputs, outputs, reviewer decisions and known failures so a later change in the model or workflow can be compared with a real baseline.

Make human review a decision, not a ceremonial click

‘Human in the loop’ can mean almost nothing unless the person has a defined job, enough context and authority to stop the output. Decide what the reviewer checks: factual support, tone, policy fit, calculation, privacy, escalation, or all of these. Give them the sources and a clear path to correct or reject the draft. A reviewer who sees only a fluent paragraph and a green approve button is being asked to rubber-stamp, not to exercise judgment.

The level of review should follow the consequence of being wrong. A low-stakes internal brainstorm may need a light check. A recommendation that affects a customer, candidate, patient, payment, safety decision or legal obligation needs much stronger controls—and may be a poor fit for an early pilot. The owner of the work, not the model supplier, should set the acceptable error types and decide which cases must always be escalated.

NIST’s playbook emphasizes defined responsibilities for human-AI configurations and documentation of oversight, overrides, reported errors and response. Translate that into a simple operating rule: name the accountable role, teach reviewers what the tool is and is not for, and give them permission to decline an output without having to defend the model. An assistant should make the responsible person more effective; it should not make responsibility harder to locate.

Choose a stop rule before the pilot starts

A pilot needs a finish line and a brake. Write down the duration, sample size, success measures, owner and review date. Useful measures might include whether a reviewer can correct a draft within a defined time, whether citations point to the approved source, how often the assistant produces an escalation when it should, and which errors require immediate suspension. Avoid treating activity—number of prompts, minutes saved or enthusiastic comments—as proof of fitness for a task.

Make the stop rule concrete. Pause the pilot after a privacy breach, an unapproved external action, a repeated failure on a critical requirement, or evidence that people are using the tool beyond its approved scope. Then investigate the workflow as well as the model: Was the source set wrong? Was the instruction unclear? Did a system integration expose too much authority? A useful incident record captures the case, impact, containment, decision and follow-up test without turning a mistake into a hunt for someone to blame.

Finally, review the pilot when the tool changes. A new model release, new retrieval source, revised prompt template or new integration can change the behavior you tested. NIST’s risk guidance frames management as ongoing rather than a one-time approval. The durable habit is modest: define the task, protect the inputs, test difficult cases, give a named person a meaningful review role, and pause when the evidence no longer supports the use. That approach will not make an AI assistant infallible. It makes the organization less likely to discover its limits only after a consequential mistake.