01 / 03

Choose a task with an answer you can check

Supplier emails make a useful example. Imagine an assistant that reads an update and proposes changes to an order’s delivery date. The trial should answer a narrow question: can it find the right order, interpret the date and recognise when the message is unclear?

Keep it in a test environment. It can suggest a change while a person continues to make the real update. You can learn a great deal before giving it permission to alter a live order.

This is a test plan, not a claim about results. The outcome will depend on your messages, your order data and the tool you choose.

02 / 03

Give it an ordinary, messy sample

Collect examples with enough variety to resemble a working week. Include forwarded conversations, changed dates, multiple orders in one email and messages that do not mention an order number. Remove personal information the test does not need.

Have someone who does the job record the expected action before the assistant sees the sample. Include “ask for clarification” as a valid answer. Guessing should never earn a better score than admitting that the information is missing.

Keep some examples out of the development set. Test on those after making changes so you do not mistake a tool that has learned the examples for one that can handle new work.

03 / 03

Measure the work left for the team

Record correct proposals, missed updates, wrong updates and cases sent to review. Also record how long the reviewer needs to check each one. An attractive accuracy number can conceal a few expensive mistakes.

Look at the worst cases together. Would the reviewer notice the error? Is the source email easy to find? Can a change be undone? The answers determine how much responsibility the tool should receive.

Keep the trial if it reduces the total work and the remaining mistakes are manageable. Expand one permission at a time, with the same checks still in place.