01 / 03

The finding worth looking at

Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the introduction of an AI assistant to customer support work. Their NBER working paper reports improvements in issues resolved per hour, with larger gains for less experienced workers than for the most experienced workers.

That difference is useful to think about. An average across a team can hide very different experiences of the same tool. The study concerns a particular support setting; it does not establish the return another company will get from another workflow.

02 / 03

Ask who needs help with which part

Our practical reading is to start by separating the work. A new team member may spend time finding the relevant policy. An experienced colleague may already know it and need help with a different part of the job.

Before a trial, ask people where they pause and where they feel confident. Give them examples of the proposed tool and watch how they check its suggestions. A tool can be useful to one group while adding little for another.

The team’s experienced people should help decide what good looks like. Their judgement belongs in the examples and the review, including the situations where the usual answer is wrong.

03 / 03

Keep the result local

For your own test, measure a defined task before and after introduction. Look at both completion time and the quality of the result. Break the view down by experience and task type so an average does not settle the whole discussion.

Ask whether people understand the work better after using the tool. Can they explain why a suggestion is right? Can they recognise when it is wrong? Those are useful questions to review alongside speed.

This is our application of the research, not a result claimed by the authors for consumer brands. The paper gives a reason to test carefully and to pay attention to who benefits.