AI sales agent trial: what 14.8m visitors show

Retailers in Australia and overseas are putting AI assistants on their websites, and most of them are still asking the same question: does an AI sales agent actually make more money than having no agent at all? A preprint posted to arXiv on 18 October 2026, Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations, reports one of the largest field tests of that question so far. It also comes from the company that sells the agent, which matters for how much weight it can carry.

The question the paper asks

Can an AI agent trained specifically on real sales conversations lift revenue on live websites, compared with no agent, with a general-purpose frontier model given the same job, and with human chat staff? The author also asks whether it can do so without pushing up returns, complaints or unsupported product claims.

What they did

The agent, called Javo, is an open-weight language model (Qwen3-32B) further trained on 4.6 million consented, de-identified commercial conversations, of which 46 per cent were retail, 28 per cent travel and 26 per cent services. It was then tested in a randomised controlled trial on 100 live websites in 14 industries, 60 of them e-commerce, between 1 May and 30 September 2026. In total 14.8 million unique visitors were randomly assigned to see no agent, Javo, or a frontier model (GPT-5.2) running inside the same software with a sales-practice instruction block written by a sales trainer. On 20 sites that already had staffed live chat, some visitors were routed to human associates instead. The sites were mostly in the United States (38), the UK (18), France (15), Spain (9) and Portugal (8); the paper does not name any Australian site.

What they found

Against no agent, revenue per visitor rose 10.0 per cent (95 per cent confidence interval +5.1 to +14.9 per cent) and conversion rose 8.0 per cent (+4.1 to +11.9 per cent). These are measured across every visitor offered the agent, whether or not they used it, which is the conservative way to count. The effect varied between sites; the author estimates that a new site could expect anywhere from +2.3 to +17.7 per cent.

Against the frontier model in the same software, visitors who engaged with Javo converted 21.6 per cent more often in relative terms (+16.9 to +26.3 per cent), and revenue per visitor was 16.9 per cent higher. Both agents were used by the same share of visitors, about 8 per cent, so the difference came from what happened in the conversation, not from more people starting one.

Against human chat associates on the 20 staffed sites, engaged conversion was 11.26 per cent for Javo and 11.20 per cent for the humans, and customer satisfaction was 4.42 versus 4.43 out of 5. The paper describes this as non-inferior, meaning no worse within a pre-set margin, not better. The abstract puts Javo's cost at 1.8 per cent of the human cost per conversation, and says returns, unsupported product claims and complaints all stayed within limits set before the trial.

What it means for an Australian retailer

If the result holds, the useful finding is not that AI chat works; it is that a model trained on real sales conversations beat a general-purpose model doing the same job by a wide margin. The assistants large Australian grocery and hardware retailers have launched on Google's agent platform are built on general-purpose models with instructions layered on top. This paper suggests the training data and the definition of a good outcome matter at least as much as the size of the model. That is a question to put to any vendor: what was the agent trained to do, and how is a sale counted?

The second implication is about measurement. The trial's design, randomly holding back the agent from a share of visitors and comparing revenue per visitor, is something any retailer with meaningful traffic can copy for its own assistant. Applying the 10 per cent figure to Australian sites would be extrapolation: no Australian site is identified in the trial, and the human comparison covered only furniture, hotels, holiday rentals and beauty.

How much to trust it

With caution. This is a preprint, version 1, and has not been peer reviewed. It has a single author, Benjamin Tannenbaum, whose listed affiliation is Javo, AISO Boost Ltd, the company behind the agent being tested, and the training data comes from the same company's conversation collection. That is a direct conflict of interest. The trial is large and randomised, which is a genuine strength, but it has not been replicated by anyone independent.

There are also inconsistencies a reader should know about. The non-inferiority margin for the human comparison is given as minus 10 per cent in the design section and minus 7 per cent in the abstract and results table. The revenue result came in above the range the author says was set in advance (+3 to +9 per cent). The human comparison is not blind: the human chat window shows the associate's first name while Javo says it is an AI, which the paper itself names as a confound, and the associates were typical staff, not top performers.

Some numbers could not be checked. We were not able to read the paper's sections on industry-by-industry results, the measured guardrail values, the detail behind the 1.8 per cent cost figure, the conflict-of-interest statement and the limitations section in full; those are reported here as the abstract states them. The paper lists its limitations as site selection, the frontier model being a snapshot in time, the disclosure confound, typical rather than best human staff, the five-month horizon, interference between arms, the use of artificial customers in the offline tests, language and region, and AI shopping agents arriving on the buyer side. The headline results are also relative percentages; absolute revenue figures are not given in the sections we read.

Treat this as the strongest field evidence yet that a purpose-trained sales agent can beat both no agent and a general model, and as a vendor's own evidence that still needs independent replication.

Citation

Tannenbaum, B. (2026). Selling in the Wild: Benchmark and Randomized Field Evidence for a Sales Agent Trained on Commercial Conversations. arXiv preprint arXiv:2610.25461 [cs.IR], submitted 18 October 2026. Not peer reviewed. https://arxiv.org/abs/2610.25461