Adtech platform: from pilot to production
In the pilot, with a comparison group, most users chose the traditional route over the tool. Many did not understand the system and made mistakes while creating a campaign; short sessions and high bounce only looked efficient.
- Result
- Comparing two consecutive quarters in production, on the same pool of users in one large organisation: conversion of new users up 36% and adoption up 20% (both relative growth, and adoption held), and monthly retention at 62% in the second quarter.
- Context
- I brought engineers, developers, designers and domain specialists into one plan, and worked next to sales. Many stakeholders, external vendors and large campaign budgets made every change sensitive. I used AI in my own work, for example to build prototypes, and designed and validated an AI feature that we did not ship (see the first decision).
- Scope and limits
- These results cover one organisation; the plan was to raise adoption there first and expand later. They do not show how the product performed elsewhere. The smaller pilot pool was used for testing and is not the baseline for this comparison.
How it was measured
- Conversion+36%New users who reached their first campaign
- Adoption+20%Users with access who have created at least one campaign
- Retention62%Users who created a campaign and created one again the following month, in the second quarter
Two decisions
Decision 1What do we build next?
Other new features were on the table, and the backlog was sizeable. One of them was an AI feature I had designed and validated with a prototype and a time measurement: AI parses the input data into the system, so users check and correct instead of filling everything in.
Remove friction from campaign creation, and ship only the features that support adoption.
Data on how users behaved, user interviews and tests. They showed that many users did not understand the system and made mistakes while creating a campaign. That convinced the team.
Reducing friction in campaign creation became our main goal. We shipped only the improvements and features that made that flow better. The AI feature would have cut the time further and raised the value of the tool, but it waited: adoption and new clients came first.
Decision 2How do we handle bugs?
Each report handled as it came in.
User and internal bug reports triaged by priority, with P0 for critical bugs on the main user path. The report form fed our tasks, and task statuses fed the messages to the client and users.
At a high pace of work, bugs and reports got lost, and users did not know when a bug had been fixed.
We fixed the most important bugs quickly. Automated support messages told users when their bug was fixed, and if it came back we had the whole thread. I designed and set up the whole system.
The triage system
- Bug reportUsers and the internal team report through one form.
- Task with a priorityEach report becomes a task, triaged by priority.
- Status changeWork on the task updates its status.
- Message outAn automated support message tells the client and the user it is fixed.
- Full threadIf the bug comes back, the whole history is there.
A product people try once and then avoid?
Tell me where users drop off and what you already measure.