KPIs to Measure Before and After Implementing an AI Agent on Your Website
Without a clear baseline, you can't prove what an AI agent actually did for your business. This guide gives you the exact KPIs to capture before and after, and how to set them up in GA4.
Measuring whether your AI agent is working isn't about checking how many conversations it had last month — it's about comparing a "before" and an "after" using the same metrics, set up correctly from day one. This guide gives you the concrete KPIs to capture before implementing an AI agent on your website, how to track them over the first 90 days, and how to set up attribution in GA4 so the conversions your agent drives don't get buried in general organic traffic — the exact data you need to bring a defensible business case to leadership.
Why "How Many Conversations the Bot Had" Isn't a Business KPI
It's the most common mistake when evaluating an AI agent: measuring activity instead of impact. Conversation count, message volume, or average session time tell you the tool is being used — they don't tell you whether it's moving anything leadership actually cares about: more leads, more sales, lower support cost, better retention.
The reason so many marketing teams show up to the results meeting without a solid case usually isn't that the AI agent isn't working — it's that nobody captured a baseline before implementing it. Without that "before," any "after" number is a claim with no point of comparison, and leadership's first question is going to be exactly that: "compared to what?"
This guide solves both halves of that problem: what to measure and when, and how to set up attribution so those numbers are real data, not estimates.
The Four KPI Categories You Need (Not Just Conversion)
A well-evaluated AI agent gets measured across four dimensions, not one. Measure only conversion and you'll miss the operational savings; measure only volume and you won't be able to justify the spend.
1. Business KPIs (what leadership looks at)
Agent-assisted conversion rate — what share of the agent's conversations end in a value action (lead captured, sale, demo booked).
Leads attributed to the agent — with their own source identifier, not mixed into "organic" or "direct" traffic.
Average value per converted conversation — if you have an average order value or lead value, every converting conversation has a dollar figure attached to it.
2. Experience and Engagement KPIs
Fallback rate — how often the agent fails to understand or resolve a query and has to escalate. A fallback rate above 20-25% is a common red flag, usually signaling the agent isn't deeply enough connected to your real data.
CSAT or end-of-conversation rating — direct user sentiment, not a proxy metric.
Average conversation duration — not too short (the user didn't get what they needed) and not too long (the agent is slow to resolve things).
3. Operational KPIs (what justifies the time saved)
Share of queries resolved without human intervention — the real indicator of how much time the agent is handing back to your team.
First response time — compared to your human first response time before implementing the agent.
Volume of tickets escalated to human support — before vs. after, to quantify how much load actually dropped.
4. Retention and Cost KPIs
Repeat purchase or retention rate for customers served by the agent — if your business can measure this, it's one of the most powerful numbers for leadership.
Cost per resolved conversation — the agent's fixed monthly cost divided by the volume of resolved conversations, compared to the equivalent cost of human support for that same volume.
What to Capture BEFORE Implementation (Your Baseline)
This is the part most chatbot guides skip entirely — and it's the one that decides whether you'll be able to prove anything in 90 days. Before you turn the agent on, document:
Your website's current conversion rate (overall, and segmented by the pages the agent will operate on, if you can).
The volume of inquiries your support or sales team currently handles across existing channels (email, form, live chat if you already have one), and how many of those arrive after hours.
Your current human first response time, pulled from real helpdesk or CRM data, not an estimate.
Leads or sales you can attribute to a slow or missing response — even if it's a qualitative estimate from your team, document it now, don't try to reconstruct it from memory later.
Your current support cost (staff hours dedicated, cost of existing support tools) so you can compare it against the AI agent's fixed cost.
If you don't have this documented before launch, you'll have to lean on industry benchmarks as a stand-in — and that's a much weaker position in front of leadership than your own historical data.
How to Set Up Agent Attribution in GA4
This is the step almost no chatbot guide covers, and it's the one that turns "I think the agent is helping" into a defensible number.
Create a custom event for every conversation started (for example,
ai_agent_conversation_start) and another for every conversion completed through the agent (ai_agent_conversion), with a parameter identifying the channel.Tag every lead captured by the agent with a source parameter before it hits your CRM, so you can later filter "leads originated by the AI agent" from everything else.
Build a dedicated audience or segment in GA4 for users who interacted with the agent, and compare their conversion rate against users who didn't — it's the most direct comparison you can show leadership.
Account for zero-click traffic. Traditional click-based attribution is losing precision as more interaction happens inside the conversation itself rather than through standard page-to-page navigation — so comparing audiences (interacted vs. didn't) is more reliable than trying to rebuild a perfect click funnel.
The Measurement Window: What to Expect at 30, 60, and 90 Days
Don't judge the agent's performance on week-one data — and leadership shouldn't expect full results that fast either.
First 30 days: expect the fallback rate to drop as the agent "learns" from real conversations and you tune what information it can access. A higher-than-normal rate of human escalations during this phase is normal.
30-60 days: agent-assisted conversion rate should start to stabilize. This is when you run your first real comparison against baseline.
60-90 days: compare the full set of KPIs (business, experience, operational, and cost) against your pre-implementation baseline. This is the review you take to leadership.
Avoid presenting a single isolated number ("the agent generated 40 leads") without the context of what share that represents of the total, and how it compares to the equivalent period before implementation.
How to Present These KPIs to Leadership
Taylor doesn't need to impress with a volume of data — just three clear numbers that answer leadership's real question: is this moving something we actually care about?
The cost of not implementing it — lost leads, support hours, after-hours opportunities, using your own documented baseline.
The cost of the tool — the flat monthly fee, no fine print.
The return measured over the evaluation period — the real agent-assisted conversion and cost-per-resolved-conversation data, not a projection.
Frame the conversation as a baseline comparison, not a promise about the future — that's the difference between presenting a solid business case and asking for a leap of faith.
You Can't Manage What You Don't Measure From Day One
Even the most effective AI agent loses its business case if nobody documented how things looked before it went live. The difference between a business case that convinces leadership and one that stays at "seems like it's working" is this data, captured from day zero — not reconstructed from memory three months later.
Aira Web and Aira Ecommerce are built to make this kind of measurement possible from day one: answers based on your real business data (which keeps fallback rate low) and a flat, predictable monthly fee, so the cost-per-resolved-conversation math stays simple instead of an estimate with fine print.
Written by the Aira content team, specialists in conversational AI agents for ecommerce and service businesses.
Frequently asked questions
How long should I wait before evaluating an AI agent's performance?
At least 60-90 days for a full evaluation. The first 30 days usually involve more tuning and human escalations than normal while the agent gets calibrated against your real data.
What if I didn't document a baseline before implementing the agent?
You can reconstruct rough approximations from historical CRM or helpdesk data (response time, inquiry volume), but it's a weaker foundation than capturing it beforehand. For your next implementation or expansion, document the baseline first.
How do I tell apart the conversions the agent brought from the ones I'd have gotten anyway?
By setting up a segment or audience in GA4 that compares users who interacted with the agent against those who didn't, within the same period and the same pages. That's more reliable than trying to attribute every sale to a single touchpoint.
If I can only present one KPI to leadership, which one matters most?
Agent-assisted conversion rate combined with cost per resolved conversation — it directly connects performance to budget, which is what leadership needs to see first.
Does this apply the same way to an AI agent for services as it does for ecommerce?
The four-category framework is the same. What changes is the detail of the business KPIs: for ecommerce, also track cart recovery and checkout-assisted conversion; for services, track captured lead quality and conversion-to-booked-meeting rate.
