# How I Evaluated an Ecommerce Analytics Agent: Answers, SQL, Tool Use, and Safety

> **Proof boundary.** I built this as a capstone for the Google × Kaggle 5-Day AI Agents course. The public repository records 17/17 Google ADK evaluation cases and 12/12 agents-cli evaluation cases passing in a single recorded local run. This is a local evaluation result, not a production deployment, customer outcome, cost study, or latency benchmark.
>
> **Metric coverage caveat.** The four custom metrics are repository-defined and uneven. `intent_satisfaction` carries markers for 12 of the 17 ADK cases, so 5 cases pass it without a marker; `grounded_response` passes on a minimum-length check; `tool_use_quality` passes on a table-name match in prose. Read the counts as coverage of the recorded suites, not as independent proof of analytical quality. Repeatability across model runs is future work, not a measured result.

## The decision I wanted to support

Operational data questions often arrive in plain language: which state has the worst on-time delivery, where freight is highest, or which seller pattern needs review? The useful system behavior is not “sound confident.” It is to choose the right data path, run a bounded query, and return an answer that can be checked.

I built the **Olist Ecommerce Analytics Agent** around Brazilian ecommerce data in BigQuery. It is a Google ADK multi-agent system for read-only questions across fulfillment, seller operations, customer experience, payments, and geography.

## System boundary

| Stage | Component | Boundary |
|---|---|---|
| Input | Operations question | Plain-language question about fulfillment, seller operations, customer experience, payments, or geography |
| Routing | ADK routing | Selects the single best agent via `transfer_to_agent` |
| Agents | Fulfillment, seller operations, customer experience, data analyst | One agent per department, plus direct specialists |
| Read path | Custom `query_bigquery` | SELECT only; 10 GB processing cap; 30-second timeout |
| Read path | First-party BigQueryToolset | `WriteMode.PROTECTED` blocks permanent writes to user tables; forecast and anomaly tools may create temporary session artifacts such as models |
| Output | Cited analysis | Query-backed answer returned to the operator |

The custom `query_bigquery` path accepts SELECT statements and enforces a 10 GB processing cap plus a 30-second timeout. The first-party BigQueryToolset has a different boundary: `WriteMode.PROTECTED` blocks permanent writes to user tables, but forecast and anomaly tools may create temporary session artifacts such as models. The cap and timeout are not documented across every first-party tool path. These controls narrow risk; they do not make the capstone production-safe.

## What I evaluated

I used two local suites recorded in the public repository.

| Suite | Recorded result | What it checks |
|---|---:|---|
| Google ADK evaluation | 17/17 cases passed | Fulfillment, seller operations, customer experience, payments, schema/BI, refusal, and cross-domain briefing cases |
| agents-cli evaluation | 12/12 cases passed | Tool-use quality, grounded response, intent satisfaction, and SQL safety |

The cases include questions such as late delivery by state, seller review gaps, payment mix, freight by seller state, schema discovery, out-of-scope refusal, and an executive briefing pipeline. A passing count matters only with a boundary: it means these recorded cases passed under this evaluation design. It does not prove performance on new schemas, live operational data, adversarial prompts, changing business definitions, or high-volume traffic.

## Why answer quality alone is too weak

For operational analytics, an answer can sound plausible while using the wrong tool, applying the wrong filter, or computing a number that does not reconcile to the source table. I kept four dimensions visible:

1. **Grounded response:** answer follows available data rather than invented values.
2. **Tool-use quality:** agent selects a relevant data path and parameters.
3. **Intent satisfaction:** answer addresses the decision question.
4. **SQL safety:** no write query, excessive processing, or unsupported behavior.

This is why I would evaluate an analytics agent as a system, not as a chat response. Future work should add repeatability across model runs, accepted SQL/result traces, semantic tests for ambiguous terms such as “on time,” “late,” “seller,” or “state,” and a human approval record for any proposed action.

## What this build does not prove

- No production deployment or customer workflow outcome.
- No measured business improvement, revenue, stockout reduction, or cost saving.
- No current latency or inference-cost benchmark in this article.
- No authorization model for a customer warehouse or ERP.
- No evidence that an agent should autonomously write purchase orders, inventory transfers, or supplier actions.

The appropriate next boundary is read-only decision support: retrieve the relevant facts, show query-backed reasoning, and let an operator decide whether to act.

## Reproduce or inspect it

The public repository includes the ADK app, tool boundary, evaluation datasets/configuration, and commands for re-running the local suites. Start with the [repository README](https://github.com/im-khang/kaggle-5days-vibe-coding), then inspect its test and tool directories before adapting the pattern to another dataset.

If you are building a similar system, begin with a small decision set, define expected answers and allowed queries, add refusal cases, and keep the database access read-only until you can inspect failure traces and human-review requirements.

## Sources

- [Olist Ecommerce Analytics Agent repository](https://github.com/im-khang/kaggle-5days-vibe-coding)
- [Google ADK documentation](https://google.github.io/adk-docs/)
- [Google agents-cli](https://github.com/google/agents-cli)
- [Databricks: Transforming Supply Chain Management with AI Agents](https://www.databricks.com/blog/transforming-supply-chain-management-ai-agents)