---
title: 13 ML problems to test a data agent
subtitle: An auto-graded data-science
benchmark for an agentic data assistant
slug: ml-benchmark-suite
hook: Multi-table data up to 33.8M rows,
hidden ground truth, and graders proven
perfect before shipping. In about 9 days.
role: Data Scientist and Benchmark Engineer,
one of about 10 developers
employer: Turing (contract, remote)
client: An enterprise data & AI platform
company
period: Sep – Oct 2025
tracks:
- data-science
- evaluation
tags:
- benchmark design
- feature engineering
- imbalanced classification
- regression
- demand forecasting
- recommender systems
- auto-grading
metrics:
- value: "13"
label: End-to-end, auto-graded ML problems
built in about 9 days
when: Oct 2025
- value: 33.8M
label: Rows in the largest multi-table
dataset (3–10 tables per dataset)
when: Sep – Oct 2025
- value: 1.67M
label: Blockchain transactions engineered
into 55,350 address-level rows for a
fraud-detection pilot
when: Sep – Oct 2025
---
## Context
An enterprise data & AI platform company
wanted to measure how well its agentic
data assistant handles real data-science
work, not toy CSVs. The plan was a
benchmark of end-to-end ML problems in the
spirit of MLE-bench. In September and
October 2025 I was one of about ten
developers building it, as a data
scientist and benchmark engineer.
## The problem
Agents are getting good at single-file,
Kaggle-style problems. Real business data
doesn't come in one tidy file. It lives in
many tables, and most of the work is
joining them sensibly and engineering
features across them. A benchmark that
hands over one flat CSV measures the wrong
thing.
## Constraints
- **Hidden ground truth.** The agent must
never see the answers the grader uses.
- **Graders must be correct before they
ship.** A wrong grader quietly corrupts
every score.
- **Reproducible data.** Splits and ingestion
must rebuild identically from scratch.
- **No duplicates across developers.** Ten
people sourcing datasets in parallel will
collide unless someone checks.
- **The format changed mid-stream.** Reviewers
asked for a new split structure in
October.
## What I built
**Dataset sourcing.** I proposed 32
high-quality open, non-Kaggle datasets,
each described against a structured
schema. 11 of them went into the team's
shared source list of 464 datasets.
**Two pilot problems.**
- **Fraud detection on 1.67M blockchain
transactions**, engineered into 55,350
address-level feature rows with extreme
class imbalance, and modeled with Random
Forest, XGBoost and logistic regression
with oversampling, scored on F1.
- **A retail site-selection recommender**,
built from public US census business
patterns and economic time series.
**13 production problems (October 2025).**
Domains: grocery e-commerce, music
streaming, grocery demand, recipe ratings,
digital-media retail, corporate bankruptcy
and bank lending. The data came from 7
public multi-table datasets, each with
3–10 tables and up to 33.8M rows. Every
problem shipped with:
- Training, Test and hidden Test_groundtruth
splits;
- an `instructions.md` task brief;
- an `eval.py` grader against hidden ground
truth (accuracy, RMSE, MAE or macro-F1).
Each problem was designed around multi-table
relational joins, so an agent has to
engineer features across tables rather
than read one flat file.
**Proof before shipping.** A grader test
harness (`test_all_evals.py`) proves that
every grader scores the ground truth
perfectly before a problem ships. Split
scripts per problem and dataset ingestion
through the Kaggle API make the data
pipeline reproducible.
**Review-driven iteration.** I moved every
October problem to the new split format
the reviewers asked for, and dropped one
dataset another developer was already
using, to keep the benchmark free of
duplicates.
## How it fits together
What the agent sees and what the grader sees
are deliberately different.
A public multi-table dataset passes a schema
review: is this really multi-table work? A
split script produces training and test
tables, plus a hidden ground-truth file.
The agent gets the task brief and the
training tables. The grader gets the
hidden answers. Before anything ships, the
test harness proves the grader gives the
ground truth a perfect score, and peer
review can send a problem back for rework.
## Stack
- **Python:** pandas, NumPy, scikit-learn,
XGBoost, imbalanced-learn, joblib,
openpyxl.
- **Data:** the Kaggle API for reproducible
ingestion, public multi-table datasets,
SQL-style relational joins.
- **Evaluation:** custom graders (accuracy,
RMSE, MAE, macro-F1, F1), a grader test
harness.
- **Delivery:** Git.
## Results
- **13 end-to-end, auto-graded ML problems in
about 9 days**, covering classification,
regression, demand forecasting and
recommender systems.
- **Multi-table data of up to 33.8M rows**,
from 7 public datasets with 3–10 tables
each.
- **Every grader proven** against ground truth
before shipping.
- **32 dataset proposals**, 11 adopted into
the team's 464-dataset source list.
## What I'd do differently
- **Ship a naive baseline score with every
problem.** An agent's score means little
until you know what predicting the mean,
or the majority class, would have scored
on the same split.
- **Test graders against plausible wrong
answers, not just the perfect one.**
Proving a perfect submission scores
perfectly is necessary, not sufficient. In
my later RL work (case study 01), every
grader also had to fail known wrong
answers. I'd bring that in here.
- **Check for leakage across tables
automatically.** Multi-table data makes it
easy to leak the label through a sibling
table. A scripted check per problem beats
eyeballing joins.
- **Freeze dataset versions with checksums
from the first commit.**
## What I left out
This is client work under NDA. Not included,
on purpose:
- the client's name and product, and the
benchmark's internal name and size
targets;
- which public datasets went into which
problem;
- task briefs, graders and hidden ground
truth;
- reviewers' comments and the client's
internal results.
## How to verify
- **LinkedIn project** "ML benchmark suite for
an agentic data assistant" (Turing, Sep –
Oct 2025): 13 problems, multi-table data
up to 33.8M rows, hidden ground truth,
self-tested graders, built in ~9 days.
- **LinkedIn, Turing experience entry:** the
fraud-detection pilot on 1.67M blockchain
transactions.
- **Employment check:** Turing, official title
Consultant, contract, Mar 2024 – present.
- **In an interview:** a walk-through of the
method, within NDA limits.
- **Code:** walkthroughs available on request.
Case study 09 · Turing (contract, remote) · Sep – Oct 2025
13 ML problems to test a data agent
An auto-graded data-science benchmark for an agentic data assistant
Multi-table data up to 33.8M rows, hidden ground truth, and graders proven perfect before shipping. In about 9 days.
✓ Compiled from this page's own markdown.
Context
An enterprise data & AI platform company wanted to measure how well its agentic data assistant handles real data-science work, not toy CSVs. The plan was a benchmark of end-to-end ML problems in the spirit of MLE-bench. In September and October 2025 I was one of about ten developers building it, as a data scientist and benchmark engineer.
The problem
Agents are getting good at single-file, Kaggle-style problems. Real business data doesn't come in one tidy file. It lives in many tables, and most of the work is joining them sensibly and engineering features across them. A benchmark that hands over one flat CSV measures the wrong thing.
Constraints
- Hidden ground truth. The agent must never see the answers the grader uses.
- Graders must be correct before they ship. A wrong grader quietly corrupts every score.
- Reproducible data. Splits and ingestion must rebuild identically from scratch.
- No duplicates across developers. Ten people sourcing datasets in parallel will collide unless someone checks.
- The format changed mid-stream. Reviewers asked for a new split structure in October.
What I built
Dataset sourcing. I proposed 32 high-quality open, non-Kaggle datasets, each described against a structured schema. 11 of them went into the team's shared source list of 464 datasets.
Two pilot problems.
- Fraud detection on 1.67M blockchain transactions, engineered into 55,350 address-level feature rows with extreme class imbalance, and modeled with Random Forest, XGBoost and logistic regression with oversampling, scored on F1.
- A retail site-selection recommender, built from public US census business patterns and economic time series.
13 production problems (October 2025). Domains: grocery e-commerce, music streaming, grocery demand, recipe ratings, digital-media retail, corporate bankruptcy and bank lending. The data came from 7 public multi-table datasets, each with 3–10 tables and up to 33.8M rows. Every problem shipped with:
- Training, Test and hidden Test_groundtruth splits;
- an
instructions.md task brief;
- an
eval.py grader against hidden ground truth (accuracy, RMSE, MAE or macro-F1).
Each problem was designed around multi-table relational joins, so an agent has to engineer features across tables rather than read one flat file.
Proof before shipping. A grader test harness (test_all_evals.py) proves that every grader scores the ground truth perfectly before a problem ships. Split scripts per problem and dataset ingestion through the Kaggle API make the data pipeline reproducible.
Review-driven iteration. I moved every October problem to the new split format the reviewers asked for, and dropped one dataset another developer was already using, to keep the benchmark free of duplicates.
How it fits together
What the agent sees and what the grader sees are deliberately different.
A public multi-table dataset passes a schema review: is this really multi-table work? A split script produces training and test tables, plus a hidden ground-truth file. The agent gets the task brief and the training tables. The grader gets the hidden answers. Before anything ships, the test harness proves the grader gives the ground truth a perfect score, and peer review can send a problem back for rework.
Stack
- Python: pandas, NumPy, scikit-learn, XGBoost, imbalanced-learn, joblib, openpyxl.
- Data: the Kaggle API for reproducible ingestion, public multi-table datasets, SQL-style relational joins.
- Evaluation: custom graders (accuracy, RMSE, MAE, macro-F1, F1), a grader test harness.
- Delivery: Git.
Results
- 13 end-to-end, auto-graded ML problems in about 9 days, covering classification, regression, demand forecasting and recommender systems.
- Multi-table data of up to 33.8M rows, from 7 public datasets with 3–10 tables each.
- Every grader proven against ground truth before shipping.
- 32 dataset proposals, 11 adopted into the team's 464-dataset source list.
What I'd do differently
- Ship a naive baseline score with every problem. An agent's score means little until you know what predicting the mean, or the majority class, would have scored on the same split.
- Test graders against plausible wrong answers, not just the perfect one. Proving a perfect submission scores perfectly is necessary, not sufficient. In my later RL work (case study 01), every grader also had to fail known wrong answers. I'd bring that in here.
- Check for leakage across tables automatically. Multi-table data makes it easy to leak the label through a sibling table. A scripted check per problem beats eyeballing joins.
- Freeze dataset versions with checksums from the first commit.
What I left out
This is client work under NDA. Not included, on purpose:
- the client's name and product, and the benchmark's internal name and size targets;
- which public datasets went into which problem;
- task briefs, graders and hidden ground truth;
- reviewers' comments and the client's internal results.
How to verify
- LinkedIn project "ML benchmark suite for an agentic data assistant" (Turing, Sep – Oct 2025): 13 problems, multi-table data up to 33.8M rows, hidden ground truth, self-tested graders, built in ~9 days.
- LinkedIn, Turing experience entry: the fraud-detection pilot on 1.67M blockchain transactions.
- Employment check: Turing, official title Consultant, contract, Mar 2024 – present.
- In an interview: a walk-through of the method, within NDA limits.
- Code: walkthroughs available on request.