The problem
A cleaned file alone hides decisions. Operators also need to know which rows failed, why they failed, which duplicate survived, and whether rerunning will create new copies.
Case study 03 · Python · data quality
Clean output plus evidence of what was rejected.
The pipeline validates a messy CSV, normalizes accepted records, applies an explicit duplicate-survivorship rule, loads idempotently, and generates a stakeholder-readable quality report.
Public demonstration using synthetic order data. Counts below describe the included sample run, not a client result.
4/4focused tests
5sample rows loaded
2sample rows rejected with reasons
// try the browser demo
Messy rows in
The sample contains mixed casing, currency formats, duplicate order IDs, a missing email, and an invalid amount.
—input rows
—loaded
—rejected
Ready. Run the synthetic sample through the documented rules.
Public demonstration · synthetic order data · sample rules shown explicitly
// pipeline flow
Ingest→Validate→Normalize→Dedupe→Load→Report
Typed validation
Required identifiers, email shapes, dates, and numeric amounts are checked explicitly. Invalid rows are separated with their original line and named problems.
Normalization and dedupe
Dates, phone numbers, currency, email casing, and whitespace become consistent. Duplicate order IDs use a documented latest-date-wins rule.
Repeatable delivery
Accepted records load idempotently into SQLite, and each run writes a human-readable report. The destination can be replaced without changing the validation contract.
// verified behaviors
8 input → 5 loaded + 2 rejected + 1 superseded duplicate → line-level quality reportThe test suite verifies missing/invalid-field reporting, type normalization, latest-record survivorship, and multiple date formats. Client work begins by agreeing the real rules on a non-sensitive sample.
Need rules applied to your files?
Send one representative non-sensitive sample, the row count, and the desired output format.
[email protected]