Processing Hundreds of Gigabytes Without a Giant Cluster

The NEMSIS source contained tens of millions of records and approximately 300 GB of data. Loading everything into memory or repeatedly scanning raw source files would have made experimentation slow and fragile.

I divided the work into stages: source conversion, column and cohort filtering, analytical joins, complete-case construction, and modeling. Intermediate Parquet datasets made each stage reproducible and prevented unnecessary reprocessing.

Columnar storage, early filtering, and DuckDB-based analytical queries reduced the amount of data carried into later stages. Separating data preparation from modeling also made it easier to validate row counts and investigate where records were removed.

I would add a formal data lineage manifest, automated schema-drift checks, and a versioned data-quality report for every pipeline run. I would also make cohort exclusions directly inspectable through an audit table.