Motivation
We receive thousands of files a year that need to be cleaned and harmonized into a standardized format. Today, we have to write a custom code script for each file. If we keep going on this way, we’ll 🔥 out our team and our budget.
Key Results
No-code dashboard replaces the need for thousands of lines of custom code
Automated cleaning and harmonization system ensures highest quality results
System is ready for future growth and scale
Challenge
- In order for new customers to be onboarded to a client, their census data must be ingested into the clients’ database.
- Census files are often messy, requiring extensive cleaning before they can be uploaded.
- The client’s data science team was on track to spend 1 FTE / year writing hundreds of custom scripts to clean files.
- Preparing for January 2023 with the current process would have taken 100% of the data foundation team capacity, blocking any new investments.
Deliverables
- No-code configuration and testing dashboard. Empowers implementation team to define company specific business logic and test outputs in a UI
- Smart cleaning package. Extensible package automates file cleaning tasks, removing the need for company-specific scripts
- Daily CRON job to process latest census files. Deployed, scheduled ETL runs without the need for manual running or babysitting
Business Impact
- All files for January 2023 launches leveraged the new process. 100% passed the new quality control checks.
- One implementation team member implemented and monitored cleaning using a no-code interface.