Data Readiness for AI: A Practical Cleanup Roadmap

Why start from a use case

Company-wide data cleanup programs rarely finish, because there is always more to clean and no clear point at which data is "ready". Readiness depends on the question being answered. Data that is good enough for a monthly sales dashboard may be inadequate for forecasting stock by branch.

Step 1: Define the use case and its data needs

• The decision, prediction or answer the system must produce.
• The fields required, and how much history is needed.
• The level of detail, such as per transaction, per day or per branch.
• How fresh the data must be.
• For supervised machine learning, the labels or outcomes that must be recorded.

Step 2: Map the sources

• Systems: ERP, CRM, point-of-sale, warehouse management, HR, spreadsheets, shared drives and email.
• Owners and the people who enter or change the data.
• Formats, access methods such as APIs or exports, and volumes.
• Known problems, which the people closest to the data usually already know.

Step 3: Assess quality

The UK Government Data Quality Framework describes six common dimensions of data quality. They make a practical checklist:

Summarize the findings by field and by their effect on the use case, so fixes can be prioritized.

Step 4: Fix what blocks the use case

• Master data: deduplicate customers, products and suppliers, and agree a single authoritative record for each.
• Standards: consistent codes, units, categories and naming.
• At the source: add validation to entry screens so errors stop being created.
• Process: clarify who enters what, when, and who checks it.
• History: backfill critical fields where it is feasible, and document where it is not.
• Clutter: archive obsolete records so they stop distorting results.

Step 5: Prepare documents for knowledge assistants

• Identify authoritative documents and remove outdated or duplicate versions.
• Give each document an owner and a review date.
• Clean up sharing permissions before connecting any AI system.
• Convert scanned documents to searchable text and keep headings meaningful.

Step 6: Confirm lawful and appropriate use

NPC Advisory No. 2024-04 confirms that the Data Privacy Act applies whenever personal data is used to develop or deploy AI systems, including training and testing.

• Check whether using personal data for the new purpose is compatible with the purposes declared when it was collected, and whether privacy notices cover it.
• Apply the Data Privacy Act's principles of transparency, legitimate purpose and proportionality.
• Use anonymized or pseudonymized data where individual identities are not needed.
• Check contracts that restrict the use of third-party or customer data.
• Restrict access and set retention periods.
• Carry out a privacy impact assessment for projects involving personal data.

Step 7: Build pipelines, documentation and monitoring

• Automate extraction and transformation instead of manual exports.
• Document definitions in a data dictionary, and record where each field comes from.
• Build quality checks into pipelines and alert when they fail.
• Track quality indicators over time on a simple dashboard.

Step 8: Assign roles

The roadmap

1. Use case definition and data map.
2. Quality assessment with prioritized issues.
3. Fixes for blocking issues, including source system changes.
4. Automated pipeline with quality monitoring.
5. Pilot of the use case, with feedback on data issues.
6. The next use case, reusing the cleaned data, definitions and pipelines.