Getting data ready for AI: the unglamorous part that decides the result

The demonstration always works. A model answers questions about a sample of documents, or predicts demand from a tidy spreadsheet. Then the project meets the real data: scanned PDFs, five versions of the same customer, sensor readings with gaps, and a question from legal about whether the data can be used at all.

Choose a narrow first use case

Pick a decision that people already make and can measure, such as which invoices to check first, which pumps to service, or which citizen requests to route where. A narrow case limits the data you need and gives you a baseline to beat.

Know what you have

• list the sources involved, their owners and how often they change;
• check completeness, duplicates and obvious errors before any modelling;
• record where each field comes from, so results can be traced back.

Arabic text needs its own preparation

Documents and messages in Arabic need normalisation of letter variants, removal of diacritics where they add noise, and careful handling of mixed Arabic and English. Optical character recognition on scanned Arabic forms needs testing on your own documents, because accuracy varies widely with fonts and quality.

Permission is part of the data

Personal data used to train or prompt a model is still personal data. Confirm the legal basis for each use, remove what is not needed, and keep access controls on the data when it is copied into a new platform. For assistants that search internal documents, the answers must respect the same permissions as the documents themselves, or the assistant becomes a way around them.

Measure before and after

Define how success will be measured before building, label a test set that represents real cases, and keep it separate from training data. Review errors with the people who will use the system, not only the accuracy figure.

Plan for running it

A model in production needs monitoring for drift, a way for users to flag wrong answers, and an owner who decides when to retrain. Budget for that from the start.

Our data and AI team builds data platforms, Arabic-aware search and AI assistants on clients' own data, in the cloud or in country.