Your Data Is the Project

If you are planning something that answers questions out of your own information, expect the bulk of the effort to land on the information rather than on the model. Finding it. Working out which copy is authoritative. Fixing the identifiers so records join properly. Deciding who may see what. The model part is often a week of work. The data part is the project.

This is the least popular sentence in any kickoff meeting and it is the one that decides how the whole thing turns out. A system that reads your records inherits every mess in them, immediately and visibly, in front of whoever is using it. Nothing about the technology cleans anything up on the way through.

The consolation is that none of this work is wasted if the plan changes. Customer identifiers that match, a document store where the current version is knowable, and permissions that reflect who actually works here are useful for reporting, for audits and for whatever you build next.

Every company has an official answer to this and a real one. The official answer names three or four systems. The real one also includes a shared drive nobody owns, a set of spreadsheets the finance team treats as the truth, email attachments that never made it into any system, and a database set up for a project that ended years ago and is somehow still being updated.

Build the inventory by asking people rather than by scanning. Sit with somebody who does the job and ask them to show you, step by step, where they look when a customer calls. You will find sources that appear on no architecture diagram, and those are usually the ones with the answers in them.

For each source write down the owner, how current it is, and whether anybody would notice if it stopped being updated. That last column reliably identifies the things that have already stopped being updated. Far better to learn that now than after the new system starts quoting them to customers.

Identity is where these projects usually stall. The same company exists in your records twice under slightly different names, a third time under the old trading name, and a fourth time because an integration could not match any of them and created a new one. People handle this constantly and invisibly. A system joining records treats them as four separate customers.

The consequences are specific and embarrassing. A summary of a customer's history that leaves out half of it. An answer saying there is no record of an order that plainly exists. A total that is wrong because it counted one branch and missed the other. Every one of those gets noticed by the customer rather than by you.

Fixing it starts with deciding which system is the master for each kind of record, then doing the unglamorous reconciliation. A good share can be matched automatically. A stubborn remainder needs a person looking at pairs and deciding. Budget for that person. Every project that assumed the matching would be automatic ended up doing it by hand anyway, later, under more pressure.

Then keep it fixed. Duplicates come back through whatever created them in the first place, which is usually a form that lets anybody type a company name however they like. Fixing the records without fixing the form buys you a year.

Retrieval finds the most relevant looking passage rather than the most current one. Relevance is about wording, and old documents are frequently worded more clearly than their replacements, because the replacement was written in a hurry by somebody who assumed everyone already knew the background.

So an archive a person would navigate by instinct becomes a hazard. The superseded pricing sheet. The policy from before the rules changed. The onboarding guide describing a system you retired. All of it sits in the same folder as the current material, and nothing in the text marks which is which.

The remedy is dull. Move historical material out of whatever the system reads and into an archive it cannot reach. Where you cannot delete for legal reasons, separate it properly. Put a review date and an owner on everything that remains, and treat anything past its review date as suspect rather than as content.

Do this before the pilot rather than during it. Bad source material makes a pilot fail for reasons that have nothing to do with whether the approach was sound, and once a room has watched a system quote an obsolete price list, the idea takes a long time to recover.

Most projects reach this question last and should reach it first. Your folder structure and your system permissions currently do a great deal of quiet work. A retrieval layer that reads everything and answers everybody removes that structure without anybody having decided to remove it.

The failure looks like this. Somebody in the warehouse asks a general question about staffing and gets an answer drawn from a document in the HR folder that they could never have opened directly. Nothing was hacked. The permissions were simply never carried through into the new path to the data.

Apply permissions at retrieval time, per user, against the same groups your existing systems already use. Every document in the index carries who may see it, and the search runs inside that constraint. Filtering the answer after it has been generated is a weaker mechanism and it will leak eventually, usually in a summary that mentions something it should not.

Test it deliberately with real accounts at different levels. Ask a junior account questions designed to pull from restricted material and see what happens. It is a short exercise and it is the one that keeps the project out of serious trouble.

Beyond correctness, a few structural habits change results more than any amount of tuning. Give every document a title that says what it is and who it applies to. Keep one authoritative version of anything that exists in several places. Put the effective date inside the document rather than only in the file name, because the file name is usually gone by the time the text is indexed.

Break long documents at meaningful boundaries. A long handbook indexed as one lump retrieves badly, because the relevant paragraph is buried among thousands of irrelevant ones. Split at sections that stand on their own, and make sure each piece still carries the name of the document it came from.

And keep tables as tables. Pricing, specifications and eligibility rules flattened into prose lose the relationships that made them answerable in the first place. That is one of the most common reasons a system produces a confidently wrong figure, and it happens during ingestion, long before anything reaches the model.

Cleaning all of your data is not a project anybody finishes. Pick the narrow slice the first use case needs and clean that properly. If you are answering questions about deliveries, you need the delivery records, the identifiers that link them to customers, and the policy documents about delivery. You do not need the archive of old marketing material.

Order the work by what blocks what. Identifiers first, because everything joins on them. Then authoritative versions, because answer quality rests on them. Then permissions, before anybody outside the project team touches the system. Then structure and metadata, which keep improving results steadily and can carry on after launch.

Expect to find things more important than the project. Teams doing this routinely discover a reconciliation that has been quietly wrong, an integration that stopped running months ago, or a board report drawing from a source nobody maintains. Those findings often justify the spend on their own, and none of them are what anybody signed up for.

You will know the data work is done when a knowledgeable member of staff can answer a question the same way the system would, using the same documents, and arrive at the same answer. Where a person cannot do that, no system will manage it either.

The second test is traceability. For any answer, somebody should be able to say which document it came from and when that document was last reviewed. Being able to trace an answer back to a page with a date and an owner is what makes it defensible on the day a customer or an auditor disputes it.

Neither test involves the model at all. Both can be run before you have chosen one, and if they fail, choosing a better model will not rescue anything.