Data Engineering & Analytics

A BI tool draws charts from whatever logic you hand it, and it will draw eleven definitions of revenue as cheerfully as one. It is the layer that displays an answer, not the layer that decides what the answer means. In most organisations we look at, the meanings are scattered across saved queries, a few spreadsheets and two dashboards where somebody adjusted a filter in a hurry before a meeting. Moving that logic underneath the BI tool, into modelled tables and a single metric definition, is what ends the disagreement. It has a second benefit: changing BI vendor later becomes a few weeks of rebuilding views rather than a rebuild of your reporting, because the meaning of your numbers is no longer held inside one supplier's file format.

Landing data and being able to read it are different jobs. Ingestion answers whether the rows arrived, on time and unbroken. This work begins after that: shaping arrival into tables built around how the business thinks, deciding what counts as a customer, keeping history the source system overwrites, and producing the handful of reports people act on. It also pushes one conversation upstream, into a data contract with each source owner covering the fields your definitions depend on and the notice you get before one is renamed. Plenty of companies have working pipelines and still argue about the numbers in them, because nobody agreed the definitions. If your ingestion is already sound, this is faster and cheaper than usual.

Team size answers this more honestly than architecture posts do. If two or three people will look after it alongside other work, take a managed warehouse: BigQuery, Snowflake, Redshift or similar, where SQL is the only language anyone has to learn and there is very little to operate. A lakehouse earns its complexity when you have engineers who will own it, genuinely large or unstructured data, or a requirement to keep files in your own storage for cost or regulatory reasons. Choosing the more sophisticated option with nobody to run it is the common failure: it works during the build, then an upgrade lands eighteen months later and nobody left understands the cluster configuration. And if the whole company runs through one system and the team is small, the honest answer is neither, yet.

Two owners, and both get named before anything goes live. A business owner decides what the metric means and signs off any change to that meaning. A technical owner is responsible for the model and the job that produce it. When a figure looks wrong, the technical owner establishes whether it is broken or merely surprising, and the business owner decides whether the definition was the problem. Without those two names, a wrong number produces a week of polite email, no correction, and a report that quietly stops being opened. We also print the contact on the dashboard itself, next to the last refresh time. It sounds like a trivial detail and it changes behaviour more than most of the modelling does.

Because access was the easy half of the problem. People get a licence, a login and a list of tables, and the list contains four things that all look like orders. Not knowing which one finance uses, and reasonably unwilling to present a wrong figure to their director, they go back to asking somebody. Self-serve starts working when a small curated set is marked as trustworthy, when columns are described in the words the business uses rather than the source system's abbreviations, when the joins are already made, and when someone spends an hour showing people how to answer the five questions they ask most. Expect a proportion of your users never to touch it. That is acceptable, as long as analysts get their time back for new questions.

The evidence already exists. Every mainstream BI tool records views per dashboard, per user, over time, and turning that into a ranked list is an afternoon of work. What comes back is usually uncomfortable: a small group taking most of the traffic, a long tail viewed once by whoever requested it, and scheduled emails still arriving for former employees. The safe way to act on it is to switch off rather than delete. Turn off the refresh, leave the dashboard in place with a note explaining why, and wait a fortnight. If nobody says anything, delete it. If somebody complains within the hour, you have just found an owner and a genuine use, which is exactly what the exercise was for.

Often, and we would rather say so than sell you a pipeline. A spreadsheet is correct when the question is genuinely one-off, when the shape of the model changes every time you look at it, when fewer than about five people need it, or when the work requires human judgement typed in beside the numbers, which describes most planning and most budgeting. It becomes the wrong answer at an identifiable point: when other people start depending on it, when somebody rebuilds it on the same day every month, or when its figures turn up in decisions and nobody can trace where they came from. Those three signals are the moment to model it properly. Before then, building anything is slower and worse.

Three ongoing costs, and only one of them is a vendor invoice. There is platform spend, which for a mid-sized company is usually lower than expected and straightforward to attribute per job. There are BI licences, which are charged per seat and grow quietly, so it pays to check every few months who still logs in. And there is a person: somebody has to own the definitions, absorb the change when a source system renames a field, and retire reports that have lost their audience. That last one is the cost nobody budgets and the one that decides whether any of this still works in two years. We can hold it, hand it over documented so your team holds it, or share it. It does not reach zero.