This post was written by Assa Efraim, Director of Data at Start.io.

Data Observability in Big Data Environment

At Start.io, we deliver a big data solution to partners and internal stakeholders alike. Big data opens doors, but it also demands constant, evolving care from data teams. Even loading aggregated data at low granularity into a BI tool is a mess — the dataset is still huge, and it often includes accounts, apps, and partners that haven’t been active or even observed in a long time. Traditional OLAP databases, like our Vertica clusters, sidestep the pricing volatility of consumption-based cloud warehouses — but they trade it for a different cap: scaling out means buying and provisioning more servers, and that cost curve is steep and discrete, not smooth. It’s not just a budget problem; it’s structural. Traditional engines scale in fixed increments, so a query that needs 2 nodes of compute ends up consuming 4, 8, or 16, because those are the only sizes available. On the consumption-based cloud side, the same structural mismatch shows up differently: elastic infrastructure creates elastic costs, and industry estimates suggest enterprises waste 30–50% of cloud data warehouse spend on idle compute and over-provisioning. Different pricing models, same root issue: real-time querying at scale strains both approaches.

Moving to DWH layer and Operational BI layer

In recent years, a new standard has been set: a hybrid approach, pairing the traditional DWH layer with an Operational BI layer that exposes partial information in near real-time, scoped to a specific business need.

Why does this matter? Because a single DWH-only model forces every question through the same slow, fully-joined, semantically heavy path, even when the business just needs a quick, narrow answer right now.

The DWH layer keeps doing what it does best: heavy, cross-joined dashboards, built on a full semantic layer, refreshed on a schedule rather than instantly. The Operational BI layer sits alongside it, built for a different job entirely — often without the full web of business logic and cross-table connections that the DWH provides. That’s a deliberate tradeoff: less completeness in exchange for freshness, so operational, in-the-moment decisions aren’t stuck waiting on stale, batch-cycle data.

How to properly use an Operational BI layer?

Using an Operational BI layer well starts with discipline, not infrastructure. Focus on the business use cases that actually affect day-to-day operations. Define the business questions first, then derive the data from them — building small, purpose-built tables that answer those questions directly, rather than exposing broad, general-purpose datasets.

Access to these tables should require applying granular filtering — customerId, accountId, date, and similar scoping parameters — so that every query touches a small, bounded slice of data. This isn’t a performance nice-to-have; it’s the whole point – limiting the number of rows a query returns lets the engine skip unnecessary processing and cuts the amount of data sent over the network (source). Full traffic snapshots, cross joins, and heavy semantic logic belong in the traditional DWH, where they surface hours or days later. The Operational BI layer trades that completeness for speed: it holds only fresh, recent history, and it’s built to support quick, tactical steps — not long-term analysis.

No company builds its strategy on what’s happening this minute, and no operational team can wait days for an answer. That’s the real distinction underneath the DWH/Operational BI split: the DWH layer serves long-term, strategic, historical decisions; the Operational BI layer serves immediate, tactical, in-the-moment ones.

Does AI kill BI?

This isn’t the first time BI has been declared dead.

So the short answer: no. Our BI team is transforming and embracing the change AI brings, not resisting it. Agents can react with dashboards, but a better approach is providing the right access to the primitives behind them: the governed tables, definitions, and boundaries that make a query trustworthy. In our case, we use agents to query the Operational BI layer, delivering the freshest, most relevant data to account managers and adOps teams — scoped to their specific accounts and partners — without an analyst or BI developer in the loop.


We’re moving deliberately, though. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value, or inadequate risk controls (source). That’s exactly why we’re focusing first on data readiness — building the right tables and access patterns before scaling out — rather than bolting agents onto whatever happened to exist already. As BI leaders increasingly put it: the job now is building guardrails, not barriers.

Our Fundamentals for AI/BI Ecosystem

This is exactly the flag DBAs raise, and rightly so. An agent isn’t a quicker, more polite version of a human analyst – think of it as a superhuman user, capable of writing and executing hundreds of queries in seconds, then deciding on its own whether more are needed. Left unbounded, that’s not just a UX risk; it’s a direct path to an overloaded queue and a DBA’s worst afternoon.

So we built the Operational BI tables with bounding in mind from the start. Every query requires parameters — customerId, accountId, date, and similar filters — injected before execution, so nothing runs unbounded. No account manager wants to wait 20 minutes for a query to return a response to the agent. No DBA wants to find the database buckling under a heavy-lifted, agent-issued scan. Guardrails sit on top of this: we cap the size of the returned dataset, since a large result set doesn’t just mean a heavier query — it means burning through tokens unnecessarily. This isn’t a novel idea, either — one guardrail library for agents querying BigQuery takes a similar approach, running a dry-run cost check before execution (source) — an option we haven’t adopted ourselves, but a validation that bounding by design, not by convention, is where the industry is heading.

On top of the bounded queries, we built a set of modular skills that guide how Claude visualizes outcomes and tells the story behind the data — one skill for showing a trend over time, another for flagging an anomaly, another for comparing two accounts side by side, and so on. Writing these skills is the BI developer’s job; recognizing which one fits a given question, and applying it, is the agent’s. The same split holds for insights: BI developers define what counts as a meaningful insight in a given business context, and the agent’s job is to extract it from the data and surface it to the user as a suggestion — not a final decision. And the best part is that all of it happens in natural language, using Claude’s skills.

The BI team hasn’t stopped building high-level dashboards; if anything, that development cycle keeps improving with Claude and MCPs in the loop. But BI’s role has expanded. It’s no longer just the team producing dashboards — it’s the team orchestrating agents that run directly against company production data on behalf of the business.