FinOps for Data Platforms

Cost and usage data is built from heterogeneous sources: billing exports, query and job execution history, compute utilisation metrics, storage metadata, orchestration logs. These sources vary in structure, frequency and granularity, and reflect transient workloads on shared resources: infrastructure data alone is not enough to understand cost drivers or attribute them correctly.

Allocation itself changes nature: where Public Cloud relies on accounts, subscriptions and persistent tags, data platforms require execution-based attribution, correlating billing with workload telemetry (query history, job or pipeline metadata, workspace or project identifiers). When that metadata is missing, physically separating compute resources by team or workload can improve attribution clarity, at the price of lost efficiency and greater operational complexity.

Day to day steering relies on a set of metrics specific to this scope: virtual currency consumption, compute utilisation rate per warehouse or cluster, volume and duration of query and pipeline execution, concurrency and autoscaling behaviour, storage growth and retention trends, cost per query or per pipeline, and burn-down tracking of commitment against contracted capacity.

Anomaly detection follows a different logic from Public Cloud: a sudden spike in consumption per query, an unexpected concurrency peak, an inefficient query scanning an entire table with no history to justify it, repeated retries of failed pipelines, an unanticipated data refresh or reload, each of these signals, invisible in an aggregate reading of the invoice, only becomes detectable by correlating billing, workload telemetry and orchestration logs.

Choosing a data platform and how it is deployed is an architecture decision in its own right, not a simple tooling choice. Cost matters, but data integration and movement, governance and security constraints, and the ability to control cost over time weigh just as much in the decision.

The choice of operating model, hub-and-spoke, federated, consolidated or hybrid, determines where data and compute reside, who carries cost responsibility, and the level of duplication and data movement introduced by the architecture. Add to that the separation of compute and storage, the management of data freshness (refresh frequency, duplication, reuse via catalogue), and the early integration of regulatory constraints (data residency, compliance, encryption), which become markedly more expensive to correct after the fact than to anticipate at design time. A pilot on a representative scope, before scaling out, validates these cost assumptions before they apply across the organisation.

Optimisation focuses first on data engineering itself: warehouse or cluster sizing, auto-suspend and auto-terminate policies, concurrency management, and above all query-writing quality. Inefficient joins, avoidable full table scans, retries of failed pipelines consume compute units without generating value, a reserve that stays invisible until visibility reaches the individual query level.

On the storage side, growth results from retention, replication, snapshot and duplicated dataset choices that are rarely audited once the initial architecture is in place. Lifecycle policies aligned with the real business value of the data, rather than with default retention, contain that growth without compromising governance requirements.

The SaaS and licensing dimension of this scope deserves specific vigilance: platform plans or editions, support tiers, and third-party connectors (ETL, BI, observability, governance) billed per pipeline, per API call or per seat, often accumulate without consolidated review. Checking the fit between the subscribed entitlement level and the real needs of the workload avoids paying for premium features that go unused.

Sustainability is measured here mainly through supplier emissions declarations, most often in Scope 3, given the absence of direct measurement of the underlying infrastructure. The failure and retry rate of processing, the intensity of inter-region data movement, and the alignment of workload placement with the organisation's sustainability objectives are the concrete levers a FinOps governance function can act on, drawing on the transparency and commitments declared by each provider.