Connecting Databricks to CostReveal, setup guide

Databricks spend is hard to read for one reason above all: the same usage can be invoiced by Databricks itself or by your cloud, and the right way to show it depends on which. This guide covers what CostReveal reads from your Databricks account's system tables, how the figures differ by who invoices you, each setup step in order, how a cloud's Databricks charges are divided among jobs and owners, and what reading costs you.

What the Databricks connection reads

CostReveal reads your Databricks account's billable usage, every workspace and region, day by day, from Databricks' system tables:

• Usage, from system.billing.usage: each day's usage by workspace, SKU and product, such as jobs, SQL, all purpose compute, pipelines and model serving, and the job, pipeline, SQL warehouse, serving endpoint, app or cluster it ran on. Corrections Databricks makes later are recorded as corrections.
• List prices, from system.billing.list_prices: the price in effect when the usage started.
• Tags: the custom tags on clusters, jobs, warehouses, pipelines and budget policies.
• Workspace names, from system.access.workspaces_latest, when you grant it.

It reads usage and prices, never your notebooks, tables or data.

Who invoices you decides the figures

This is the most important choice in the setup, and it is fixed for the connection.

Databricks invoices you. Databricks does not share what it invoices, so CostReveal prices your usage at list price. These are estimates, before any discount in your contract. Home says so, and shows no second source for them, because there is none.

A cloud invoices you. Azure Databricks, Databricks on Google Cloud, and Databricks bought through AWS Marketplace are on that cloud's invoice. Their spend is therefore already in the cloud's connection, exactly as invoiced. Connect that cloud in the same project too. CostReveal then divides the cloud's Databricks charges by the list cost of the usage they bill for, each day on its own, and adds nothing to them, so nothing is counted twice.

Under the Databricks connection, the latest month is shown by workspace, product and resource either way.

What you need

• Access to the Databricks account console, to add a service principal and its federation policy.
• A metastore admin, to run the system table grants.
• A SQL warehouse the service principal can use.
• An owner or admin role in CostReveal.

Step by step setup

1. Start the connection

In CostReveal, go to Connections and choose Connect next to Databricks. CostReveal makes a key for the connection.

2. Add a service principal

In the account console, under User management, add a service principal, and add it to a workspace.

3. Create the federation policy

On the service principal, under Credentials & secrets, Federation policies, create a policy with the issuer, subject, audience and JWKS CostReveal shows, or create it with the Databricks CLI command shown. The service principal then trusts CostReveal's key, and no client secret is ever created or exchanged.

4. Grant read access, as a metastore admin

As a metastore admin, run the grants CostReveal shows in the SQL editor, which let the service principal read the billing system tables, and give the service principal Can use on a SQL warehouse.

5. Enter the details and verify

Enter the workspace URL, the service principal's application ID and the warehouse ID, and say who invoices you. Choose Verify and connect. Only the format is checked as you type. When you verify, CostReveal signs in and reads the account's usage before saving anything. If something is missing, it says which: a policy that does not match, a warehouse that is not there, or grants that are missing.

No password or secret of yours changes hands. Deleting the federation policy, or the service principal, ends access at once.

How a cloud's Databricks charges are divided

When a cloud invoices you, CostReveal finds the Databricks charges on that cloud's bill and divides them by what your Databricks usage says:

• Found on the bill by what each cloud says of them: an Azure Databricks workspace's own resource, a Google Cloud service named Databricks, or an AWS Marketplace product named Databricks.
• Divided by usage. On Azure, each workspace's charge is divided by that workspace's usage, when its name is granted. Otherwise, and on the marketplaces, by the whole account's usage that day.
• Owners named by the usage's tags are allocated their share of what the cloud line's own tags left unexplained, with the basis Allocated. A day with no usage read yet is shown as not divided.
• Claimed once. A Databricks charge is never also shared by a split rule or by usage data. A Kubernetes cluster's billing tag claims a line first.

Freshness, history and what reading costs

Databricks records usage within hours, and corrects it later. CostReveal reads every six hours, and reads the current and previous month each time; the first read also brings in the twelve months before the current one, as far as Databricks holds them. Where Databricks holds no older months, Connections says how far back it reached, and each read asks again. A month is summarized first, and read in full only when its usage has changed. Reading takes a few seconds a day of the SQL warehouse, which Databricks charges you for, as for any warehouse.

Connection states

Under Connections, the connection is always in one of four states:

• Setup not finished: the details have not been verified yet.
• Waiting for first data: everything was verified, and the first read is due.
• Connected: usage was read and the latest read succeeded.
• Needs attention: the latest read failed, for example because the federation policy was deleted. Everything already read is kept.

How Databricks spend is explained

• Direct: the service, which comes from the bill or the usage itself.
• Inferred: the team, feature and customer, when a custom tag on a cluster, job, warehouse, pipeline or budget policy names them, read with the project's tag keys. A job tagged team = payments is the payments team's.
• Allocated: a cloud's Databricks charge divided among owners by their usage, as above, or a share an allocation rule divides.
• Unknown: spend no tag, usage or rule explains. It is shown as unknown, never spread across teams.

Names never explain spend on their own: a job called payments-etl is not the payments team's until a tag or a rule says so. Budget policies are a practical place to set tags once for many workloads.

How figures are checked

Usage and list prices are recorded as Databricks states them, and a correction Databricks makes later is recorded as a correction that keeps what was said before, before attribution runs again. Attribution on every dimension must then add up exactly to the ledger.

For accounts Databricks invoices, there is no second source: the figures are list price estimates, and Home says so. For accounts a cloud invoices, the amount is the cloud's own invoiced figure, checked where that cloud's connection is checked.

Getting more of Databricks spend explained

1. Tag jobs, warehouses, pipelines and clusters with a key the project reads, such as team.
2. Set tags on budget policies, so serverless and policy governed workloads carry them without tagging each one.
3. Grant the workspace names, so Azure Databricks charges are divided per workspace.
4. Connect the cloud that invoices you, in the same project, if it is not connected yet.

Troubleshooting

• Verify says the policy does not match. Check the federation policy's issuer, subject, audience and JWKS against what CostReveal shows.
• Verify says grants are missing. Have a metastore admin run the grants shown, and give the service principal Can use on the warehouse.
• Cloud invoiced Databricks shows as not divided. The day's usage has not been read yet, or the cloud's connection is not in the same project.
• Changing who invoices you. It is fixed per connection: remove the connection and connect again.

Frequently asked questions

Why are my Databricks figures called estimates? When Databricks invoices you, it does not share what it invoiced, so usage is priced at list price, before any discount in your contract. Home says so.

I pay for Databricks through Azure. Is it counted twice? No. Its spend is on the Azure invoice and comes through the Azure connection. CostReveal divides those charges by your Databricks usage and adds nothing to them.

Does CostReveal read my data or notebooks? No. It reads billing system tables: usage, list prices, and workspace names when granted.

What does reading cost? A few seconds a day of the SQL warehouse you name, billed by Databricks like any warehouse.

How fresh are the figures? CostReveal reads every six hours. Databricks records usage within hours and corrects it later; corrections are recorded as they land.

Can I change who invoices the account later? Not on the same connection. Remove it and connect again with the other choice.

Keep reading

• Connecting Azure to CostReveal: where Azure Databricks charges arrive.
• Connecting Snowflake to CostReveal: warehouse charges divided by credits.
• Connect Databricks in the docs: the reference for every setup step.