Why I chose this topic: In financial services, "Role-Based Access Control" (RBAC) is a ticking time bomb that inevitably explodes once your headcount hits double digits. I’m writing this because I spent three weeks cleaning up a PII leak caused by a developer who "just needed" a role that had too much power, and I never want to do that again.
The PagerDuty alert hit at 3:14 AM on a Tuesday: HIGH PRIORITY: Data Governance Violation - Unauthorized PII Exposure.
My logs showed that a junior analyst running a notebook in a sandbox environment had successfully queried the transactions_prod table. They shouldn’t have seen the customer_ssn or credit_card_number columns. But there they were, printed out in cleartext in the stdout of a cluster that had no business touching production data. The cost? A mandatory report to our DPO, a week of manual audit logs review, and a very awkward conversation with our internal compliance team.
What we saw
The initial panic was centered on the GRANT statements. We checked sys.access in the Unity Catalog (UC) metastore. Everything looked correct. The analyst’s user group, analyst_sandbox_role, did not have SELECT on the transactions_prod table.
We chased shadows for four hours. We checked if the user had elevated their privileges via a service principal. We looked for rogue GRANT commands in the audit logs. We even scrutinized the cluster configuration, wondering if someone had bypassed the spark.databricks.passthrough.enabled setting.
Nothing. The permissions were locked down. The user technically didn't have access. But they still had the data.
Photo by Josh Snader on Unsplash
Root cause
The culprit wasn't a bypass of the security layer; it was a "Role Explosion" bug combined with a classic "God-mode" service principal.
We had an automated pipeline that synced our Active Directory (AD) groups to Databricks. The analyst was part of a group called global_data_science. That group had been granted SELECT access to a view called transactions_masked. This view, however, was built on top of a table where column-level security had been misconfigured.
Specifically, the view relied on a hard-coded CASE statement to nullify PII. When a senior engineer updated the underlying table schema to add a new PII field, they forgot to update the SELECT clause in the view. The view effectively defaulted to "show all" for any new columns added to the base table. Because the global_data_science role inherited broad read permissions from a legacy environment, the analyst just queried the underlying table directly.
Our RBAC model was brittle. We had thousands of granular roles, and the logic was "who you are" rather than "what the data is." We were managing permissions like we were still using SQL Server 2008.
Photo by Morthy Jameson on Unsplash
The fix
We nuked the old role-based approach and moved to Attribute-Based Access Control (ABAC) using Unity Catalog’s TAG system and dynamic masking functions.
Instead of managing who gets to see what table, we defined a policy based on the data’s properties. We tagged the sensitive columns in our UC catalog:
ALTER TABLE transactions_prod ALTER COLUMN customer_ssn SET TAGS ('pii' = 'high');
ALTER TABLE transactions_prod ALTER COLUMN credit_card_number SET TAGS ('pii' = 'high');
Then, we wrote a standard User-Defined Function (UDF) that masks data based on the presence of these tags. The core of the fix was applying a MASK to the column, which is evaluated at query time:
CREATE OR REPLACE FUNCTION mask_pii(col STRING)
RETURN CASE WHEN IS_ACCOUNT_GROUP_MEMBER('pii_authorized_team') THEN col
ELSE '***-**-****' END;
ALTER TABLE transactions_prod ALTER COLUMN customer_ssn SET MASK mask_pii;
Now, the permission isn't attached to the user’s role; it’s attached to the data attribute itself. If you aren't in the pii_authorized_team, the UDF fires automatically. It doesn't matter what role the user has, what notebook they are using, or if they are on a sandbox or production cluster. The UC engine enforces the function at the metadata level.
What we changed so it never happens again
We stopped thinking about "permissions" and started thinking about "data contracts."
First, we implemented a mandatory tag-enforcement policy. We use the Databricks Terraform provider to ensure that any new table created in the prod catalog must have a sensitivity tag. If a developer tries to create a table without tagging columns as public, internal, or pii, the CI/CD pipeline fails the deployment. You cannot create a table in production without explicitly telling the system what kind of data lives inside it.
Second, we moved away from managing access via individual role membership. We use IS_ACCOUNT_GROUP_MEMBER inside our masking UDFs. This decouples the security policy from the underlying object-level permissions. Even if someone accidentally grants SELECT access to the transactions_prod table to an intern, the masking UDF will still render the PII as ***-**-****. The data is protected by its metadata, not by the user's role.
Third, we automated our audit process. We use the Databricks system.access logs to monitor for any queries that trigger the masking function. If an unauthorized user attempts to query a column tagged with pii='high', an alert is sent to Slack. We don't wait for a compliance audit to tell us we're bleeding data; we see the attempts in real-time.
The biggest lesson here is that in complex financial environments, RBAC is a maintenance nightmare that scales linearly with chaos. When you add a new team, you have to update a dozen roles. When you add a new column, you have to update a dozen views.
ABAC forces you to classify your data once. Once the data is classified, the security policy is global and immutable. The developer can’t "forget" to include a column in a mask because the mask is tied to the column attribute, not the SQL view definition.
I’d rather spend three days setting up a robust tagging strategy than three hours explaining to a regulator why the PII is sitting in a text file on a sandbox cluster. If your security model requires you to remember to do something, it’s already broken. Move to attributes, let the catalog do the heavy lifting, and get some sleep.
Cover photo by Albert Stoynov on Unsplash.
