From Field to Boardroom: Building ExxonMobil's Hydrocarbon Reporting Platform on Azure Databricks

In oil and gas, data that arrives late is data that costs money. ExxonMobil needed their hydrocarbon reporting to move at the speed of the business — not the speed of their legacy ETL pipeline.

Hydrocarbon reporting is one of the most data-intensive operations in the energy sector. Every barrel produced, transported, refined, and sold generates data across multiple systems — and that data needs to reconcile to the cent at month end. When those systems don't talk to each other, and when the glue holding them together is a collection of manually maintained Excel files and overnight batch jobs, the result is predictable: delays, errors, and engineers spending their Fridays debugging pipelines instead of building the future.

That was the situation ExxonMobil's data engineering team brought to CodeSizzler. Five source systems. No unified data layer. Reports that took four-plus hours to generate and still required manual sign-off before anyone trusted them. The mandate was clear: build a production-grade data platform that the business could actually rely on.

The Problem: Five Systems, No Single Source of Truth

Before we could design anything, we needed to understand what we were connecting. ExxonMobil's hydrocarbon data landscape spanned five distinct systems, each owned by a different team and running on a different refresh cadence:

• SCADA systems — High-frequency sensor telemetry from wellheads and pipelines. Critical for real-time operational data but notorious for clock drift, duplicate records, and late arrivals.
• SAP ERP — Financial and procurement records for production accounting and revenue recognition. High data quality but slow batch extraction windows and query sensitivity.
• Production Management System (PMS) — Reservoir and allocation data governing how production volumes are attributed to wells, leases, and business units.
• Lab Information Management System (LIMS) — Quality and composition data for crude grades, including API gravity, sulphur content, and energy density — essential for commercial reporting.
• Field engineer Excel workbooks — Manual exception and adjustment files maintained by field teams for volumes that didn't reconcile automatically. Inconsistent schemas, version control by filename date.

No single team had a complete picture of hydrocarbon production for any given day. Month-end close required a dedicated three-person reconciliation exercise that consumed the better part of a week. And because the reports were known to be imperfect, finance added manual review checkpoints that added days to the close cycle.

The Architecture: Azure Databricks Medallion Lakehouse

We chose Azure Databricks on ADLS Gen2 as the foundation — not because it was fashionable, but because the medallion architecture maps directly to how oil and gas data actually flows: from raw sensor telemetry, through business rule application, to finance-grade reporting. The three-layer structure gave us a clean separation of concerns that both engineering and finance could reason about.

The decision to put the Excel exception files through the same Bronze → Silver → Gold pipeline as the structured systems was deliberate. By treating them as first-class data sources rather than one-off imports, we ensured that field engineer adjustments carried full audit lineage from workbook cell to finance report — and that they could be replaced by structured system data as ExxonMobil's upstream systems matured.

Unity Catalog: Governance That Finance Actually Trusts

Data quality is only half the battle in a regulated energy company. The other half is demonstrating to finance, compliance, and external auditors that the data is what it claims to be — and that only the right people can see the right data. Unity Catalog addressed this directly:

• Column-level security on financial data — Revenue, margin, and contract price columns in Gold tables are restricted by role. Field operations teams see volumes; finance teams see volumes and value.
• Row-level security for country and business unit — Production data is siloed by operating region at the data layer. Users in the Gulf operations team cannot query North American production records, regardless of which tool they use.
• Data lineage from SCADA to boardroom — Unity Catalog tracks the full journey of every data point — from raw SCADA record in Bronze, through Silver transformation, to the Gold table cell that appears in the executive dashboard. Auditors can trace any figure back to source in seconds.
• Audit logs for regulatory compliance — Every query, every access, every schema change is logged with user identity, timestamp, and query text. The logs are immutable and retained per ExxonMobil's regulatory retention schedule.

CI/CD: Pipelines That Don't Break at Month End

The old ETL pipelines had no automated testing. A schema change upstream could silently corrupt downstream reports without anyone noticing until the Monday morning reconciliation run. We replaced the entire deployment lifecycle with a Git-based CI/CD workflow:

• Feature branch development in Azure DevOps — Every pipeline change — new source integration, transformation rule update, or schema change — goes through a pull request with a required code review before it can merge to main.
• Automated unit tests on all transformation logic — PySpark transformation functions are tested against synthetic data covering edge cases: SCADA late arrivals, SAP duplicate invoices, LIMS null quality readings, and cross-unit volume mismatches.
• Delta Live Tables pipeline validation in staging — The full DLT pipeline runs against a staging environment with a representative data slice before any merge to main. Data quality expectations are enforced as first-class pipeline constraints, not afterthoughts.
• Automated production deployment on merge to main — Merge to main triggers a GitHub Actions workflow that deploys the updated DLT pipelines to the production Databricks workspace, updates the Unity Catalog schema if required, and sends a deployment notification to the data engineering Slack channel.

Three Technical Decisions That Made the Difference

Any Azure Databricks engagement involves dozens of small architectural decisions that don't show up in the architecture diagram. These three had the largest impact on the final platform's reliability and performance:

ROI: What the Numbers Actually Showed

ExxonMobil's data engineering leadership asked us to be conservative with projections before the build. Eight weeks post-deployment, the actuals were ahead of every estimate:

Metric	Before	After
Report generation time	4+ hours	90 minutes
Data freshness	T+24 hours	T+3 hours
Manual reconciliation effort	~12 hrs/week (3 engineers)	~0 — fully automated
Month-end close cycle	5–7 business days	2–3 business days
Annual engineering hours reclaimed	—	480 hrs

The 480 engineering hours recovered annually didn't disappear from the budget — they were redirected to the next phase of the data platform, compounding the return on the original investment.

The harder-to-quantify benefit was organisational trust. Finance stopped adding manual review checkpoints to the report cycle because they trusted the data. That trust took three months to build — and it was built on Unity Catalog audit lineage, not on a slide deck.

Lessons Learned: What We'd Tell the Next Team

Every engagement surfaces things that no architecture document could predict. Here are the five we'd pass on to any team embarking on a similar build:

• Fix SCADA clock drift at ingestion, not at transformation. Field devices don't always agree on timestamps — different firmware versions can be minutes apart. Normalise everything to UTC at the Bronze layer before any join logic runs, or you'll spend weeks debugging phantom duplicates in Silver.
• Never query SAP directly from Databricks. Extract to ADLS Gen2 first via SAP Data Services or an RFC-based connector, then read the files. Direct JDBC connections to SAP from Databricks are fragile under load and can trigger SAP's RFC connection throttling in production environments.
• Treat Excel exception files as first-class data. The instinct is to handle them as a temporary workaround until the proper system is fixed. In energy, "temporary" often means five years. Invest in proper schema validation and version tracking for these files from day one — your future self will thank you.
• Build realistic synthetic test data before you build the pipeline. Generic test data won't catch the edge cases that actually occur in hydrocarbon calculations — negative volume adjustments, cross-border allocations, retroactive price corrections. Work with a domain expert to build test cases that reflect how the data actually behaves.
• Roll out Unity Catalog column-level security before row-level. Column-level security is easier to validate — you can test it by querying a table and confirming specific columns are hidden. Row-level is harder to verify comprehensively. Get column-level working and trusted before adding the additional complexity of row filters.

Why Azure Databricks for Oil & Gas?

There's a reason the Medallion architecture + Unity Catalog combination has become the reference pattern for regulated data platforms in energy. Oil and gas data is inherently multi-source, multi-frequency, and regulation-sensitive — and those three properties push the platform requirements in directions that traditional data warehouses struggle to satisfy.

The ability to enforce column and row security, maintain complete audit lineage from sensor to report, and serve both operational and analytical workloads from the same Delta Lake without data duplication is a genuine differentiator. ExxonMobil's finance, operations, and compliance teams now query the same Gold tables — with access controlled by Unity Catalog policy rather than by maintaining separate reporting databases per audience.

That's not an architecture diagram selling a dream. That's what the system has been doing in production for the last two quarters.