Sr. Data Platform Operations Lead

About the role

Responsibilities

• Own end-to-end incident and problem management for the data platform: detection, triage, resolution, and root-cause analysis across pipelines, jobs, and the downstream consumers.
• Own job-failure remediation and recovery, restoring critical data flows within targets calibrated to the close and investor-reporting calendars.
• Define and operate data freshness and data-quality monitoring, alerting, and SLA tracking against reporting deadlines.
• Author and maintain runbooks, escalation paths, and on-call procedures; lead post-incident reviews and drive corrective actions to closure.
• Carry on-call responsibility on a severity-based basis, with heightened coverage around quarter-end close and investor-reporting cycles.
• Own day-to-day operations of the Azure Databricks Lakehouse: cluster policies, instance pools, SQL warehouse sizing, serverless configuration, and runtime currency.
• Maintain workflow and Delta Live Tables pipeline reliability and performance at the platform level.
• Note: Unity Catalog data-object privilege administration is owned by Data Stewardship and Data Engineering; this role partners on the operational impact but does not own catalog grants.
• Own end-to-end FinOps for the data platform: resource tagging, budget policies, and consumption monitoring via Databricks system tables, and keeping spend predictable and defensible.
• Deliver cost transparency and chargeback/showback by business line, giving Fund Accounting, Portfolio Oversight, and other consumers visibility into what they drive.
• Right-size compute (cluster policies, instance pools, SQL warehouse and serverless configuration) and drive continuous optimization of the platform's cost-to-serve.
• Forecast platform spend, flag variances early, and partner with Finance and the Head of Enterprise Data Management on the platform budget.
• Administer the DevOps toolchain for data — GitHub organization, repositories and access, CI runners, and pipeline secrets — operating with the advanced / administrator privileges these systems require.
• Own and deploy CI/CD for data assets — GitHub Actions pipelines and Databricks Asset Bundle-based deployment of notebooks, jobs, DLT pipelines, and platform objects across dev, test, and production.
• Build and administer the Infrastructure-as-Code toolchain (Terraform with the Databricks provider, and Azure IaC for surrounding resources), and own the environment-promotion and release-management model.
• Establish the observability strategy — monitoring, logging, alerting, and data-quality gates — and hold the practice to it.
• Set the deployment standards and review gates the wider Data Engineering team builds within, and make decisive calls on what gets promoted to production.
• Own Databricks account and workspace administration and partners with Infrastructure team for provisioning, the account console, identity federation (SCIM and SSO via Entra ID), and group and entitlement management.
• Partner with Infrastructure on platform security — Private Link and network connectivity, secret scopes, token governance, and exfiltration controls.
• Maintain deep conceptual command of these areas to translate operational needs into infrastructure requirements and to resolve platform issues that cross the operations / infrastructure boundary.
• Drive outcomes across a matrixed organization — Infrastructure, Data Engineering, Fund Accounting, and Portfolio Oversight — setting priorities and securing alignment without sole positional authority.
• Make decisive operational and platform decisions.
• Coordinate platform change with the firm’s data-governance and change-management processes rather than operating in parallel.

Requirements

• 8–12 years in data platform operations, DataOps, or a closely related discipline, with demonstrated hands-on production-support ownership.
• Strong, current Databricks platform operations experience — compute, cost and FinOps governance, SQL warehouses, workflows/DLT, and runtime management on Azure.
• Hands-on experience administering and deploying CI/CD and DevOps tooling (e.g., GitHub / GitHub Actions), Infrastructure-as-Code (Terraform), and observability tooling for a data platform.
• A proven track record of partnering with an infrastructure / cloud team on identity, networking, and platform security.
• Evidence of driving cross-functional outcomes in a matrixed environment and exercising sound, decisive judgment under operational pressure.
• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent practical experience.
• Experience in financial services, alternative investments, or real estate / infrastructure asset management, with familiarity with close cycles and investor-reporting calendars.
• Databricks Academy Accreditation – Platform Administrator, and/or Databricks Certified Data Engineer (Associate or Professional).
• Microsoft Certified: DevOps Engineer Expert (AZ-400).
• HashiCorp Certified: Terraform Associate.
• Experience operating BI / reporting platforms such as Domo as downstream consumers of a data platform.