How you'll make an impact
- Design and build production pipelines on Databricks using Spark Declarative Pipelines (SDP) and PySpark, from raw ingestion through business-ready data products.
- Define all pipelines, jobs, and schedules as code in Databricks Asset Bundles, deployed to every environment through automated CI/CD.
- Build data quality, monitoring, and lineage into pipelines so issues are caught and diagnosed before they reach consumers.
- Turn recurring solutions into reusable frameworks, standards, and shared libraries that raise delivery speed across the team.
- Own your pipelines in production — performance, cost, reliability, and incident response.
- Partner with Digital Product Managers, architects, and business stakeholders to translate requirements into technical designs, and mentor engineers newer to the platform.
What you'll need (Required)
- Bachelor's degree in computer science, engineering, or a related technical field, plus five or more years of data engineering experience, including hands-on production experience on Databricks.
- Demonstrated experience with Spark Declarative Pipelines (SDP / Delta Live Tables) — streaming tables, materialized views, expectations, Auto Loader, and CDC patterns.
- Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments.
- Strong Spark and PySpark skills, including performance tuning, and production-quality Python beyond notebook scripting.
- Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL.
- Experience with Git-based CI/CD in a shared repository — code review, automated validation, and promotion across environments.
- Demonstrated ability to take ambiguous requirements through design to production independently, and to make and defend sound technical decisions.
- "Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication."
- "Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads."
What else we look for (Preferred)
- Hands-on experience with Spark Declarative Pipelines (SDP / Delta Live Tables), including streaming tables, materialized views, expectations, Auto Loader, and CDC patterns.
- Hands-on experience deploying Databricks workloads with Databricks Asset Bundles (DABs) across multiple environments.
- Strong Spark and PySpark development skills, including performance tuning, and production-quality Python beyond notebook scripting.
- Working knowledge of Delta Lake, Unity Catalog, medallion architecture, and strong analytical SQL.
- Experience with Git-based CI/CD for data platforms, including code review, automated validation, and promotion across environments.
- Experience with cloud data platforms on AWS, metadata-driven ingestion frameworks, and infrastructure-as-code.
- Experience with interoperable catalog architectures across Unity Catalog and Snowflake Horizon, including Iceberg REST Catalog and catalog-linked databases for cross-platform table access without data duplication.
- Working knowledge of Apache Iceberg as a table format, including managed versus external Iceberg tables and the performance trade-offs of cross-engine reads.
- Familiarity with the broader modern data ecosystem, such as Snowflake, dbt, Kafka, and Airflow.
- Experience integrating enterprise and clinical source systems, such as Epic, SAP, or Salesforce, or migrating workloads from legacy ETL platforms onto a lakehouse.
- Practical understanding of governance, quality, security, validation, and support expectations in a regulated enterprise environment.
- Ability to provide technical guidance, coach team members, and contribute to reusable standards, documentation, and delivery practices.
