The Opportunity:

As a Senior Data Engineer, you will design, build and operate scalable data platforms and pipelines for global customers. You will work closely with customers, product teams and engineers to deliver reliable, production-grade data systems. You will play a key role in shaping data architecture, data engineering best practices and AI-driven data platforms at Digit88, enabling customers to move from raw data to actionable insights and intelligent systems.

Key Responsibilities:

- Modernize legacy, non-scalable architectures and define a scalable target-state platform

- Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake and Databricks components such as Delta Live Tables (DLT), Delta Sharing and Workflows (LakeFlow, LakeBase)

- Design and build scalable, reliable and production-ready ETL/ELT pipelines using PySpark, SQL and Databricks notebooks to ingest and transform data from diverse sources.

- Create and manage workflows using Databricks Workflows (Jobs) or orchestration tools to automate pipelines and dependencies

- Optimize Spark jobs for performance, scalability and cost efficiency (partitioning, caching, query tuning, cluster optimization)

- Implement data quality checks (e.g., DLT Expectations) and enforce governance via Unity Catalog (access control, PII masking, lineage)

- Design and implement event-driven and streaming pipelines (Kafka or equivalent)

- Ensure high data reliability through monitoring, observability and alerting

Requirements:

- BE/MS in Computer Science or a related field with 8+ years of experience in data engineering

- Strong experience in ETL/ELT pipelines, data modeling, and distributed data systems, with hands-on expertise in Databricks

- Deep proficiency in PySpark, including performance optimization, job orchestration, and large-scale data processing

- Good understanding of event streaming systems such as Kafka or equivalent technologies

- Experience working with Azure data ecosystem (ADLS, Azure services, AHDS or similar data platforms)

- Strong foundation in event-driven architecture and scalable distributed systems

- Proven ability to design, review, and clearly articulate system architecture

- Experience leveraging AI-assisted development tools (e.g., Claude, Antigravity, etc.) to improve productivity

- Solid experience in Agile delivery, estimation, and program execution

- Experience working with global customers (US/EU) in a client-facing role

- Excellent written and verbal communication skills across engineering, business, and customer stakeholders

- Strong analytical thinking and structured problem-solving ability

- High ownership, reliability, and execution focus; consistently delivers despite constraints

- Strong attention to detail while maintaining a clear big-picture perspective