Senior Data Engineer - Databricks & AWS

Job Description :

We are looking for a Senior Data Engineer with 5+ years of experience in designing, developing, and operating scalable data platforms and production-grade data pipelines.

The ideal candidate will have strong hands-on expertise in Python, SQL, PySpark, Databricks, AWS, and modern data lakehouse technologies, along with exposure to healthcare data standards and privacy requirements.

The role involves building resilient data ingestion and processing pipelines, optimizing distributed workloads, implementing data quality and observability controls, and contributing to production-grade data engineering practices.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines and data processing solutions using Python, SQL, Apache Spark/PySpark, and Databricks.

- Build and operate production-grade pipelines using Databricks Auto Loader, Delta Lake, Workflows, Unity Catalog, Notebooks/Jobs, and SQL Warehouses.

- Optimize Spark and Databricks workloads, including cluster configuration, job performance, query optimization, and distributed processing.

- Develop cloud-native data solutions using AWS services, including S3, IAM, VPC, ECS/EKS or Lambda, Step Functions, EventBridge, CloudWatch, Secrets Manager, and KMS.

- Work with Apache Iceberg, including tables, catalogs, snapshots, schema and partition evolution, compaction, and interoperability with different query engines.

- Design resilient ingestion frameworks for structured and semi-structured data, with appropriate validation, data quality checks, error handling, monitoring, and replay mechanisms.

- Implement engineering best practices around Git, pull requests, CI/CD, Docker, Infrastructure as Code, automated testing, observability, and incident response.

- Collaborate with cross-functional teams to understand data requirements and translate them into scalable technical solutions.

- Ensure appropriate security, access controls, data governance, and privacy measures are incorporated into data engineering workflows.

- Contribute to the responsible adoption of AI-assisted coding tools, critically reviewing, testing, securing, and productionizing AI-generated code.

Required Skills & Experience :

- 5+ years of overall experience in Data Engineering.

- Advanced proficiency in Python and SQL.

- Strong hands-on experience with Apache Spark/PySpark, distributed processing, and performance tuning.

- Production experience with Databricks, including :

1. Auto Loader

2. Delta Lake

3. Databricks Workflows

4. Unity Catalog

5. Notebooks and Jobs

6. SQL Warehouses

7. Cluster and job optimization

- Strong hands-on experience with AWS, particularly :

1. Amazon S3

2. IAM

3. VPC

4. ECS/EKS or Lambda

5. Step Functions

6. EventBridge

7. CloudWatch

8. Secrets Manager

9. KMS

- Practical knowledge of Apache Iceberg, including catalogs, snapshots, schema/partition evolution, compaction, and query-engine interoperability.

- Experience with Git, CI/CD, Docker, Terraform/CloudFormation, automated testing, observability, and production incident management.

- Experience working with semi-structured data and building reliable ingestion pipelines.

- Strong understanding of data quality, validation, error handling, monitoring, and replay/reprocessing mechanisms.

Healthcare Data Experience :

- Experience with healthcare data and standards is highly desirable, including :

1. FHIR R4 and/or HL7 v2

2. OMOP Common Data Model (OMOP CDM)

3. Clinical terminologies and healthcare data standards

4. Handling of PHI (Protected Health Information)

5. HIPAA-aligned engineering and security controls

6. Data privacy and de-identification concepts

AI & Engineering Practices :

- Demonstrated responsible use of AI coding assistants/tools in software and data engineering workflows.

- Ability to critically review AI-generated code for correctness, security, performance, maintainability, and production readiness.

- Strong focus on testing, documentation, code quality, and engineering best practices.

Education :

- UG : Any Graduate

Employment Type :

- Full Time, Permanent

Department :

- Data Science & Analytics