Role Overview:

We are looking for an experienced Data Engineer to design, develop, and optimize scalable data engineering solutions across cloud-based data platforms.

The role will focus on building robust ETL/ELT pipelines, data processing frameworks, and modern data solutions using AWS Databricks, Python, SQL, PySpark, and Apache Spark.

Experience in Healthcare, Medical, or Pharmacy Claims data will be an added advantage.

Key Responsibilities:

- Design, develop, and maintain scalable data pipelines using AWS Databricks.

- Build robust ETL/ELT frameworks for processing large volumes of structured and semi-structured data.

- Develop data transformation and processing solutions using Python, SQL, PySpark, and Apache Spark.

- Design and implement Delta Live Tables (DLT) pipelines for reliable and scalable data processing.

- Develop and manage Databricks Workflows for pipeline orchestration, scheduling, dependency management, and monitoring.

- Build reusable data engineering frameworks that support enterprise data platforms and analytics use cases.

- Develop optimized Spark-based solutions for large-scale data processing.

- Perform data ingestion, transformation, cleansing, validation, and enrichment across multiple data sources.

- Work on data migration and modernization initiatives, including migration from legacy platforms to Databricks-based architectures.

- Implement data pipelines supporting batch and, where required, near-real-time processing requirements.

- Apply appropriate data partitioning, caching, optimization, and Spark tuning techniques to improve pipeline performance.

- Implement data quality checks, validation frameworks, monitoring, logging, and error-handling mechanisms.

- Troubleshoot pipeline failures, performance issues, data inconsistencies, and production incidents.

- Translate business and functional requirements into scalable and maintainable technical solutions.

- Collaborate with business stakeholders, data architects, solution architects, analysts, and technology teams to understand requirements and define appropriate data solutions.

- Participate in solution design, technical discussions, Agile ceremonies, development, testing, deployment, and production support.

- Prepare and maintain technical documentation covering data flows, pipelines, transformations, dependencies, and operational processes.

- Follow engineering best practices around coding standards, version control, testing, deployment, and maintainability.

Tech Stack:

- AWS Databricks, Python, SQL, PySpark, Apache Spark, Delta Live Tables (DLT), Databricks Workflows.

AWS Cloud Exposure:

- Amazon S3, AWS Glue, Amazon Redshift, AWS Lambda, Amazon ECS/EKS, AWS Fargate, Amazon Aurora, AWS IAM.