Ekloud - Senior Data Engineer - Python/SQL

Job Title : Senior Data Engineer

Experience : 6 to 8 Years

Location : Remote

Job Overview :

We are seeking an experienced Senior Data Engineer with 68 years of proven expertise in designing, developing, and optimizing large-scale data processing and analytics platforms using Python and Databricks. The ideal candidate will have strong proficiency in Delta Lake, Spark, and MLflow, as well as experience building scalable data pipelines and AI/ML workflows using orchestration tools such as Airflow, Prefect, or Dagster.

A deep understanding of cloud platforms (AWS, Azure, or GCP) and knowledge of data governance, security, and compliance standards (HIPAA, SOC 2, GDPR) is essential, especially in healthcare or life sciences domains.

Key Responsibilities :

- Design and develop data pipelines and ETL workflows using Python, Spark, and Databricks.

- Implement and optimize Delta Lake architectures for structured and semi-structured data processing.

- Build, orchestrate, and automate AI/ML pipelines using Airflow, Prefect, or Dagster.

- Work closely with Data Scientists to operationalize ML models using MLflow and Databricks Workflows.

- Ensure data quality, lineage, and governance compliance across all data processes.

- Optimize data ingestion, transformation, and storage for performance and cost efficiency.

- Collaborate with cross-functional teams to design scalable, cloud-native data architectures (AWS, Azure, or GCP).

- Implement monitoring, alerting, and performance tuning for production data pipelines.

- Maintain compliance with data security and privacy standards in healthcare/life sciences environments.

Technical Skills Required :

- Languages : Python, SQL, PySpark

- Frameworks & Tools : Databricks, Delta Lake, MLflow

- Orchestration : Airflow, Prefect, or Dagster

- Cloud Platforms : AWS (Glue, EMR, S3, Lambda), Azure (Data Factory, Synapse, ADLS), or GCP (BigQuery, Dataflow)

- Data Architecture : ETL/ELT pipelines, Data Lakes, Data Warehouses