Senior Data Engineer - ETL/PySpark

Description :

Notice Period : Immediate to 25 Days

Interview Process : 4 Rounds (2 Internal + 2 Client)

Job Summary :

We are looking for a highly skilled Senior Data Engineer with strong expertise in Azure Data Engineering stack to design, build, and optimize scalable data pipelines and analytics solutions. The ideal candidate will have hands-on experience with Azure Data Factory (ADF), Databricks, Spark (including Lakeflow Declarative Pipelines), SQL, and PySpark, and a deep understanding of modern data architectures.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Azure services such as ADF and Databricks

- Build and manage data ingestion frameworks for structured and unstructured data from multiple sources

- Develop ETL/ELT processes using PySpark and Spark (Lakeflow Declarative Pipelines)

- Optimize and tune big data workflows for performance, scalability, and cost efficiency

- Implement data transformation logic using SQL and PySpark

- Collaborate with data scientists, analysts, and business stakeholders to deliver data solutions

- Ensure data quality, integrity, and governance across pipelines

- Work on data lake architecture and implement best practices for data storage and processing

- Monitor, troubleshoot, and resolve data pipeline issues in production environments

- Implement CI/CD pipelines and automate deployment processes

- Mentor junior engineers and contribute to team knowledge sharing

Mandatory Skills :

- Strong experience with Azure Data Factory (ADF)

- Hands-on expertise in Azure Databricks

- Solid understanding of Apache Spark (including Lakeflow Declarative Pipelines)

- Proficiency in PySpark for large-scale data processing

- Strong SQL skills for data manipulation and querying

- Experience working with Azure Data Lake / Delta Lake architecture

- Knowledge of ETL/ELT design patterns

- Experience in performance tuning and optimization

- Experience with Azure Synapse Analytics

- Knowledge of data warehousing concepts (Star/Snowflake schemas)

- Familiarity with DevOps tools (Azure DevOps, CI/CD pipelines)

- Exposure to streaming frameworks (Kafka, Event Hub)

- Understanding of data governance and security best practices

- Experience with Python programming beyond PySpark

- Bachelors/Masters degree in Computer Science, Engineering, or related field