Description :

Mandatory Skills Required: ADB, ADF, Deltalake/Datalake and Pyspark, Data Modelling and Optimization, Unity Catalog

Experience Required : 6+ years of relevant experience

Location : Bengaluru/Trivandrum

Notice Period : Immediate joiners to 30 days

Job Summary :

We are looking for a highly skilled Senior Data Engineer with over 6 years of experience in data engineering and a strong focus on data migration projects.

The ideal candidate will have deep expertise in Azure Data Factory (ADF), ETL/ELT processes, and Python/PySpark, with proven experience building scalable and efficient data pipelines across diverse data environments.

Key Responsibilities :

- Design, develop, and implement end-to-end data pipelines across various sources and destinations.

- Lead and support data migration initiatives from legacy to modern data platforms.

- Develop robust ETL/ELT workflows using Azure Data Factory.

- Write clean, efficient, and scalable code in Python or PySpark for data transformation and automation.

- Utilize SQL for complex querying, data manipulation, and performance tuning.

- Collaborate with DevOps teams to manage CI/CD pipelines using Azure DevOps.

- Deploy and manage infrastructure using Terraform and other IaC tools.

- Ensure data solutions align with data governance, data warehousing, and dimensional modeling best practices.

- Work closely with cross-functional teams to deliver high-quality, timely data solutions.

Mandatory Skills:

- 6+ years of experience in data engineering with a focus on data migration projects.

- Hands-on expertise with Azure Data Factory (ADF).

- Strong experience in ETL/ELT process design and implementation.

- Proficiency in Python/PySpark for data manipulation and automation.

- Expert-level SQL skills.

- Solid experience with Azure DevOps for CI/CD pipeline management.

- Experience with Terraform for deploying and managing Azure infrastructure.

- Strong knowledge of data warehousing, dimensional modeling, and data governance principles.

Good-to-Have Skills :

- Experience with other Azure services (e.g., Azure Synapse, Azure Data Lake).

- Knowledge of data quality frameworks and monitoring tools.

- Familiarity with Agile/Scrum development methodologies.

- Certifications in Azure Data Engineering or related technologies.

- Exposure to big data tools and frameworks (e.g., Databricks, Spark).

Soft Skills :

- Strong analytical and problem-solving skills with attention to detail.

- Excellent communication and interpersonal skills.

- Ability to work both independently and within a team in a fast-paced environment