Responsibilities :

- Design, develop, and maintain data pipelines using Azure Data Factory.

- Manage and optimize data storage using Azure Data Lake Gen 2.

- Build and process large-scale data solutions with Azure Databricks and Apache Spark.

- Collaborate with cross-functional IT and business teams to translate requirements into data solutions.

- Ensure data quality, security, and compliance in line with organizational standards.

Core Skills Required :

- Azure Data Factory : Experience in building and managing data pipelines.

- Azure Data Lake Gen 2 : Proficient in data storage and management within Azure's data lake environment.

- Azure Databricks : Proficient skill with distributed data processing, cluster setup, cluster scaling, cost optimization, Delta concepts.

- PySpark : Proficient skill with distributed data processing, transformations, analytics and optimization techniques.

- Python : Good skill with programing for data activities, should be very good with libraries like NumPy, Pandas, Dasks, Airflow etc.

- PySpark : Proficient skill with distributed data processing and transfer.

- Should have good exposure to streaming live tables.

Should-Have Skills :

- Strong SQL Performance Tuning : Ability to write and optimize SQL queries.

- Data Governance & Unity Catalog : Understanding of data governance principles and experience with Unity Catalog for data management.

Qualifications :

- Bachelor's degree in Computer Science, Information Technology, or related field (or equivalent experience).

- Professional experience with Azure data services.

- Strong analytical and problem-solving skills.

- Effective communication and team collaboration abilities.

Preferred Traits :

- Continuous learner, staying updated with Azure and data engineering advancements.

- Experience working in Agile environments.

- Passion for optimizing data workflows and enabling data-driven decisions.