Key Responsibilities :

- Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.

- Build and optimize data processing solutions using Databricks and cloud-based data platforms.

- Develop data integration solutions using Azure Data Factory (ADF) and other ETL tools.

- Work with structured and unstructured data from multiple data sources.

- Develop and optimize data models, ensuring data quality, integrity, and consistency.

- Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver scalable solutions.

- Monitor, troubleshoot, and optimize data pipelines for performance and reliability.

- Implement data governance, security, and best practices across the data platform.

- Support data migration, transformation, and modernization initiatives.

- Participate in code reviews, testing, deployment, and documentation activities.

Required Skills :

- Strong hands-on experience with Databricks.

- Experience with Azure Data Factory (ADF) or equivalent ETL tools.

- Proficiency in Azure and/or AWS cloud platforms.

- Good understanding of Hadoop ecosystem and distributed data processing.

- Strong experience in designing and developing ETL/ELT pipelines.

- Proficiency in Python, SQL, or Scala for data engineering.

- Experience with relational and NoSQL databases.

- Knowledge of data warehousing concepts and dimensional data modeling.

- Familiarity with version control tools such as Git.

- Strong analytical, debugging, and performance optimization skills.

Preferred Skills:

- Experience with Apache Spark and PySpark.

- Knowledge of Delta Lake, Apache Kafka, or streaming data pipelines.

- Exposure to CI/CD pipelines using Azure DevOps, Jenkins, or similar tools.

- Familiarity with containerization technologies such as Docker and Kubernetes.

- Experience with data governance and cloud security best practices.

Educational Qualification:

- B.E./B.Tech in Computer Science, Information Technology, or a related discipline.

- MCA or M.Sc. in Computer Science, Information Technology, or a related field is also acceptable.