Data Engineer - Azure Databricks

Description :

We are looking for a skilled Data Engineer - Databricks to design, develop, and optimize scalable data pipelines and analytics solutions on the Databricks platform. The ideal candidate should have strong experience in big data technologies, cloud platforms, and ETL/ELT processes.

Key Responsibilities :

- Design and develop scalable ETL/ELT pipelines using Databricks.

- Build and maintain data workflows using Apache Spark and PySpark.

- Integrate data from multiple sources including APIs, databases, and cloud storage.

- Optimize data processing performance and ensure data quality.

- Work with large-scale structured and unstructured datasets.

- Collaborate with data analysts, architects, and business stakeholders.

- Implement data governance, security, and monitoring practices.

- Support CI/CD deployment and automation for data engineering workflows.

- Troubleshoot production issues and provide performance tuning.

Required Skills :

- Strong experience with Databricks and Apache Spark.

- Proficiency in PySpark, Python, and SQL.

- Experience with cloud platforms like Amazon Web Services, Microsoft, or Google.

- Hands-on experience with Delta Lake, Data Lake, and Data Warehousing concepts.

- Knowledge of workflow orchestration tools such as Airflow or ADF.

- Familiarity with Git, CI/CD pipelines, and Agile methodology.

- Understanding of data modeling and performance optimization.

Preferred Qualifications :

- Bachelors degree in Computer Science, IT, or related field.

- Databricks or cloud certification is a plus.

- Experience working in enterprise-level data engineering projects.