Data Engineer - Azure Databricks

Description :

We are seeking a Databricks Data Engineer to design, build, and optimize scalable data pipelines and analytics solutions on the Databricks Lakehouse Platform. You will be responsible for transforming raw data into high-value datasets that power analytics, AI/ML models, and business insights. We're looking for a dynamic, self-motivated, and proactive leader who thrives in a fast-paced environment and can foster a high-performing culture.

Responsibilities :

- Develop scalable ETL/ELT pipelines using PySpark, SQL, and Delta Lake.

- Build and maintain batch and stream data pipelines using Databricks Workflows.

- Design and implement Lakehouse architecture and optimize Delta Lake tables.

- Integrate data from multiple sources, including APIs, databases, cloud storage, and third-

party platforms.

- Ensure data quality and reliability using Delta Live Tables, expectations, and monitoring tools.

- Work with CI/CD pipelines using Git, Databricks Repos, and DevOps tools.

- Implement governance, security, and compliance standards (Unity Catalog, lineage, RBAC).

Requirements :

- Strong knowledge of data modeling, data warehousing concepts, and Lakehouse

fundamentals.

- Strong experience with Databricks, PySpark, Apache Spark, and SQL.

- Hands-on with Delta Lake, Databricks Workflows, Unity Catalog.

- Experience building scalable pipelines on Azure, AWS, or GCP.

- Solid understanding of distributed systems and performance optimization.

- Experience with streaming frameworks (Structured Streaming, Kafka, Auto Loader).

- Passion for problem-solving, both technical and business-oriented

- User-centric mindset focused on building delightful software experiences

- Lead by example : implement best practices and maintain high-quality code standards

Nice to have skills :

- Databricks certification (Data Engineer Associate / Professional).

- Experience with ML workflows and feature engineering.

- Hands-on with Databricks Photon, serverless SQL, and optimization techniques.

- Familiarity with BI tools (Power BI, Tableau, Looker).