Senior Data Engineer

Description :

KEY SKILLS AND EXPERTISE :

- Strong hands-on experience with Databricks, including :

- Databricks Notebooks (Python, SQL)

- Unity Catalog for governance & security

- Advanced Databricks performance optimization techniques

- Best practices for cost and compute efficiency

- Proficient in Azure Data Factory (ADF) for orchestrating ETL workflows.

- Excellent programming in Python, with advanced PySpark skills.

- Solid understanding of Apache Spark internals & tuning.

- Expertise in SQL writing complex queries, optimizing joins, working with large datasets.

- Familiar with data warehousing principles & modeling techniques (e.g., star/snowflake schemas).

- Knowledge of Azure data services : Data Lake Storage, Synapse Analytics, SQL Database.

- Experience implementing data governance, access control, and data lineage.

WHAT YOU'LL DO :

- Design and implement robust, scalable, and efficient data pipelines using Databricks and ADF.

- Leverage Unity Catalog to secure and govern sensitive data.

- Optimize Databricks jobs & queries for speed, cost, and scalability.

- Build and maintain Delta Lake tables and data models suitable for analytics and BI.

- Collaborate with stakeholders to define data needs, provide solutions, and deliver business value.

- Automate manual workflows, improve reliability, and ensure data quality.

- Troubleshoot and monitor pipelines to guarantee uptime and data accuracy.

- Mentor junior engineers and foster best practices in Databricks & Azure data engineering.

PREFERRED BACKGROUND :

- 5+ years in data engineering with a focus on Azure.

- Demonstrated ability to work with large-scale distributed systems.

- Strong communication and teamwork skills.

- Certifications in Databricks and/or Azure Data Engineering are a plus.