Description :

We are seeking an experienced Data Engineer with strong hands-on expertise in Databricks, Delta Lake, and Unity Catalog to design, build, and optimize our modern data platform. The ideal candidate has a solid background in distributed data processing, data modeling, cloud technologies, and data governance.

Key Responsibilities :

- Build and maintain data pipelines using Databricks (PySpark, SQL).

- Develop and optimize Delta Lake tables and data workflows.

- Manage Databricks clusters, jobs, notebooks, and workflows.

- Implement data governance and access control using Unity Catalog.

- Set up catalogs, schemas, tables, and permissions in Unity Catalog.

- Work with cloud storage (S3/ADLS/GCS) for data ingestion and processing.

- Improve performance of Spark jobs and data pipelines.

- Ensure data quality, reliability, and accuracy across datasets.

- Collaborate with analysts, architects, and business teams on data requirements.

- Support production data pipelines and troubleshoot issues.

Required Skills :

- 10 to 15 years of experience as a Data Engineer or similar role.

- Strong expertise in Databricks : Notebooks, Jobs, Workflows, SQL Warehouses.

- Hands-on experience with PySpark/Spark SQL for large-scale data processing.

- Deep understanding of Delta Lake, ACID transactions, Z-ordering, CDF, and optimization techniques.

- Proven experience implementing Unity Catalog for governance and security.

- Strong knowledge of cloud platforms (AWS, Azure, or GCP).

- Experience with CI/CD, Git, DevOps, and modern data engineering workflows.

- Strong SQL skills and understanding of data modeling principles.

Nice-to-Have Skills :

- Experience with streaming technologies like Kafka, Delta Live Tables, Structured Streaming.

- Exposure to Airflow, Azure Data Factory, or other orchestration tools.

- Familiarity with MLflow and MLOps concepts.

- Knowledge of Lakehouse Architecture and BI integrations (Power BI, Tableau, Looker).