Job Description:

We are looking for an experienced Databricks Developer to design, develop, and optimize scalable data engineering solutions using the Databricks platform. The candidate should have strong expertise in Apache Spark, PySpark/SQL, Delta Lake, data pipelines, and cloud data platforms.

Key Responsibilities:

- Develop and maintain scalable data pipelines using Databricks and Apache Spark.

- Build batch and streaming data processing solutions using PySpark and SQL.

- Design and implement Delta Lake / Lakehouse architecture.

- Develop ETL/ELT workflows for structured and unstructured data.

- Optimize Spark jobs, queries, clusters, and data pipelines for performance and cost.

- Integrate Databricks with cloud storage, databases, APIs, and enterprise data platforms.

- Implement data quality, validation, reconciliation, and monitoring processes.

- Work with data architects and data engineers to implement scalable data solutions.

- Support CI/CD and automated deployment of Databricks notebooks and workflows.

- Troubleshoot production data pipelines and resolve performance or data-quality issues.

- Follow data governance, security, and engineering best practices.

Required Skills:

- 6-11 years of experience in Data Engineering.

- Strong hands-on experience with Databricks and Apache Spark.

- Strong knowledge of PySpark and SQL.

- Experience with Delta Lake and Lakehouse architecture.

- Experience building enterprise-grade ETL/ELT pipelines.

- Strong understanding of data warehousing and data modelling.

- Experience with AWS, Azure, or GCP data platforms.

- Exposure to Databricks Workflows, Unity Catalog, MLflow, and Delta Live Tables is preferred.

- Experience with Git, CI/CD, and DevOps practices is an advantage.