About the Role :

We are looking for a highly skilled Databricks Engineer to design, develop, and optimize scalable data solutions on the Databricks platform. The ideal candidate should have strong expertise in big data technologies, cloud-based data engineering, and modern data processing frameworks to support enterprise analytics and business intelligence initiatives.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using Databricks.

- Build and optimize ETL/ELT processes for large-scale structured and unstructured data.

- Develop data transformation workflows using PySpark, Spark SQL, and Databricks notebooks.

- Integrate data from multiple sources, ensuring data quality, reliability, and governance.

- Collaborate with data architects, analysts, and business stakeholders to understand data requirements.

- Implement performance tuning and optimization techniques for Spark workloads.

- Monitor, troubleshoot, and resolve issues related to data pipelines and platform performance.

- Support data warehousing, reporting, and advanced analytics initiatives.

- Participate in code reviews, testing, deployment, and production support activities.

- Follow best practices for security, scalability, and data governance.

Required Skills :

- 5 to 10 years of experience in Data Engineering or Big Data technologies.

- Strong hands-on experience with Databricks and Apache Spark.

- Proficiency in PySpark, Python, and SQL.

- Experience building and managing large-scale ETL/ELT pipelines.

- Knowledge of data lakes, data warehouses, and modern data architectures.

- Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP).

- Understanding of performance optimization and Spark cluster management.

- Strong analytical and problem-solving skills.

Preferred Skills :

- Experience with Delta Lake and Lakehouse architecture.

- Knowledge of workflow orchestration tools such as Airflow or Azure Data Factory.

- Familiarity with CI/CD practices and DevOps methodologies.

- Experience in data governance, security, and compliance frameworks.

- Exposure to real-time data processing and streaming technologies.

Education :

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Key Performance Indicators (KPIs) :

- Timely delivery of data engineering projects.

- Performance and reliability of data pipelines.

- Data quality, accuracy, and governance compliance.

- Platform optimization and cost efficiency.

- Successful implementation of scalable data solutions.