Responsibilities :

- Design, develop, and implement data pipelines using Apache Spark and Databricks notebooks to process and transform large datasets.

- Optimize data pipelines for performance, scalability, and cost-effectiveness.

- Automate data pipelines and integrate them with our existing data infrastructure.

- Develop and maintain unit tests to ensure the quality and reliability of data pipelines.

- Collaborate with data scientists, analysts, and other engineers to understand data requirements and translate them into technical solutions.

- Configure and manage Databricks clusters for optimal performance and resource utilization.

- Implement security best practices for data access and authorization within the Databricks environment.

- Monitor and troubleshoot Databricks jobs and identify potential performance bottlenecks.

- Stay up-to-date with the latest advancements in Databricks features and functionalities.

- Work collaboratively with data warehouse/data lake engineers to ensure seamless data ingestion and transformation processes.

- Develop data quality checks and data validation procedures within data pipelines.

Qualifications :

- Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field (or equivalent experience).

- 4-9 years of experience in data engineering or a related field.

- Proven experience working with Apache Spark and the Databricks platform.

- Strong understanding of data processing concepts, ETL/ELT pipelines, and big data architectures.

- Experience with SQL and familiarity with relational databases (e.g, MySQL, PostgreSQL).

- Proficiency in Python programming with experience in data manipulation libraries (e.g, Pandas).

- Experience with cloud platforms (AWS, Azure, GCP) a plus.

- Excellent problem-solving and analytical skills.

- Strong communication and collaboration skills.

- Ability to work independently and as part of a team