Role Overview :

We are seeking a highly motivated and experienced Databricks Developer to join our dynamic team. In this role, you will be responsible for designing, developing, and maintaining data pipelines and solutions using Azure Databricks, Spark, and Delta Lake. You will collaborate closely with data scientists, data engineers, and business stakeholders to understand their requirements and translate them into efficient and scalable data solutions. Your work will directly impact our ability to derive valuable insights from data, improve business processes, and drive data-informed decision-making across the organization.

Key Responsibilities :

- Design and develop robust and scalable data pipelines using Azure Databricks, Spark, and Delta Lake to ingest, process, and transform large datasets.

- Implement ETL processes to load data into data warehouses and data lakes, ensuring data quality and consistency.

- Optimize Spark jobs for performance and efficiency, leveraging best practices for data partitioning, caching, and query optimization.

- Collaborate with data scientists and business analysts to understand their data needs and develop solutions to support their analytical requirements.

- Develop and maintain data documentation, including data dictionaries, data flow diagrams, and technical specifications.

- Monitor and troubleshoot data pipelines, identifying and resolving issues in a timely manner to ensure data availability and accuracy.

Required Skillset :

- Demonstrated ability to design, develop, and deploy data pipelines using Azure Databricks, Spark, and Delta Lake.

- Proven expertise in programming languages such as Python and Scala, with a strong understanding of data structures and algorithms.

- Solid understanding of data warehousing concepts and ETL processes, with experience in designing and implementing data models.

- Excellent communication and collaboration skills, with the ability to effectively communicate technical concepts to both technical and non-technical audiences.

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- Ability to work independently and as part of a team in a fast-paced, agile environment.

- 4 to 12 years of relevant experience.