Lead Data Engineer - Scala/Spark

About the Role :

We are seeking a highly skilled and experienced Lead Data Engineer/Scala Developer with strong hands-on expertise in Scala programming, Apache Spark, Databricks, Python, and PySpark. The ideal candidate will lead the design, development, and deployment of scalable big data solutions, working closely with cross-functional teams to deliver high-quality data processing pipelines and analytics platforms.

Key Responsibilities :

- Lead the development and implementation of big data solutions using Scala and Apache Spark on Databricks.

- Design, develop, and optimize Spark jobs and ETL pipelines using Scala, Python, and PySpark.

- Collaborate with data engineers, data scientists, and business stakeholders to understand requirements and translate them into technical solutions.

- Mentor and guide junior developers and team members on best practices in Scala and Spark development.

- Ensure code quality, performance tuning, and scalability of Spark applications.

- Participate in architectural discussions and contribute to technology roadmap decisions.

- Troubleshoot and resolve production issues related to Spark jobs and data pipelines.

- Implement CI/CD pipelines for Spark applications and automate deployment processes.

- Stay updated with the latest trends and advancements in Scala, Spark, and big data technologies.

Required Skills and Qualifications :

- Strong hands-on experience in Scala programming with a deep understanding of functional programming concepts.

- Extensive experience with Apache Spark (RDDs, DataFrames, Datasets) and Databricks platform.

- Proficient in Python and PySpark for big data processing.

- Experience in building and optimizing large-scale ETL pipelines and data workflows.

- Solid understanding of distributed computing and big data ecosystem.

- Familiarity with cloud platforms such as AWS, Azure, or GCP, especially Databricks on cloud.

- Experience with version control systems (Git) and CI/CD tools.

- Strong problem-solving skills and ability to work in an agile environment.

- Excellent communication and leadership skills to manage and mentor a development team.