Big Data Engineer - Spark/Scala

Role & responsibilities :

- Evaluate domain, financial and technical feasibility of solution ideas with help of all key stakeholders

- Design, develop, and maintain highly scalable data processing applications

- Write efficient, reusable and well documented code

- Deliver big data projects using Spark, Scala , Python, SQL

- Maintain and tune existing Spark applications to the fullest.

- Find opportunities for optimizing existing spark applications.

- Work closely with QA, Operations and various teams to deliver error free software on time

- Actively lead / participate daily agile / scrum meetings

- Take responsibility for Apache Spark development and implementation

- Translate complex technical and functional requirements into detailed designs

- Investigate alternatives for data storing and processing to ensure implementation of the most streamlined solutions

- Serve as a mentor for junior staff members by conducting technical training sessions and reviewing project outputs

Qualifications :

- Engineering graduates in computer science backgrounds preferred with 8+ years of software development experience with Hadoop framework components(HDFS, Spark, Spark, Scala, PySpark)

- Excellent at verbal, written and presentation skills

- Ability to present and defend a solution with technical facts & business proficiency

- Understanding of data-warehousing and data-modeling techniques

- Strong data engineering skills

- Knowledge of Core Java, Linux, SQL, and any scripting language

- At least 6+ years of experience using Python / Scala, Spark, SQL

- Knowledge of shell scripting is a plus

- Knowledge of Core and Advance Java is a plus.

- Experience in developing and tuning spark applications

- Excellent understanding of spark architecture, data frames and tuning spark

- Strong knowledge of database concepts, systems architecture, and data structures is a must Process oriented with strong analytical and problem solving skills

- Experience in writing Python standalone applications dealing with PySpark API

- Knowledge of DELTA.

- IO package is a plus