Data Engineer - Python/PySpark

Description :

Responsibilities :

- Design and development of applications for Advanced analytical, data innovation systems.

- Participate in multiple software development processes, including Coding, Testing, Debugging, and Documentation.

- Develop software applications based on clear business specifications.

- Work on new initiatives and support existing Ratings applications.

- Perform Application and System Performance tuning and troubleshoot performance issues.

- Integrate in-house applications with various vendor software platforms.

- Set up a development environment/sandbox for application development.

- Check the application code changes in the source repository.

- Perform unit testing of application code and fix errors.

- Effectively interact with customers, business users, and IT staff.

- Participate in internal/cross-team meetings, project scoping, functional reviews, and technical reviews for assigned projects in an Agile environment.

Requirements :

- Hands-on development experience of 7+ years in the Databricks platform and writing Apache/PySpark jobs using Python and Scala.

- Must possess working experience in Kafka, Hive, AWS, and Azure platforms.

- Garbage collection-friendly programming experience - tuning Java garbage collection and performance is a must.

- Excellent communication skills are essential, with strong verbal and written proficiency.