We are hiring an experienced Data Engineer with strong hands-on expertise in Databricks, Spark, Python, SQL, and PySpark to build and manage large-scale data ingestion and processing pipelines on AWS/Azure cloud platforms. The candidate will work on high-volume data systems, ensuring scalability, performance optimization, and data quality.

Job Description :

- Minimum of 7+ years of significant experience in application development.

- Previous experience in the areas of design and implementation of Ratings Data Ingestion pipelines with modern AWS/Azure cloud and other technologies such as Databricks, Python, SQL and PySpark.

- Development, design, and architecture exposure & the ability to ensure quality across various technology components that are developed by geographically diversified software engineers with superior knowledge of system architecture, object-oriented design, and design patterns.

- Proficient with software development lifecycle (SDLC) methodologies like Agile and test-driven development.

- Experience working with high-volume data and computationally intensive systems;

- Garbage collection friendly programming experience - tuning Java garbage collection & performance is a must.

- Proficiency in the development environment, including IDE, web & application server, GIT, Continuous Integration, unit-testing tools, and defect management tools Domain knowledge in the Financial Industry and Capital Markets is a plus.

- Excellent communication skills are essential, with strong verbal and writing proficiencies;

- Mentor teams, innovate and experiment, give a face to business ideas, and present to key stakeholders.across distributed environments.

- hands-on experience with Databricks, Spark Core, Spark SQL, RDDs, Python, SQL, Kafka, HDFS/ADLS, Hive/ HQL (HiveQL), MapReduce, Sqoop, Pig, Power BI and DAX.