Data Engineer - ETL/Python

Description :

Job Summary

We are seeking an experienced Data Engineer with strong hands-on expertise in Scala, Big Data technologies, and enterprise-grade data processing. The ideal candidate will design, develop, and optimize large-scale ETL pipelines, work across diverse data platforms, and collaborate with global teams in an Agile environment. The role requires strong technical skills, analytical mindset, and the ability to work on complex data-driven systems in a fast-paced enterprise environment.

Required Technical Skills :

Big Data & Programming :

- 5+ years of hands-on experience with Scala SDK (2.11.12 or higher), Apache Spark, Hadoop, Hive, and related Big Data ecosystem tools.

- Excellent knowledge of Scala EE, Web Services (REST/SOAP), Spring Framework, MQ, Kafka, and databases such as DB2/SQL.

- Experience working with Apache Tomcat or similar web servers.

Scheduling & Automation :

- Working experience with Autosys or similar job scheduling platforms.

Development Best Practices :

- Strong understanding of Design Patterns and their application in enterprise software development.

- Good understanding of DevOps practices and tools used for CI/CD, monitoring, and automation.

Enterprise Application Knowledge :

- Experience working with large-scale enterprise systems, including security, entitlements, and data governance requirements.

- Experience working within Agile development methodologies.

Desired Skills :

- Working knowledge of building and deploying applications in the Cloud.

- Experience with Unix/Linux environments and proficiency in scripting languages (Shell, Python, etc.).

- Familiarity with financial markets, document platform-based products, or Wealth Management systems.

Education :

- Minimum Bachelors degree (BS) in Computer Science, Engineering, or related field.

Job Functions/Duties & Responsibilities :

Design & Development :

- Design and develop enterprise-scale ETL processes for data ingestion, transformation, and distribution integrating with multiple systems.

- Implement requirements in Scala and ensure robust, scalable, and efficient data pipelines.

Analysis & Collaboration :

- Perform business analysis in coordination with BA/QA teams to ensure requirements are properly understood and documented.

- Collaborate with upstream and downstream technology teams, including Books & Records Systems, Pricing Systems, and Data Warehouses.

Agile Delivery :

- Work within Agile development methodologies, collaborating with global business and technology partners.

- Participate in sprint planning, daily stand-ups, and retrospectives to ensure timely and high-quality delivery.