Senior Data Engineer - Big Data Technologies

Description :

Roles & Responsibilities :

Data Ingestion & Pipeline Development :

- Design, build, and maintain scalable data ingestion pipelines for structured and unstructured data.

- Develop batch and real-time data processing workflows to support analytics and business reporting.

- Ensure the accuracy, quality, and reliability of end-to-end data pipelines.

- IDMC (Informatica Intelligent Data Management Cloud)

- Develop and manage data integration workflows using IDMC / Informatica Cloud.

- Configure mappings, tasks, and orchestrations for efficient data movement and transformation.

- Troubleshoot IDMC pipelines and optimize for performance and scalability.

Big Data Engineering :

- Work with Big Data ecosystems and distributed processing technologies such as Hadoop, Spark, Hive, Kafka, etc.

- Build scalable and high-performance data processing solutions for large datasets.

- Optimize data transformation jobs for speed, cost efficiency, and reliability.

Cloud & Data Platform Support :

- Work with cloud data platforms (AWS, Azure, or GCP) for storage, ETL, and data flow orchestration.

- Utilize cloud services such as object storage, data ingestion services, and compute engines.

- Follow cloud security, governance, and best practices for data handling.

Collaboration & Documentation :

- Collaborate with Architects, Analysts, and Product teams to gather requirements and convert them into data solutions.

- Document data pipelines, workflows, specifications, and data flow diagrams.

- Participate in code reviews, performance tuning, and continuous improvement initiatives.

Must-Have Skills :

- Strong experience in Data Ingestion and pipeline development

- Hands-on expertise with IDMC / Informatica Intelligent Data Management Cloud

- Experience with Big Data Technologies (e.g., Hadoop, Spark, Hive, Kafka)

- Strong SQL knowledge and data handling skills

- Experience working with large datasets and distributed systems