Data Integration Engineer

Description :

Position : Data Integration Engineer

Experience : 2-5 Years

Location : Bangalore

Industry Type : Logistics / Supply Chain Technology (Assumed)

Education : Bachelor's degree in Computer Science or a related field.

Job Summary :

We are seeking a proactive Data Integration Engineer with 2-5 years of hands-on experience in building and optimizing robust data pipelines. The role requires strong proficiency in Python (including Pandas/NumPy), an excellent grasp of data structures and algorithms, and practical experience with ETL/data integration workflows. The Engineer will be responsible for designing and managing data exchange across various formats (JSON/XML, CSV, EDI) and sources (REST APIs, relational/NoSQL databases), with mandatory experience in workflow orchestration using Apache Airflow.

Job Description :

Data Pipeline Design and Development (ETL) :

- Design, build, and maintain robust ETL/data pipelines or data integration workflows to ensure the seamless, high-volume flow of data between internal and external systems.

- Demonstrate mandatory Proficiency in Python, utilizing related libraries such as Pandas and NumPy for complex data manipulation, cleansing, and transformation tasks.

- Apply a strong understanding of object-oriented programming (OOP) principles, data structures, and algorithms to develop scalable and efficient code.

- Ensure the quality and reliability of integrated data by implementing rigorous testing and error handling within all ETL processes.

Data Management and Integration Technologies :

- Possess comprehensive knowledge of both relational and non-relational databases, including expertise in data modeling and normalization techniques to optimize storage and retrieval efficiency.

- Work hands-on with various data formats, including structured files like JSON and XML, and flat file formats such as CSV and EDI (Electronic Data Interchange), which are critical for the logistics domain.

- Develop and manage data retrieval mechanisms by working extensively with REST APIs for integrating external partner and client data sources.

- Create, schedule, and manage complex workflows using Apache Airflow to orchestrate pipelines, manage task dependencies, and monitor job execution effectively.

Collaboration and System Optimization :

- Analyze data flow requirements and perform detailed impact analysis for changes to upstream and downstream systems.

- Demonstrate the ability to collaborate effectively within teams and exhibit the discipline to work independently when necessary to meet project deadlines.

- Contribute to system optimization efforts, focusing on improving the performance, resilience, and cost-effectiveness of data integration infrastructure.

Required Skills & Qualifications :

- Experience : Mandatory 2-5 years of experience in data engineering or integration.

- Core Programming : Proficiency in Python and related libraries (Pandas and NumPy).

- Fundamentals : Strong understanding of object-oriented programming, data structures, and algorithms.

- Databases : Knowledge of relational and non-relational databases, data modeling, and normalization.

- Pipelines : Hands-on experience designing and building ETL/data pipelines or data integration workflows.

- Orchestration : Experience creating workflows using Apache Airflow.

- Formats/APIs : Experience with structured (JSON/XML) and flat files (CSV, EDI), and working with REST APIs.

- Education : Bachelor's degree in Computer Science or a related field.

Preferred Skills :

- Cloud : Exposure to AWS or other cloud platforms (e.g., Azure, GCP) and cloud-native data services.

- Client Engagement : Prior client-facing experience for requirement gathering and technical discussion.

- Domain : Familiarity with data standards and integration challenges within the Logistics or Supply Chain domain.