Job Title : Lead Data Engineer

Location : Kochi / Trivandrum

Experience : 10+ years

Notice Period : Immediate to 15 days

Primary Skills : AWS, Python

Job Overview :

Seeking an experienced Senior/Lead Data Engineer to design and implement scalable data ingestion frameworks, ensure data quality and validation, and build robust APIs for seamless data integration. The ideal candidate will have hands-on expertise with AWS-based big data pipelines, automation, and high-performance API development.

Key Responsibilities :

Data Ingestion Framework :

- Architect and maintain ETL pipelines using AWS Glue, Lambda, EMR, ECS, EC2, Step Functions.

- Optimize data ingestion for scalability, resiliency, and automation.

Data Quality & Validation :

- Build automated validation and error-handling mechanisms.

- Implement monitoring, logging, and alerting for data quality assurance.

API Development :

- Design and develop secure RESTful APIs for internal/external data integration.

- Maintain API documentation and ensure adherence to best practices.

Collaboration & Agile :

- Partner with business stakeholders, data scientists, and operations.

- Contribute to CI/CD pipelines (preferably GitLab) and Agile processes.

Required Qualifications :

- Experience : Minimum 5+ years in data engineering / analytical platform development.

- Programming : Strong in Python / PySpark / SQL for ETL and large-scale data handling.

- AWS Expertise : Hands-on with AWS Glue, Lambda, Step Functions, S3, and data platforms (EMR, Apache Spark, Iceberg).

- Database Exposure : DynamoDB, Aurora, Postgres, Redshift.

- API Development : Proven REST API design & integration skills.

Preferred Qualifications :

- Bachelors/Masters in Computer Science, Data Engineering, or related field.

- Familiarity with AWS Kinesis, Firehose, SQS, and data lakehouse architectures.

- Experience with data quality frameworks and API-driven integrations in multi-cluster setups.

- Commitment to Information Security policies and best practices.