Techno-Comp - Data Engineer - ETL/Python

Job Title : Data Engineer

Exp : 6 - 9 Yrs

Location : Pan India

Description :

Key Responsibilities :

Data Pipeline Development :

- Design, develop, and maintain scalable and robust data pipelines for collecting, processing, and transforming large datasets.

- Implement ETL (Extract, Transform, Load) processes to ensure efficient movement of data across multiple systems.

Database Management :

- Design, implement, and optimize relational and non-relational databases to support business needs.

- Ensure data quality, consistency, and accuracy by building validation checks, monitoring systems, and automating data reconciliation processes.

Cloud and Infrastructure Management :

- Work with cloud platforms (AWS, Azure, GCP) to deploy and manage data storage, compute, and processing resources.

Monitor and optimize the performance of data pipelines, queries, and data processing workflows

Experience :

- 5.5 years IT experience and good expertise in SDLC/Agile

- 5.5 years experience in SQL, complex queries, and optimization

Mandatory skills and qualifications required :

- 5.5 years experience in Spark ecosystem, Python/Scala programming, MongoDB data loads, Snowflake and AWS platform (EMR, Glue, S3), Kafka

- Hands on experience in writing advanced SQL queries, familiarity with variety of databases.

- Experience in coding solutions using Python/Spark and performing performance tuning/optimization.

- Experience in building and optimizing Big-Data pipelines in Cloud.

- Experience in handling different file formats like JSON, ORC, Avro, Parquet, CSV.

- Hands on experience in data processing with NoSQL databases like MongoDB.

- Familiarity and understanding of jobs scheduling.

- Hands on experience on working with APIs to process data.

- Understanding of data streaming, such as Kafka services

- 2+ years experience in Healthcare IT projects

- Certification on Snowflake (Snow PRO certification), and AWS (Cloud Practitioner/Solution Architect)

- Hands-on experience on Kafka streaming pipelines implementation.