Data Engineer

Job Title : Data Engineer

Experience : 4 - 6 Years

Location : Pune

Job Type : Full-Time

Shift Timing : 2:00 PM - 11:00 PM

Job Overview :

We are looking for an experienced Data Engineer to design, develop, and maintain scalable data pipelines and data integration workflows that support analytics, operational reporting, and AI/ML applications. The ideal candidate should have strong expertise in Python, SQL, cloud-based data engineering, and distributed data processing systems.

Key Responsibilities :

- Design, build, and maintain scalable batch and streaming data pipelines for structured and unstructured data.

- Develop data ingestion, transformation, and integration workflows using Python, SQL, and modern data engineering frameworks.

- Build and maintain streaming pipelines using Kafka or equivalent messaging platforms.

- Work with cloud data services, preferably GCP, including BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Cloud Spanner.

- Develop and maintain REST and GraphQL APIs for data access and integration across microservices.

- Integrate data pipelines with AI/ML models and collaborate with Data Scientists and MLOps teams.

- Build data processing workflows for NLP applications and healthcare datasets, including claims and clinical documents.

- Implement data quality checks, pipeline monitoring, logging, and observability frameworks.

- Automate data pipeline deployment using CI/CD tools such as GitHub Actions, Jenkins, or Argo CD.

- Optimize pipeline performance, data reliability, scalability, and processing costs.

Required Qualifications :

- 4 - 6 years of experience in data engineering, data pipelines, or backend data workflows.

- Strong proficiency in Python and SQL.

- Experience with relational databases such as PostgreSQL.

- Hands-on experience building batch and streaming data pipelines.

- Experience with Kafka or similar streaming technologies.

- Experience with REST APIs and GraphQL for data integration and service development.

- Experience with cloud platforms, preferably Google Cloud Platform (GCP).

- Familiarity with BigQuery, Cloud Storage, Dataflow, Pub/Sub, or equivalent cloud data services.

- Experience with CI/CD practices and version control tools.

- Understanding of distributed systems, data modeling, data quality, and pipeline monitoring.

Preferred Qualifications :

- Experience processing healthcare data, claims data, or clinical documents.

- Exposure to AI/ML data pipelines and NLP workflows.

- Familiarity with GCP Composer, Cloud Spanner, or Cloud Functions.

- Knowledge of Docker, Kubernetes, and Infrastructure as Code.

- Experience with large-scale data processing and cloud cost optimization.

Education :

- Bachelors degree in Computer Science, Data Engineering, Information Systems, or a related field, or equivalent practical experience.