Senior Data Engineer - Python/Spark

Key Responsibilities :

Data Architecture & Pipeline Development :

- Design, implement, and maintain scalable, reliable, and efficient data pipelines.

- Build and optimize data models for structured, semi-structured, and unstructured datasets.

- Develop real-time and batch processing solutions.

Data Integration & Management :

- Integrate data from multiple internal and external sources into a unified data platform.

- Ensure data quality, consistency, integrity, and governance across all systems.

- Implement best practices for data ingestion, transformation (ETL/ELT), and storage.

Performance Optimization :

- Optimize data workflows for performance, scalability, and cost-effectiveness.

- Monitor and troubleshoot issues with data pipelines and cloud infrastructure.

Collaboration & Leadership :

- Partner with data scientists, analysts, and product managers to deliver data solutions.

- Provide technical mentorship to junior engineers.

- Drive the adoption of modern data engineering practices, tools, and frameworks.

Security & Compliance :

- Implement data security standards, compliance policies, and governance protocols.

- Ensure adherence to GDPR, HIPAA, or other relevant regulatory requirements.

Required Qualifications :

- Bachelors or Masters degree in Computer Science, Engineering, Information Systems, or related field.

- 5 - 8 years of experience in data engineering or related roles.

- Strong proficiency in SQL and programming languages (Python, Scala, or Java).

- Hands-on experience with big data frameworks (e.g., Spark, Hadoop, Kafka, Flink).

- Expertise in data modeling, warehousing, and ETL/ELT development.

- Solid experience with cloud platforms (AWS, Azure, GCP) and data services like Redshift, BigQuery, Snowflake, or Databricks.

- Strong understanding of data governance, security, and compliance.

Preferred Qualifications :

- Experience with containerization & orchestration tools (Docker, Kubernetes, Airflow).

- Familiarity with real-time streaming technologies (Kafka, Kinesis, Pulsar).

- Knowledge of machine learning pipelines and integration with AI/ML workflows.

- Strong problem-solving and analytical skills with attention to detail.

- Excellent communication skills with the ability to work across technical and business teams.