Agrim - Senior Data Engineer - Python/SQL

Senior Data Engineer (SDE-3)

Location : Gurugram (Work from Office)

Experience : 5+

Tech Stack : Python, SQL (PostgreSQL/MySQL), Apache Spark, Kafka, Airflow, BigQuery/Redshift, Druid, Hadoop, Snowflake

Responsibilities :

- Architect and optimize data warehouses and data lakes for high-performance analytics.

- Implement real-time and batch data processing using Apache Spark, Kafka, and Airflow.

- Work with structured and unstructured data across multiple storage solutions (Snowflake, BigQuery, Redshift, Hadoop, S3, etc.

- Ensure data quality, consistency, and integrity across multiple sources.

- Optimize SQL queries and database schemas for performance and cost efficiency.

- Implement data governance, security, and access controls.

- Collaborate with engineering, product, and analytics teams to design data models.

- Monitor and troubleshoot data pipeline failures and performance issues.

- Contribute to system design discussions, code reviews, and documentation.

Requirements :

- 5+ of experience in data engineering and big data processing.

- Strong proficiency in Python and SQL for data transformation and querying.

- Expertise in Apache Spark, Kafka, and Airflow for real-time and batch processing.

- Experience with cloud data warehouses (BigQuery, Redshift, Snowflake).

- Hands-on experience in ETL development, data modeling, and schema design.

- Strong knowledge of data partitioning, indexing, and query optimization.

- Experience with data lake architectures and storage formats like Parquet, ORC, Avro.

- Familiarity with containerization (Docker, Kubernetes) and CI/CD pipelines.

- Knowledge of monitoring tools (Prometheus, Grafana, ELK stack) for data infrastructure.

- Ability to mentor junior engineers and lead data-driven decision-making