Senior Data Engineer II

Position : Data Engineer II

Location : Bengaluru (Bangalore), India

Duration : 6-12 Months (Contract)

About the Role :

We are seeking a highly skilled and experienced Data Engineer II to join our dynamic team in Bengaluru. In this contract role, you will be instrumental in designing, building, and maintaining robust and scalable data pipelines and infrastructure. You'll work with large datasets, leveraging your expertise in SQL, AWS, and Python to transform raw data into actionable insights that drive critical business decisions. This is an exciting opportunity to contribute to a data-driven culture and directly impact our analytical capabilities.

Key Responsibilities :

- Data Pipeline Development : Design, develop, and optimize scalable ETL/ELT data pipelines using AWS services and Python to ingest, process, and transform diverse datasets from various sources.

- AWS Cloud Infrastructure : Build and manage data infrastructure components within the AWS ecosystem, including but not limited to Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift/Snowflake, and other relevant data services.

- Data Modeling & Warehousing : Develop and maintain optimized data models for our data warehouse, ensuring data integrity, consistency, and performance for analytical and reporting needs.

- Performance Optimization : Identify and implement solutions for improving data pipeline efficiency, reliability, and cost-effectiveness.

- Data Quality & Governance : Establish and enforce data quality standards, implement monitoring, and develop mechanisms for data validation and error handling.

- Collaboration & Mentorship : Work closely with data scientists, analysts, and product teams to understand data requirements and deliver solutions. Provide technical guidance and mentorship to junior engineers.

- Troubleshooting & Support : Troubleshoot and resolve complex data-related issues, ensuring minimal disruption to data availability and accuracy.

- Documentation : Create and maintain comprehensive documentation for data pipelines, architecture, and processes.

Required Qualifications :

- Experience : Minimum of 5-10 years of progressive experience in data engineering, backend engineering with a strong data focus, or a related field.

- AWS Expertise : Strong proficiency and hands-on experience in building and maintaining data solutions within the AWS cloud environment. This includes deep familiarity with services such as :

- Storage : Amazon S3

- Compute/Orchestration : AWS Glue, AWS Lambda, AWS Step Functions, EC2

- Data Warehousing : Amazon Redshift (or similar cloud data warehouses like Snowflake, Google BigQuery)

- Networking/Security : VPC, IAM

- Python Proficiency : Advanced proficiency in Python for data manipulation, scripting, automation, and developing data processing applications. Experience with libraries like Pandas, PySpark, or similar is a plus.

- Advanced SQL Knowledge : Expert-level SQL skills for complex querying, data manipulation (DDL/DML), performance tuning, and optimizing database interactions.

- Data Warehousing Concepts : Solid understanding of data warehousing principles, dimensional modeling (Kimball/Inmon), and star/snowflake schemas.

- Pipeline Orchestration : Experience with workflow orchestration tools (e.g., Apache Airflow, AWS Step Functions) for scheduling and monitoring data pipelines.

- Version Control : Proficient with Git and collaborative development workflows.

- Education : Bachelors degree in Computer Science, Engineering, Information Technology, or a related quantitative field is preferred; equivalent practical experience and demonstrated expertise are highly valued.

Preferred Qualifications :

- Experience with real-time data processing and streaming technologies (e.g., Kinesis, Kafka).

- Familiarity with containerization technologies (Docker, Kubernetes).

- Experience with CI/CD practices for data pipelines.

- Knowledge of big data frameworks (e.g., Apache Spark).

- AWS Certifications (e.g., AWS Certified Data Analytics Specialty, AWS Certified Solutions Architect Associate)