Data Engineer

Mode Of Work : Remote(from any metro city)(Bangalore/ Pune/Hyderabad/Chennai/Noida)

Education : Bachelors degree in Computer Science, Information Systems, Engineering, or a related field

Job Description :

Data engineers design, build, and maintain the infrastructure for an organization's data, ensuring it is reliable, accessible, and optimized for use by data scientists and analysts. Key responsibilities include creating and managing data pipelines (ETL processes), optimizing data storage and processing, ensuring data quality and security, and collaborating with other teams to meet data needs.

Responsibilities :

Design and build data infrastructure :

- Create and maintain scalable data pipelines, databases, and data warehouses to handle large volumes of data from various sources.

Ensure data quality and reliability :

- Implement data cleansing, validation, and quality checks to ensure the data is accurate and trustworthy.

Optimize data systems :

- Monitor and tune performance of databases and data processing tasks to ensure efficiency and speed.

Collaborate with stakeholders :

- Work with data scientists, analysts, and other teams to understand their data requirements and make data accessible for their use.

Key skills :

- Data Platform Project

- Autonomous level people

- Skillsets needed (Mandatory)

- Pyspark, Spark SQL

- Databricks

- Data Governance Unity Catalog

- Python Pandera Framework

- REST API Frameworks

Nice to have :

- Cloud platforms : Experience with cloud services (e.g. GCP Services - Cloud build, BigQuery, Firestore etc)

- Airbyte/Fivetran

- Nutter Framework Experience in writing unit test in Nutter Framework

- Terraform Mandatory.

- CI/CD Concepts

- OAuth 2 Protocol

- Analytical and problem-solving skills : Strong analytical abilities are needed to troubleshoot issues and find solutions.

Skill Matrix(in years) :

- Data Engineering

- Data Bricks

- Pyspark

- Spark

- Sql

- Python

- GCP

- Unity Catalog

- Unit Testing

- CICD/ Terraform

- Rest API