Job Title : Data Engineers (Azure Databricks + PySpark)

Location : Remote

Contract : 6 Months (C2C)

Experience Required : 5 - 6 Years

Job Description :

Key Responsibilities :

- Design and Implement ETL Pipelines : Design, develop, and deploy scalable ETL pipelines using Azure Databricks and PySpark, ensuring efficient data processing and transformation.

- Data Wrangling and Transformation : Perform data wrangling, cleaning, and transformation of large datasets, ensuring data quality and integrity.

- Collaboration : Collaborate with data scientists, analysts, and architects to build integrated data solutions, providing data engineering expertise and support.

- PySpark Optimization : Optimize PySpark code for performance and scalability, ensuring efficient data processing and minimizing latency.

- Data Quality Checks : Implement data quality checks, logging, and monitoring to ensure data accuracy, completeness, and consistency.

- Data Solution Development : Develop and implement data solutions that meet business requirements, leveraging Azure Databricks, PySpark, and other relevant technologies.

Must-Have Skills :

- Azure Databricks : Strong expertise in Azure Databricks, including cluster configuration, job scheduling, and notebook development.

- PySpark : Proficient in PySpark, including data frames, RDDs, and Spark SQL.

- Python : Proficient in Python programming language, with experience in developing data engineering solutions.

- Distributed Data Processing : Solid understanding of distributed data processing concepts, including data partitioning, caching, and parallel processing.

- Data Engineering Experience : 5-6 years of experience in data engineering, with a focus on designing and implementing scalable data solutions.

Preferred Skills :

- Azure Data Lake : Hands-on experience with Azure Data Lake, including data ingestion, processing, and storage.

- Delta Lake : Experience with Delta Lake, including data versioning, transactional writes, and data lakes.

- Azure Data Factory : Experience with Azure Data Factory, including data pipeline development, deployment, and management.

- Data Architecture : Understanding of data architecture principles, including data warehousing, data governance, and data security.