Job Description :

We are seeking a highly skilled Data Engineer to design, build, and optimize scalable data pipelines and data platforms that support analytics, reporting, and machine learning initiatives. The ideal candidate will have strong expertise in Python, PySpark, Scala, Apache Spark, Hadoop, SQL, Azure Cloud, and Databricks, along with experience in developing and maintaining large-scale distributed data processing systems.

Key Responsibilities :

- Design, develop, and maintain robust ETL/ELT pipelines for processing large volumes of structured and unstructured data.

- Build scalable data solutions using Apache Spark, PySpark, and Scala.

- Develop and optimize data workflows on Databricks and Azure Cloud platforms.

- Create and maintain data models, data lakes, and data warehouses to support business intelligence and analytics.

- Write efficient and optimized SQL queries, stored procedures, and data transformations.

- Integrate data from multiple sources, ensuring data quality, consistency, and reliability.

- Implement data governance, security, and compliance best practices.

- Monitor and troubleshoot data pipeline performance and resolve production issues.

- Collaborate with data analysts, data scientists, architects, and business stakeholders to deliver data-driven solutions.

- Automate deployment, testing, and monitoring of data engineering solutions.

- Participate in architecture discussions and recommend improvements for scalability and performance.

Required Skills:

- Strong proficiency in Python for data engineering and automation.

- Hands-on experience with PySpark and Apache Spark for distributed data processing.

- Solid programming experience in Scala.

- Extensive knowledge of SQL and database optimization techniques.

- Experience working with Hadoop Ecosystem.

- Strong expertise in Azure Cloud services such as:

i. Azure Data Factory (ADF)

ii. Azure Data Lake Storage (ADLS)

iii. Azure Synapse Analytics

iv. Azure Key Vault

v. Azure Monitor

- Hands-on experience with Databricks for data processing and analytics.

- Experience building and maintaining data lakes and modern data architectures.

- Knowledge of CI/CD practices and version control tools such as Git.