Cognine - Data Engineer - SQL/PySpark

Data Engineer (Azure Data Factory and Azure Databricks)

Job Description :

We seek a Data Engineer with at least 5 years of experience in Azure Databricks, Azure Data Lake Storage gen2 (ADLS), and Azure Data Factory (ADF).

The candidate will design, build, and maintain data pipelines using Azure ADF/Databricks.

The candidate should be able to create Data Modelling & Governance, they will also work closely with our data science and engineering teams to develop and deploy machine learning models and other advanced analytics solutions.

Primary skills : Azure Data Factory, Azure Data Bricks, SQL, ADLS, Spark-SQL, Python (Pandas), Pyspark or Scala.

Secondary Skills : Basics of Azure Security (RBAC, Azure AD), Hadoop, HDFS, ADLS, Azure DBFS, PowerBI, and Tableau visualization tool is a plus.

Responsibilities :

- Design, build, and maintain data pipelines using Azure ADF/Databricks.

- Strong expertise in Azure Data Factory, Databricks, and related technologies such as Azure Synapse Analytics and Azure Functions.

- Hands-on experience in designing and implementing data pipelines and workflows in Azure Data Factory and Databricks.

- Experience in Data Extraction, Transformation, and Loading of data from multiple data sources into target databases, using Azure Databricks, Spark SQL, Pyspark, and Azure SQL.

- Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks.

- Sound working experience in Cleansing, Transformation, Business Logic, Incremental Transformation of Data, and merging the data with datamart tables,.

- Developing scalable and reusable frameworks for ingesting data sets.

- Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times.

- Interacting with stakeholders and leaders to understand business goals and data requirements.

- Experience in working with Agile (Scrum, Sprint) and Waterfall Methodologies.

- Collaborate with data engineers, data architects, and business analysts to understand and translate business requirements into technical designs.

- Provide technical guidance and support to junior team members.

- Design, develop, and maintain SQL databases, including creating database schemas, tables, views, stored procedures, and triggers.

- Self-starter and team player with excellent communication, problem-solving skills, and interpersonal skills, and a good aptitude for learning.