Key Responsibilities :

- Design, develop, and maintain robust and scalable data pipelines using Python/PySpark and Azure services.

- Implement ETL/ELT processes to ingest, transform, and load data from various sources into our data lake and data warehouse.

- Automate data pipelines using Azure Data Factory and other orchestration tools.

- Design and implement efficient data models to support business requirements.

- Optimize data structures for performance and scalability.

- Perform data analytics tasks to provide insights and support business decision-making.

- Develop reports and dashboards to visualize data and communicate findings.

- Leverage Azure cloud services such as Azure Databricks, Azure Data Factory, Azure Data Lake Storage, and Azure DevOps to build and maintain our data platform.

- Optimize Azure resources for cost-effectiveness and performance.

- Implement and maintain CI/CD pipelines using Azure DevOps to automate code deployment and testing.

- Ensure code quality and adherence to best practices.

- Implement and maintain data security practices and methods, including API/web service security.

- Ensure compliance with data security policies and regulations.

- Work with relational databases such as SQL Server, Oracle, or MySQL.

- Write and optimize SQL queries for data access and manipulation.

- Develop interactive data applications and dashboards using Streamlit or RShiny (preferred).

- Collaborate effectively with other engineers, analysts, and stakeholders.

- Communicate clearly and concisely, both orally and in writing.

- Troubleshoot and resolve data-related issues.

- Identify and implement solutions to improve data quality and performance.

Required Skills and Qualifications :

Experience : 5+ years of experience in data engineering.

Technical Skills :

- Strong proficiency in Python/PySpark and SQL.

- Expertise in Azure cloud services, including Azure Databricks, Azure Data Factory, Azure Data Lake Storage, and Azure DevOps.

- Knowledge of CI/CD principles and tools.

- Good understanding of API/web service security and data security practices.

- Experience working with relational databases (SQL Server, Oracle, MySQL)