Overview :

The Data Engineer is a crucial member of our data team, responsible for designing, building, and maintaining robust data infrastructure. Utilizing platforms such as Databricks and Azure Data Factory, this role focuses on creating efficient data pipelines and managing large datasets for analysis. By employing tools like PySpark and SQL, the Data Engineer ensures data integrity, accessibility, and performance, thus empowering business intelligence initiatives across the organization.

With the escalating importance of data-driven decision-making, the Data Engineer plays an integral role in transforming raw data into actionable insights, facilitating the delivery of high-quality data to stakeholders. Collaboration with data analysts, data scientists, and other cross-functional teams is essential to enhance data usage throughout the organization.

The ideal candidate will have a strong background in data systems, excellent technical skills, and the ability to work in a dynamic environment while meeting stringent deadlines.

- Design, develop, and maintain scalable data pipelines using Databricks and PySpark.

- Implement data transformation processes within Azure Data Factory.

- Create and optimize SQL queries for efficient data retrieval.

- Collaborate with data scientists and analysts to understand data requirements.

- Ensure data integrity and quality through rigorous testing and validation processes.

- Build and maintain data models to support business intelligence efforts.

- Monitor and troubleshoot data pipeline performance issues.

- Document data engineering processes, architectures, and workflows.

- Develop and enforce best practices for data governance and security.

- Participate in Agile development cycles and project management activities.

- Stay updated on the latest data technologies and industry trends.

- Perform ad-hoc data analysis and report generation as required.

- Assist in capacity planning and optimization of data storage solutions.

- Engage in continuous improvement of data management processes.

- Ensure compliance with data privacy regulations and policies.

- Provide support and training to team members on data engineering tools and practices.

Required Qualifications :

- Bachelor's degree in Computer Science, Information Technology, or related field.

- Proven experience as a Data Engineer.

- Strong proficiency with Databricks.

- Extensive experience in PySpark.

- Hands-on experience with Azure Data Factory (ADF).

- Solid knowledge of SQL and relational databases.

- Experience with big data technologies such as Hadoop or Spark.

- Familiarity with data modeling and warehousing.

- Understanding of ETL (Extract, Transform, Load) processes.

- Ability to work with large datasets and optimize performance.

- Strong analytical and problem-solving skills.

- Excellent communication and teamwork abilities.

- Experience with cloud technologies, particularly Azure.

- Understanding of data security best practices.

- Proven ability to manage multiple tasks and meet deadlines.