Senior Data Engineer - Azure Databricks

Description :

Duties and Responsibilities :

- Design, build and test end to end data pipeline including data ingestion (streaming, events and batch), data integration, data curation

- Build and support data platform on the cloud

- Define and implement automation of jobs and testing

- Optimize the data pipeline to support workloads and use cases.

- Support mission critical applications and near real time data needs from the data platform.

- Build and support reusable framework to ingest, integration and provision data.

- Capture and publish metadata and new data to subscribed users

- Work collaboratively with product managers, data scientists as well as business partners and actively participate in solution thinking sessions

- Participate in design and code reviews

- Leads data disaster recovery planning and execution with the disaster recovery team and internal team members.

- Mentors more junior members of the team

- Determines risk of various configurations and options for data administration and is able to communicate issues to managers and team.

- Leads and develops internal standards for data administration and data system documentation.

- Develops physical data design. Defines, documents and executes, referential integrity requirements, constraints and techniques. Identifies index requirements and determines storage requirements

Requirements :

Basic Qualifications :

- Bachelors degree in computer science, Information Systems, Engineering, or a related field

- 4+ years of hands-on experience in data engineering and building data pipelines

- 4+ years of experience in writing complex SQL and Python

- Solid hands-on experience with data warehousing concepts and implementations

- 4+ years of experience with Snowflake or another modern cloud data warehouse

- 4+ years of hands-on Python development

- Understanding of Data modeling and Data warehousing concepts

- 2+ years experience with cloud computing (Azure)

- Familiarity with at least one cloud platform (AWS, Azure, or GCP)

- Strong analytical, problem-solving, and communication skills

- Retail/e-comm experience

Preferred Qualifications :

- Experience with DBT (Data Build Tool) for data transformations

- Exposure to real-time/streaming platforms (Kafka, Spark Streaming, Flink)

- Familiarity with CI/CD and version control (Git) in data engineering projects

- Exposure to the e-commerce or customer data domain

- Experience working on multiple concurrent deliverables