Data Engineer - Azure Databricks/Apache Spark

Description :

- 2-6 years of experience in data engineering or analytics engineering roles.

- Strong problem-solving and ability to work independently with guidance from senior engineers.

- Good communication skills to translate business requirements into technical data solutions.

- Ability to work in Agile environments and collaborate with global teams across time zones

- Hands-on experience with Azure Databricks and Apache Spark (PySpark or Scala) for large-scale data processing.

- Strong SQL skills, including complex queries, window functions, CTEs, and query optimization for both transactional and analytical workloads.

- Experience with SQL Server : data warehousing concepts, schema design, indexing, and optimization for analytics.

- Familiarity with Azure data services : Data Lake Storage, Synapse Analytics, Data Factory, or Blob Storage.

- Understanding of ETL/ELT concepts, data modeling, dimensional modeling (fact/dimension tables), and slowly changing dimensions.

- Experience with batch and/or stream processing concepts; familiarity with event- driven architectures.

- Knowledge of data quality frameworks, testing strategies, and monitoring/alerting for data pipelines.

- Basic understanding of relational and NoSQL databases (Cosmos DB exposure is beneficial)

What would you do here :

The Data Engineering team builds and operates robust, scalable data platforms on Azure and Databricks, closely partnering with domain PODs, Risk, Finance, and Analytics teams across the globe. Youll work on end-to-end data pipelines that ingest, transform, and serve data to multiple downstream consumers with strict quality and SLA requirements.

- Build and maintain ETL/ELT pipelines on Azure Databricks using PySpark or Scala to ingest, transform, and deliver data products at scale.

- Ingest data from SQL Server, Cosmos DB, event streams, APIs, and external sources into cloud data lakes and Databricks lakehouses.

- Design and implement data quality checks, validation rules, and monitoring to ensure reliability and early detection of anomalies.

- Optimize Spark jobs for performance and cost, including cluster sizing, partitioning strategies, and caching mechanisms.

- Document data pipelines, lineage, and schemas; maintain data dictionaries and metadata catalogs for consumer discovery and understanding.

- Collaborate with data analysts, scientists, and domain engineers to understand requirements and refine data products.

- Contribute to infrastructure and tooling for data workflows (scheduling, orchestration, monitoring, alerting)