Digile - Senior Data Engineer - Python/Spark

Responsibilities :

- 6-10 years of overall experience mainly in the data engineering space building data ingestion and transformation pipelines.

- Design and implement data pipelines using best practices and industry-leading tools like Databricks and Azure Data Factory (ADF).

- Extract, transform, and load large datasets from various sources, ensuring data quality and integrity.

- Utilize Python and Spark to perform complex data manipulations and aggregations.

- Integrate data pipelines with APIs and external systems using efficient methods.

- Experience in Cleansing and Azure Data Explorer workflows.

- Monitor and maintain data pipelines, ensuring smooth operation and identifying potential issues.

- Collaborate with data scientists, analysts, and engineers to understand data needs and deliver valuable insights.

Qualifications :

- Must have implementation experience of Azure based cloud Data project/program as solution architect.

- Expert level knowledge on Spark.

- Extensive hands-on experience working with data using SQL, Python, Java, Scala.

- Strong experience and understanding of very large-scale data architecture, solutioning, and operationalization of data warehouses, data lakes, and analytics platforms (Both Cloud and On-Premise).

- Excellent communications skills with the ability to clearly present ideas, concepts, and solutions.

- Bonus points if you have experience on Insurance Domain for data engineering project.

Data Quality/Catalog :

- Strong attention to detail and accuracy, knowledge of data quality frameworks and standards, experience with data profiling and data analysis tools, understanding of data management best practices and excellent communication and collaboration skills.