Virtusa - Data Engineer - ETL/Python

Description :

Job Description : Senior Data Engineer (AWS & Databricks)

Role Overview :

We are looking for a Senior Data Engineer to architect and execute the migration of our data estate from SQL Server to an AWS-based Databricks Lakehouse. You will lead the transition from legacy stored procedures and manual processes into scalable PySpark pipelines. This role is critical for establishing our "Medallion Architecture" (Bronze/Silver/Gold) on AWS.

Key Responsibilities :

- Migration Leadership : Lead the technical migration strategy for moving data and logic from on-prem/RDS SQL Server to AWS S3 and Databricks.

- Pipeline Engineering : Build high-performance ETL/ELT pipelines using PySpark and Scala, leveraging Databricks Workflows and Unity Catalog.

- AWS Integration : Configure and optimize data ingestion patterns using AWS Glue (Crawlers), S3, and IAM for secure data access.

- Code Quality : Set standards for Python/Scala development, including unit testing and CI/CD integration via AWS CodePipeline or GitHub Actions.

Required Skills & Qualifications :

- 8+ years in Data Engineering with at least 3 years focused on AWS.

- Expert level : PySpark and Databricks (Delta Lake, Photon engine, Liquid Clustering).

- Strong Proficiency : SQL Server (T-SQL) for reverse-engineering legacy logic and optimizing source extraction.

- Languages : Mastery of Python and SQL; professional experience with Scala.

- Cloud Infrastructure : Hands-on experience with AWS S3, IAM, and Secrets Manager.

Nice to Have :

- Experience reading/decoding SSIS packages to extract business logic.

- Databricks Certified Data Engineer Professional