Description :

Roles & Responsibilities :

- Data Engineering Excellence : Design and implement data pipelines using formats like JSON, Parquet, CSV, and ORC, utilizing batch and streaming ingestion.

- Cloud Data Migration Leadership : Lead cloud migration projects, developing scalable Spark pipelines.

- Medallion Architecture : Implement Bronze, Silver, and gold tables for scalable data systems.

- Spark Code Optimization : Optimize Spark code to ensure efficient cloud migration.

- Data Modeling : Develop and maintain data models with strong governance practices.

- Data Cataloging & Quality : Implement cataloging strategies with Unity Catalog to maintain high-quality data.

- Delta Live Table Leadership : Lead the design and implementation of Delta Live Tables (DLT) pipelines for secure, tamper-resistant data management.

- Customer Collaboration : Collaborate with clients to optimize cloud migrations and ensure best practices in design and governance.

Educational Qualifications :

- Experience : Minimum 5 years of hands-on experience in data engineering, with a proven track record in complex pipeline development and cloud-based data migration projects.

- Education : Bachelors or higher degree in Computer Science, Data Engineering, or a related field.

- Skills :

i. Must-have : Proficiency in Spark, SQL, Python, and other relevant data processing technologies. Strong knowledge of Databricks and its components, including Delta Live Table (DLT) pipeline implementations. Expertise in on-premises to cloud Spark code optimization and Medallion Architecture.

Good to Have :

- Familiarity with AWS services (experience with additional cloud platforms like GCP or Azure is a plus).

- Soft Skills :

i. Excellent communication and collaboration skills, with the ability to work effectively with clients and internal teams.

- Certifications:

i. AWS/GCP/Azure Data Engineer Certification.