Senior Data Engineer - AWS/Spark

Data Engineer Spark | Databricks | AWS

Key Skills :

- Design and build scalable batch and streaming data pipelines using PySpark and Spark SQL.

- Process and transform data using formats such as JSON, Parquet, CSV, and ORC.

- Optimize Spark and PySpark code for performance and cost efficiency in cloud environments.

Cloud & Migration Leadership :

- Lead on-prem to cloud migration projects, primarily on AWS, using Spark and Databricks.

- Develop and optimize cloud-native Spark pipelines to support large-scale data workloads.

Databricks & Medallion Architecture :

- Implement Medallion Architecture (Bronze, Silver, Gold layers) for reliable and scalable data systems.

- Design and manage Delta Live Tables (DLT) pipelines for secure, auditable, and high-quality data processing.

- Use Unity Catalog for data governance, access control, and cataloging.

Data Modeling & Quality :

- Build and maintain robust data models aligned with business requirements.

- Enforce data quality, consistency, and governance standards across pipelines.

Stakeholder Collaboration :

- Work closely with clients and internal teams to design best-practice cloud data solutions.

- Provide guidance on architecture, performance tuning, and governance.

Experience & Education :

- Proven experience in cloud-based data migration projects.

- Bachelors or higher degree in Computer Science, Data Engineering, or related field.

Good to Have :

- Knowledge of data governance, security, and cost optimization in cloud environments.

Certifications (Preferred) :