AWS Data Engineer - PySpark/Business Intelligence

Responsibilities :

- Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and other relevant technologies.

- Build and optimize ETL processes to ingest, transform, and load data from various sources into the data warehouse.

- Implement data modeling and data warehousing solutions to support business intelligence and analytics needs.

- Develop and maintain data governance policies and procedures to ensure data quality, security, and compliance.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver data solutions.

- Monitor and troubleshoot data pipeline performance issues and implement optimizations.

- Participate in code reviews, testing, and deployment processes.

- Stay up-to-date with the latest data engineering technologies and best practices.

Qualifications :

- Strong programming skills in Python, PySpark, and SQL.

- Hands-on experience with Databricks and AWS cloud services (S3, IAM, Lambda, etc.).

- Experience with workflow orchestration tools such as Apache Airflow.

- Familiarity with FastAPI for building high-performance APIs.

- Solid understanding of data modeling, data warehousing, and ETL processes.

- Experience with version control systems (e.g., Git) and CI/CD pipelines.

- Strong problem-solving skills and ability to work in a fast-paced environment.

- Good communication skills and ability to collaborate in cross-functional teams.

- Experience with data governance, security, and compliance best practices.

- Proficiency in using Spotfire for data visualization and reporting.

- Experience with Databricks Unity Catalog or similar data governance and metadata management tools.

Preferred Qualifications :

- Familiarity with machine learning workflows and MLOps.

- Certifications in Databricks, AWS.

- Experience implementing data mesh or data fabric architectures.

- Knowledge of data lineage and metadata management best practices.

Tech Stack : Databricks, Python, PySpark, SQL, Airflow, FastAPI, AWS (S3, IAM, ECR, Lambda), Spotfire.