Happiest Minds - Module Lead - Big Data

Job Title : Module Lead - Big Data

About the Role :

We are looking for a skilled Module Lead - Big Data with strong experience in Big Data and Data Engineering. The candidate should have hands-on expertise in building scalable data pipelines, ETL/ELT processes, cloud-based data platforms, and modern data engineering technologies.

Key Responsibilities :

- Use the latest technologies to build data pipelines and integrate machine learning models.

- Build and expand the enterprise data platform.

- Develop applications that run on Google Cloud-based infrastructure.

- Design and develop data pipelines and ETL/ELT processes.

- Work with advanced SQL and data warehousing concepts including BigQuery.

- Develop solutions using :

i. Python

ii. PySpark

iii. SQL

- Work with relational and NoSQL databases including :

i. PostgreSQL

ii. MongoDB

- Use workflow management tools such as :

i. Airflow

ii. AWS Data Pipeline

iii. Google Cloud Composer

- Build Docker images and deploy them to production environments.

- Integrate Docker container orchestration frameworks using Kubernetes.

- Create :

i. Pods

ii. ConfigMaps

iii. Deployments

using Terraform.

- Work with Azure DevOps or similar CI/CD tools and Git.

Required Skills & Experience :

- 6+ years of experience in :

i. Big Data

ii. Data Engineering

- Strong knowledge of :

i. Advanced SQL

ii. Data Warehousing concepts

iii. BigQuery

- Strong programming skills in :

i. SQL

ii. Python

iii. PySpark

- Experience with :

i. Data pipeline development

ii. ETL/ELT processes

- Experience working with one of the public cloud providers :

i. GCP

ii. Azure

iii. AWS

- Experience with relational SQL and NoSQL databases.

- Experience with workflow management and orchestration tools.

- Hands-on experience with :

i. Docker

ii. Kubernetes

iii. Terraform

- Familiarity with :

i. Azure DevOps

ii. CI/CD tools

iii. Git

- Strong problem-solving and communication skills.

Nice to Have :

- Experience with :

i. C#

ii. .NET Core

iii. Microservices

Education :

- Bachelor's degree or advanced degree in Computer Science or related engineering discipline.

Key Skills :

- Python

- PySpark

- Data Engineering

- ETL

- SQL

- BigQuery

- Kubernetes

- Docker

- Terraform

- GCP

- Airflow

- MongoDB

- PostgreSQL