Job Description: Big Data Platform Engineer

Role Summary:

Total Experience: 6 - 12 years

Location: Bangalore, Chennai

Key Responsibilities:

Data Engineering & Development:

- Design, develop, and maintain scalable data pipelines using Python and Apache Spark.

- Implement efficient ETL/ELT processes for large-scale structured and unstructured datasets.

- Develop and optimize complex SQL queries, data models, and transformations.

- Ensure data quality, integrity, and reliability across the platform.

Platform Operations:

- Work with Linux-based environments for deployment, troubleshooting, and performance tuning.

- Develop and maintain shell scripts for automation and operational tasks.

- Monitor and optimize Spark jobs for performance, scalability, and resource utilization.

DevOps & Automation:

- Implement CI/CD pipelines and deployment automation.

- Participate in infrastructure provisioning, monitoring, and release management activities.

- Collaborate with DevOps teams to improve platform reliability and operational efficiency.

Collaboration & Governance:

- Work closely with Data Architects, Product Owners, and Business Stakeholders.

- Participate in code reviews and ensure adherence to engineering best practices.

- Create and maintain technical documentation and operational runbooks.

Mandatory Skills:

Core Big Data Platform Skills:

- Strong programming experience in Python.

- Hands-on expertise with Apache Spark (PySpark preferred).

- Strong SQL development and query optimization skills.

- Apache Airflow for workflow orchestration and scheduling.

- Kubernetes (AKS/SKE) for container orchestration and deployment.

- MinIO / S3 Compatible Object Storage.

- Apache Iceberg/Delta Lake / Apache Hudi.

Technical Competencies:

- Data Pipeline Development, ETL/ELT Processing, Data Modelling, Performance Tuning, Version Control (Git), Agile/Scrum Delivery Model.

Good to Have Skills:

- Basic Linux Administration, Shell Scripting (Bash/KSH), DevOps practices, CI/CD pipelines, Docker, Cloud Platform Experience (Azure/AWS), Monitoring & Logging Tools.

Platform Development:

- Experience working with enterprise-scale data platforms, distributed computing environments, and strong analytical/problem-solving skills.