Senior Data Engineer - ETL/Apache Spark

Job Title : Senior Data Engineer

Location : Remote/Hybrid

Plant Location : Jhagadia, Bharuch, Gujarat

Type : Full-time (1 year contract)

Experience : 7 to 10 years

Function : Data Engineering / Digital Operations / Industry 4.0

About the Role

We are seeking a Senior Data Engineer to join our digital transformation team at a leading chemical manufacturing plant in India. The ideal candidate will be responsible for independently designing, developing, and deploying scalable data infrastructure to support analytics, optimization, and AI-driven use cases in a low-tech maturity environment.

You will own the data architecture end-to-end, work closely with data scientists, full stack engineers, and operations teams, and be a driving force in creating a robust Industry 4.0-ready data backbone.

Key Responsibilities

1. Data Architecture & Infrastructure

- Design and implement a scalable, secure, and future-ready data architecture from scratch.

- Lead the selection, configuration, and deployment of data lakes, warehouses (e.g., AWS Redshift, Azure Synapse), and ETL/ELT pipelines.

- Establish robust data ingestion pipelines from PLCs, DCS systems, SAP, Excel files, and third-party APIs.

- Ensure data quality, governance, lineage, and metadata management.

2. Data Engineering & Tooling

- Build and maintain modular, reusable ETL/ELT pipelines using Python, SQL, Apache Airflow, or equivalent.

- Set up real-time and batch processing capabilities using tools such as Kafka, Spark, or Azure Data Factory.

- Deploy and maintain scalable data storage solutions and optimize query performance.

3. Collaboration & Enablement

- Work closely with Data Scientists to provision clean, structured data for ML/analytics use cases.

- Support Full Stack Engineers in delivering data-driven applications.

- Educate and onboard non-technical users and plant engineers to leverage dashboards and reporting tools.

4. Strategy & Leadership

- Translate business problems into scalable data engineering solutions.

- Guide the digital transformation roadmap and ensure alignment with plant operations and management.

- Mentor junior data engineers or analysts and define best practices for the team.

Required Qualifications

Education : B.Tech/M.Tech in Computer Science, Data Engineering, or equivalent field.

Experience : 10 + years in data engineering, with 2+ years in an industrial/operations-heavy environment (manufacturing, energy, supply chain, etc.)

Proven experience independently architecting and deploying data platforms in greenfield or brownfield digital environments.

Strong hands-on expertise in :

- Python, SQL, Spark, Pandas

- ETL tools: Airflow, Azure Data Factory, or equivalent

- Cloud platforms: Azure (preferred), AWS or GCP

- Databases: PostgreSQL, MS SQL Server, NoSQL (MongoDB, etc.)

- Data lakes/warehouses: S3, Delta Lake, Snowflake, Redshift, BigQuery

- Monitoring and Logging: Prometheus, Grafana, ELK, etc.

Good to Have :

- Experience working with low maturity data environments familiarity with manual logs, Excel-based systems, or legacy SAP setups.

- Exposure to OT systems (PLC, SCADA, DCS) and IoT device integration.

- Knowledge of data mesh, data contracts, or microservice-based data flows.

- Experience with CI/CD pipelines for data applications (GitHub Actions, Azure DevOps).

Soft Skills :

- Strong problem-solving mindset and business acumen.

- Self-starter attitude and ability to work with limited supervision.

- Excellent communication skills ability to interface with plant engineers, business leaders, and technical teams alike.

- Comfortable working in ambiguous, high-stakes environments with evolving goals.

Why Join Us :

- Opportunity to lead digital transformation in core manufacturing a greenfield revolution.

- Ownership of data systems across the entire value chain from raw material to finished goods.

- Collaborative, impact-driven work environment with high visibility to leadership.

- Be a key architect in building a lighthouse-ready Industry 4.0 digital stack.