Role Overview :

As a Data Engineer, you will architect and maintain the next generation of data pipelines designed to handle high-velocity IoT and streaming data. You will work closely with cross-functional teams of data scientists, infrastructure engineers, and product stakeholders to ensure that data flows seamlessly from edge devices to analytical engines. Your work will directly influence the reliability and speed of decision-making for our clients, ensuring that our data platforms remain robust, secure, and capable of supporting mission-critical business operations.

Key Responsibilities :

- Design and implement high-throughput data ingestion pipelines using Kafka and MQTT protocols to ensure real-time data availability for downstream analytics.

- Architect scalable storage solutions leveraging Apache Iceberg and MinIO to optimize data lakehouse performance and query efficiency.

- Develop and optimize complex data transformation workflows using PySpark to process large-scale datasets across distributed environments.

- Manage and tune high-performance query engines like Trino to provide sub-second latency for complex analytical workloads.

- Integrate graph-based data models using Neo4j and relational structures in PostgreSQL to support diverse application requirements.

- Implement rigorous data governance frameworks to ensure data quality, lineage, and compliance across the entire data lifecycle.

- Lead DevOps initiatives to automate deployment, monitoring, and scaling of data infrastructure, ensuring high availability and system resilience.

- Mentor junior engineers and establish technical best practices to maintain high standards of code quality and system architecture.

Required Skillset :

- Demonstrated expertise in building and managing large-scale data engineering platforms with 8 - 16 years of relevant industry experience.

- Advanced proficiency in streaming architectures, specifically utilizing Kafka, EMQX, and MQTT protocols for IoT-heavy environments.

- Deep technical command over PySpark, Apache Iceberg, and MinIO for constructing modern, performant data lakehouse architectures.

- Proven ability to optimize query performance in Trino and manage complex data relationships within Neo4j and PostgreSQL databases.

- Strong background in DevOps practices, including CI/CD automation, container orchestration, and infrastructure-as-code to support production-grade systems.

- Exceptional analytical and problem-solving skills, with the ability to communicate complex technical concepts to non-technical stakeholders effectively.

- A collaborative mindset with the ability to thrive in a fast-paced, hybrid work environment based in Bangalore.

- A degree in Computer Science, Engineering, or a related quantitative field from a premier institute is preferred.