Cognite - Senior Data Platform Engineer - Python/Scala

About The Role :

We're seeking a Senior Data Platform Engineer who excels at building high-performance distributed systems and thrives in a fast-paced startup environment. You'll be working on cutting-edge data infrastructure challenges that directly impact how Fortune 500 industrial companies manage their most critical operational data.

How You'll Create An Imapct :

- Build high-performance data pipelines using Spark, Flink, and Kafka to process terabyte-scale industrial datasets.

- Develop low-latency APIs and services supporting thousands of concurrent users with sub-second response times.

- Optimize time-series, sensor, and operational data storage and retrieval for massive scale.

- Engineer distributed processing solutions, including real-time streaming that handles millions of events per second.

- Design and evolve cost-efficient data lake architectures (S3/GCS) using modern formats like Parquet/ORC.

- Optimize OLAP query performance across systems such as ClickHouse, Pinot, or Druid.

- Build fault-tolerant, highly-available systems with circuit breakers, retries, and distributed tracing.

- Drive technical innovation through research, prototyping, and contributions to open-source.

- Collaborate with product and architecture teams to translate complex industrial data problems into scalable, production-grade solutions.

What You Bring :

- Deep expertise in Spark internals, including Catalyst optimizer, partitioning, and performance tuning.

- Strong hands-on experience with Flink/Kafka for both streaming and batch workloads.

- Proven ability to scale systems to 10K+ QPS with strong foundations in JVM performance, GC tuning, and memory management.

- Experience building high-throughput REST/gRPC services and advanced caching with Redis/in-memory data structures.

- Strong understanding of monitoring and observability (Prometheus, Grafana, OpenTelemetry).

- Solid architectural thinking with an ability to design distributed, secure, and resilient platforms.

- Collaborative mindset with the ability to work across engineering, product, and platform teams.

Qualifications :

- 12-15 years of experience in distributed data systems, backend engineering, or platform engineering.

- Strong programming skills in Scala, Java, or Python.

- Experience with cloud platforms (AWS/GCP/Azure) and data lake/object storage.

- Prior experience with large-scale data, OLAP systems, or industrial/IoT data is a plus.

- Contributions to open-source or experience in industrial data/AI platforms is an added advantage.

Whar We Offer :

- Opportunity to build world-class Industrial DataOps systems powering global energy, manufacturing, and industrial AI customers.

- A fast-growing, high-ownership environment in Cognites India R&D Center.

- Work on cutting-edge distributed systems, streaming architectures, AI-ready data platforms, and industrial-scale challenges.

- A culture that rewards innovation, autonomy, continuous learning, and open-source contribution.

- Competitive compensation, equity, and benefits aligned with Cognites global standards.