I build data platforms and cloud infrastructure, then make them smaller.
I'm Rishabh Mehta, a cloud and data engineer at Jio Platforms in Mumbai. In the last three years I built a Rust framework that replaced Spark across 17 application groups, moved 400+ Spark jobs to Structured Streaming, and scaled ClickHouse past two trillion rows. Now I'm building the cloud and network foundations of India's largest AI datacenter.
- Role
- Cloud & Data Engineer (SDE)
- Team
- Jio Cloud
- Focus
- Rust, Spark, ClickHouse, Kubernetes, AI infra
- Recognition
- Star Performer, Jio Cloud Townhall
- Before
- Google DSC Lead, B.E. 9.3 CGPA
- Based
- Mumbai · UTC+5:30
8.7 TB→<200 GB
Executor RAM across 17 application groups after moving them from Spark to Rust
3,500→~200
CPU cores for the same log-parsing workload
200+ PB
IPDR data on ClickHouse, on roughly a tenth of the servers first planned
<20 ms
Subscriber lookups on ClickHouse tables past 2 trillion rows
The rewrite
Same traffic, 48 times fewer cores.
The log parser for MyJio ran as a Spark 2 job calling Perl, sized at 120 executors to keep up. The Rust version reads the same Kafka topic, matches the old output field for field, and fits in one pod. Toggle between them.
Figure 1
Spark to Rust, measured
120 executors × 4 cores on ~12 nodes1 cell = 1 core
- CPU cores
- 480
- Memory
- 480 GB
- Footprint
- 12 nodes
- Batch latency
- 10 s
Selected work
Mostly infrastructure nobody sees until it breaks.
- Rust · Kafka · Kubernetes
Replaced Spark with a Rust framework I designed
Rewrote the MyJio real-time log parser, proved parity on live traffic, took it to production, then turned it into DataCraft so the other sixteen application groups could follow.
Read more → - ClickHouse · Linux
Telecom-scale analytics on a fraction of the hardware
Sort-key, partition and ingestion design on a nationwide IPDR platform with tables past two trillion rows. The legacy plan needed 1,700 servers. It runs on fewer than 180.
Read more → - Scala · Spark · Kafka
400+ Spark jobs moved to Structured Streaming
Took the in-house ETL framework off the legacy DStream API. About 25% more throughput per core across the fleet, and a custom Spark UI built into the framework itself.
Read more → - BlueField-3 · WireGuard · F5
Cloud and network foundations for an AI datacenter
Out-of-band DPU onboarding over Redfish, a self-hosted WireGuard mesh into GPU pods, bastion access, and durable Temporal workflows for firewall changes.
Read more → - SNMP · Cisco · Spark
The 497-day bug
Switches and routers with long uptimes were showing phantom outages in SLA reports. The cause was a 32-bit SNMP counter quietly wrapping around.
Read more → - MCP · Python · Kubernetes
AI tooling for the data platform
An in-house Model Context Protocol server that lets AI coding tools scaffold, validate and test-run pipelines against the platform.
Read more →
Systems
Bare-metal onboarding through the DPU
In a bare-metal GPU cloud the tenant owns the host. So the platform lives on the DPU instead: provisioned out-of-band over Redfish, then joined to Kubernetes as an ARM64 worker, with no agent on the host at all.
Writing
Notes from production.
- Moving 400+ Spark jobs from DStreams to Structured StreamingA naive port made our hardest job twice as slow and quietly lost data. Here is what it took to reach parity and then pull ahead: bounded batches, fewer scheduling waves, a watermark guard, two timezone bugs and one very deep query plan.6 min
- Building the Spark UI that Structured Streaming should have hadThe stock Spark UI forgets a streaming query the moment it stops, and it cannot tell you how far behind Kafka you are. So I built a Streaming Stats tab into our framework: a SparkPlugin, a query listener, a ring buffer and server-rendered SVG.5 min
- Replacing a 480-core Spark job with 10 cores of RustHow the MyJio log parser went from 120 Spark executors to a single pod, what parity testing looked like, and the thread-count bug that almost made it look worse than it was.5 min