fin1te

Systems

Six systems, drawn the way I explain them on a whiteboard.

Click any component to see what it does, what it talks to, and the trade-off behind it. Names and topology are generalised for a public page; the engineering decisions are the ones we made in production.

01 · Rust ETL framework

DataCraft runtime

A pipeline is a manifest. The manifest runs on one native binary per pod, three pods sharing a consumer group. Inside, stages are joined by bounded channels, so backpressure is built in and memory stays flat under load.

Fig. 01DataCraft runtime15 components · 15 links
Kubernetes · 3 replicas, one consumer groupPod · one native binaryAuthoring & deliverycommit offsetsdeploy ×3Kafka topicup to 110 partitionsConsumerrdkafka, 1 per podChannelboundedWorker poolparse · enrichBatchersize / time flushHealth/metrics · probesClickHousebatched insertsKafkaparsed topicsObject storagereplay, retentionPrometheuslag · rate · healthPipeline editordrag and dropManifestversioned in GitCIAzure DevOpsContainer imageone static binaryCargo registrydc-sdk crates
  • MyJio pilot on this runtime: 480 cores and 480 GB of Spark executors replaced by 10 cores and 4.2 GB peak.
  • The one production scare was a volume gap during the parallel run. Cause: 48 worker threads on 12 cores and a single consumer reading 110 partitions. Fixed by sizing workers to cores and running three pods.

02 · Telecom-scale columnar storage

IPDR analytics on ClickHouse

Every hour, session and lookup feeds from across the network land as files, get parsed inside ClickHouse itself, and are routed by a hash of the subscriber IP so every record for an IP lands on the same shard. That makes a six-step identity join entirely local: no shuffle, no network. Enriched rows ship compressed to a replicated serving cluster. Hostnames, node counts and anything operational are left out.

Fig. 02IPDR analytics on ClickHouse18 components · 13 links
Landing zoneProcessing cluster · N shardsServing clusterBatch control planeunmatchedlz4Session feedsCGNAT · 4G · 5G · FTTHLookup feedscharging · AAA · WiFiHourly dropsSFTP / NFSRedistributorbyte-weightedFile lifecyclelanding → processedServer-side ingestfile() · parallel parseDistributed insertsipHash64(subscriber IP)Co-located RAW tablessession + lookup, same shardChained join × 6local, zero shuffleUnresolved buffer8 h TTL, retriedEnriched tablesORDER BY subscriber, timeReplicated storeReplicatedMergeTreeLookup APIsubscriber + timeClickHouse Keeperreplication quorumBatch orchestratorthread census · staggerMerge governorSTOP / START MERGESFile trackerReplacingMergeTree, idempotentPipeline monitorheatmap per feed, region
  • Server count: the legacy design used 1,200+ servers and was projected to need 500 more. Co-locating joins on ClickHouse brought it under 180.
  • The biggest single idea is boring: shard every feed by the same hash of the same key, and a distributed join stops being distributed.

03 · Availability accounting

Cloud SLA engine

A daily Spark job that turns raw monitoring into an auditable downtime record per asset, per day. Restarts are found in uptime counters, HA pairs are checked second by second, silent multi-day outages are back-dated, and every outage is cut against asset lifetimes and approved change windows before anything is counted.

Fig. 03Cloud SLA engine16 components · 15 links
SourcesSpark job · runs daily for D−1Audit martMonitoringuptime countersProblem eventsICMP node-downCMDBlifetime · scopeChange ticketsITSM APIRestart detectionLAG(uptime) dropsRollover guardICMP cross-checkHA overlap1 s explode + self-joinProblem windowsclipped to the dayLifecycle clippingkeep only created < t < retiredRetrofitback-date recoveryDay splitterone row per dayChange windowsplanned vs unplannedNoise filter≤ 60 s droppedDowntime martasset × dayUptime summaryMERGE dailySLA reportsper asset, customer
  • Six corrections stand between raw monitoring and a trustworthy number: restarts, counter wraps, HA overlap, silent multi-day outages, asset lifetimes and change windows.
  • Nothing is overwritten. Every derived row sits next to the raw row it came from, so any disputed number can be traced back to the samples behind it.

04 · BlueField-3 · DPF · Redfish

Bare-metal onboarding through the DPU

In a bare-metal GPU cloud the tenant owns the host. So the platform lives on the DPU instead: provisioned out-of-band over Redfish, then joined to Kubernetes as an ARM64 worker, with no agent on the host at all.

Fig. 04Bare-metal onboarding through the DPU7 components · 7 links
Management clusterBare-metal serverRedfishflashjoinPCIeKubernetes APIcontrol planeDPF operatorDOCA Platform FrameworkBFB imageDPU OS bundleBMCRedfish, OOBBlueField-3 DPUARM cores, own OSkubeletARM64 workerHost CPUs and GPUstenant-owned, untouched by the platform
  • Why out-of-band: using the host to provision its own DPU would give the platform a foothold on hardware it is supposed to hand over clean.
  • Getting the first DPU to 9 of 9 health checks was mostly firmware and network sequencing, not Kubernetes.

05 · Headscale · WireGuard · DERP

Private mesh into GPU pods

Engineers get a direct, encrypted path from their laptop into a GPU pod, with no public IPs, jump hosts or port forwards. The control plane is self-hosted, so no traffic or metadata leaves the network.

Fig. 05Private mesh into GPU pods6 components · 7 links
Corporate networkGPU clusterkeysWireGuard, direct UDPEngineer laptopTailscale clientHeadscalecontrol planeDERP relayinternal, fallbackWireGuard netnsisolated interfaceGPU workloadnotebook, trainingCalico pod networkuntouched
  • The bug that took longest: Calico and WireGuard both believed they owned the pod's routes. A separate namespace ended the argument.

06 · Temporal · Crossplane · F5 BIG-IP

Firewall change automation

Create, modify and delete of ACL rules on F5 BIG-IP, end to end, from a portal request to a device change. Temporal makes every change durable and reversible; Crossplane makes the desired state declarative.

Fig. 06Firewall change automation7 components · 7 links
OrchestrationcompensatePortal requestcreate / modify / deleteAPIGEEVTL schemasBlueprintRavelWorkflowTemporalCompositionCrossplane XRF5 BIG-IProute domain ACLsCMDBrecord of change
  • The hard part is not the happy path. It is making modify and delete safe when the rule order on the device has drifted from what the system thinks it is.