Skip to content

jamylak/sensor_ingestion02

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Sensor Ingestion Pipeline Visualizer

This visualisation uses AI Generated code, finetuned for the best visualisation, not code quality

Interactive C + Raylib visualizer for exploring how a telemetry event changes shape as it moves from a vehicle-side sensor sample into binary encodings, protocol headers, kernel ingress, physical transport, edge fan-in, stream partitions, and backend processors.

sensor_ingestion3.mov

What This Repo Does

  • Shows the same telemetry datum across eight stages of an ingestion pipeline
  • Lets you pin a specific stage or follow the automatic live cycle
  • Compares JSON, protobuf-like, and packed-binary layouts for the same fields
  • Visualizes protocol encapsulation from payload through UDP, IP, and Ethernet
  • Breaks packet ingress into NIC, DMA, kernel queueing, and userspace handoff
  • Compares RF, fiber, and wired transmission media
  • Illustrates gateway admission, rate limiting, routing, and shard fan-out
  • Shows partitioned stream-log delivery and downstream backend processing

At A Glance

Area What You See
Vehicle-side structure A concrete telemetry message with timestamp, vehicle ID, position, and IMU samples
Binary layout The same fields rendered as JSON, protobuf-like bytes, and fixed packed spans
Protocol wrap Nested UDP, IP, and Ethernet ownership layers around the payload
Machine ingress Queueing and DMA movement from link arrival into userspace
Physical transmission How one packet becomes a medium-specific signal event
Edge fan-in Authentication, shaping, translation, balancing, and shard routing
Stream log Partition-local ordering, offsets, and different consumer positions
Backend compute Transform, rolling state, anomaly detection, storage, and observability

Core Journey

flowchart LR
    A["Vehicle Sensors<br/>timestamp + gps + imu"]
    B["Binary Layout<br/>JSON vs tagged bytes vs packed spans"]
    C["Protocol Wrap<br/>UDP -> IP -> Ethernet"]
    D["Machine Ingress<br/>NIC -> DMA -> kernel -> userspace"]
    E["Transmission Medium<br/>RF / fiber / wired"]
    F["Edge Gateway<br/>auth + rate shaping + routing"]
    G["Stream Log<br/>partition append + consumer lag"]
    H["Backend Compute<br/>state + alerts + storage + metrics"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H

    classDef source fill:#17304a,stroke:#8fcfff,color:#eef8ff,stroke-width:2px;
    classDef binary fill:#134439,stroke:#95efc4,color:#f3fff8,stroke-width:2px;
    classDef protocol fill:#573109,stroke:#ffc76b,color:#fff8eb,stroke-width:2px;
    classDef compute fill:#4a1f5b,stroke:#f0a0ff,color:#fff6ff,stroke-width:2px;

    class A source;
    class B,D,E binary;
    class C,F protocol;
    class G,H compute;
Loading

How To Read The Scene

flowchart TB
    T["Top Bar<br/>page tabs + stage slider + control hints"]
    S["Eight Large Panels<br/>one panel per ingestion stage"]
    A["ACTIVATE Button<br/>pin one stage instead of live cycle"]
    D["Detail Cards<br/>hovered or selected field/node explanation"]
    M["Minimap<br/>whole-world navigation"]
    O["Journey Overlay<br/>animated emphasis for current stage"]

    T --> S
    S --> A
    S --> D
    S --> M
    S --> O

    classDef ui fill:#12283f,stroke:#8ac6ff,color:#eef8ff,stroke-width:2px;
    classDef focus fill:#4e214d,stroke:#ff9fdc,color:#fff7fd,stroke-width:2px;
    classDef nav fill:#224835,stroke:#9ae8b7,color:#f4fff7,stroke-width:2px;

    class T,S ui;
    class A,D focus;
    class M,O nav;
Loading

Visual Legend

Visual cue Meaning
Bright panel border The currently active journey stage
ACTIVATE badge Click to pin that stage and stop following the live cycle
Coloured byte spans One field tracked through a specific encoding
Routing arrows and rails Ownership or movement through the pipeline
Consumer markers in the stream section Relative lag and replay position
Highlight card The currently hovered or selected field, node, or consumer

Controls

  • q: quit
  • 1: open the ingestion explorer page
  • 2: open the empty spare page
  • space: pause or resume the live journey cycle
  • r: reset the camera and runtime state
  • left drag: pan the world view
  • middle drag: pan the world view
  • mouse wheel: zoom around the cursor
  • h, j, k, l: fine camera nudges
  • left click on a stage ACTIVATE button: pin that stage
  • drag the top stage slider: scrub the active journey stage manually
  • right click: clear the pinned stage and return to live cycling
  • left click on highlighted protocol, gateway, stream, or backend elements: keep their detail card selected

Build

Dependencies

  • raylib
  • pkg-config
  • a C compiler such as cc or clang

Mac

brew install pkg-config raylib

Run

make run

If Raylib is not available in the default Homebrew path, set RAYLIB_DIR=/path/to/raylib.

Repo Structure

  • src/main.c: window bootstrap and app lifecycle
  • src/blueprint.c: shared camera, top bar, minimap, page shell, and common world-space helpers
  • src/page1_explorer.c: the full sensor-ingestion explorer scene and its interaction logic
  • src/mathviz.c: reusable visualization helpers used by the shared blueprint layer
  • include/blueprint.h: shared engine types and helper declarations
  • Makefile: build and run targets

About

Sensor Ingestion

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors