Project not finished(due to not enough free time) "Full-Stack" data project This project showcases different approaches in data consumption and data transform/loading To create this system, we need: Data generator(we simulating and are not scraping real data) Dockerize the whole infra(MinIO, Kafka(Brokers,Connect), Postgres) Kafka logic for data validation rules Write Kafka Consumer, Producer Write Flink Streaming job for data transformation Spark Batch jobs for MinIO data transformations and SparkSQL analytics (Optional) Write React dashboard for data viz. Spring Boot backend REST API Simple architecture diagram
