For knowledge document of this organization, you can follow this 📖 Data Developer & Engineer document.

flowchart LR
   1([📌 ddeutil]) --> 2([🚅 ddeutil-io]) --> 3([🏃 ddeutil-workflow]) ---> 4

   subgraph observe
      4([📡 ddeutil-observe])
      6([🔭 ddeutil-observe<br>streamlit])
   end

   3 --> 6
   0([✂️ fmtutil]) -.-> 2

This organization has the propose to make lightweight data orchestration framework for small - middle data platform project (🏃Around 10K workflows).

Firstly, I will implement base projects, 📌 Core (utility functions) and 🚅 IO (Input/Output transport utility objects) for the first dependency packages because it has a lot of base code to make main package and I do not want to develop this code on the main package, for example, it do not good if I want fix bug on the merge key function that no relate with the workflow package

🎯 The main package of this organize orchestration framework has 2 layers and I split it with 2 projects for optional installation requirement (you can only use just one of these layers without raise error).

• 🏃 Workflow - Lightweight workflow orchestration in Python with less dependencies.
• 📡 Observe (FastAPI) - Lightweight observation application with FastAPI for the workflow package.
• 🔭 Observe (Streamlit) - Lightweight observation application with Streamlit for the workflow package.

flowchart LR
   1([🏃 ddeutil-workflow]) ---> 2([⚡ deflow])
   3([🚅 ddeutil-io]) ---> 4([⚙️ jett])
   5([airflow]) ---> 6([🏭 dagtool])

• ⚡ DeFlow - Lightweight Declarative Data Workflow Framework.
• ⚙️ Jett - Just a Template Engine Tool
• 🏭 DagTool - Friendly Airflow DAG Build Tool

This organize has some mini-projects that develop for specific usecase:

• data-orchestra - Full-Stack Data Orchestration from Yaml template with Flask & HTMX
• load-routing - Routing Application Service deploy to On-Premise server with FastAPI
• 🏗️ Extensions - An additional practices to use any 3rd API connect data source.