A repository dedicated to the entire data lifecycle: from acquisition with web scraping and bots, through analysis, SEO and modeling with Machine Learning, to final presentation in web applications, interactive visualizations and generative patterns.

Welcome to data_sci-webdev! This space is a laboratory for creating data-driven systems and applications. We believe that the true value of information is unlocked when it is intelligently collected, thoroughly analyzed, and presented through interactive, secure, and aesthetically rich experiences.

Our approach is to build a bridge between technical complexity and human experience, following these principles:

1. End-to-End Lifecycle: Tracking data from its acquisition (scraping, APIs), storage (databases), modeling (ML/DL), and analysis (Analytics/SEO) to its presentation in rich web interfaces.

2. Visualization as a Language: Treating data visualization not as a final step, but as an essential language for communicating insights, exploring everything from traditional infographics to dynamic ideography and generative patterns.

3. Security and Privacy by Design: Integrating NetSec and Privacy concepts at every stage, ensuring that data is handled ethically and securely.

4. Agile and Interdisciplinary Prototyping: Building proof-of-concept projects that combine diverse areas, demonstrating the value of an idea quickly and effectively.

This repository explores a wide range of interconnected domains:

• Data Science & Machine Learning:

• Data Analytics

• Machine Learning (ML)

• Deep Learning (DL)

• Web Development & Engineering:

• Frontend Development (GUI, UX/UI)

• Backend Development (Servers, APIs)

• Search Engine Optimization (SEO)

• Data Acquisition and Storage:

• Web Scraping

• Databases (SQL, NoSQL - MongoDB, JSON)

• Visualization and Interaction:

• Data Visualization (Reference Catalog)

• Generative Patterns (Asemic Writing, Infographics)

• Infrastructure and Automation:

• Servers and Deployment

• Bots (Discord, Telegram, Twitter)

• Network Security and Privacy (NetSec/Priv8)

The content is organized into modules that reflect the data lifecycle.

• 01-Data-Acquisition-and-Storage/

Focus on collecting and storing data from various sources.

• 01.1-Web-Scraping/ (Ex: Scripts in Python/BeautifulSoup, R/rvest, JS/Puppeteer to collect data from websites)

• 01.2-API-Integration/ (Ex: Connecting to APIs to collect data in a structured way)

• 01.3-Database-Schemas/ (Ex: Models and schemas for SQL, MongoDB, and JSON)

• 02-Analysis-and-Modeling/

Where data is processed, analyzed, and transformed into models.

• 02.1-Analytics-and-SEO/ (Ex: Notebooks for traffic analysis, keyword optimization, reports)

• 02.2-Machine-Learning-Models/ (Ex: Implementations of regression, classification, and clustering models)

• 02.3-Deep-Learning-Projects/ (Ex: Projects with neural networks (CNNs, RNNs) for image, text, etc.)

• 03-Visualization-and-Frontend/

The presentation layer: transforming data into interfaces and art.

• 03.1-GUI-and-UX-UI-Principles/ (README.md with design principles for data interfaces)

• 03.2-Data-Visualization-Cookbook/ (Ex: Implementations of different types of graphs - D3.js, Plotly, Matplotlib)

• 03.3-Generative-Patterns/ (Ex: Generative art projects with data, explorations of asemic writing)

• 04-Full-Stack-Data-Applications/

Complete projects that integrate all previous modules.

• 04.1-Real-Time-Analytics-Dashboard/ (Ex: Complete dashboard with scraping, processing, and real-time visualization)

• 04.2-ML-Model-as-a-Service/ (Ex: An API that serves an ML model and a frontend to interact with it)

• 05-Infrastructure-and-Security/

The foundation that supports and protects the applications.

• 05.1-Servers-and-Deployment/ (README.md with guides for deploying apps (Docker, Nginx, etc.))
• 05.2-Bots-as-Interfaces/ (Ex: Discord/Telegram bots that serve data or insights from models)
• 05.3-NetSec-and-Privacy/ (README.md with best practices for data security, anonymization, etc.)
• 05.4-AI-Gateway-Infrastructure/

• Data Collection (Web Scraping): Python (BeautifulSoup, Scrapy, Selenium), R (rvest), JavaScript (Puppeteer, Cheerio), Ruby (Nokogiri), PHP (Goutte)
• Backend & Machine Learning: Python (Pandas, Scikit-learn, TensorFlow, PyTorch), FastAPI, Flask, Django
• Frontend & Visualization (GUI): HTML5, CSS, XML, JavaScript, TypeScript, React, Vue, Svelte, D3.js
• Databases: SQL (PostgreSQL, MySQL), NoSQL (MongoDB), JSON
• Infrastructure & Bots: Docker, Nginx, Discord.py, python-telegram-bot

• glowing-system: Acts as the main data ingestion system via API that feeds the projects in this repository.

• computhink-101: Provides the algorithmic foundations for ML models and for optimizing application performance.

• learning-how_to_learn: "Dynamic ideography" is a central concept for data visualization and the creation of generative patterns.

This repository is distributed under the MIT license.