You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Objective: Create a web application that scrapes, visualizes, and allows users to save data via a Flask API.
Tools and Technologies:
Python (Pandas, Matplotlib, Flask, BeautifulSoup for scraping)
PostgreSQL on Neon.tech
Render for deploying the Flask API
🌐 2. Web Scraping Setup
Task: Scrape weather data from the National Weather Service website.
Details:
Use Python’s requests and BeautifulSoup libraries to scrape weather forecasts.
Example URL: https://forecast.weather.gov/MapClick.php?lat=37.7772&lon=-122.4168.
🗃️ 3. Database Setup
Task: Set up and configure the PostgreSQL database on Neon.tech.
Details:
Design and create database schema suitable for storing scraped weather data, user settings, and saved visualizations.
Establish database connections and authentication securely.
📊 4. Data Manipulation with Pandas
Task: Write Python scripts to process and prepare data for visualization.
Details:
Use Pandas to clean and transform scraped data into a usable format for analysis.
📉 5. Data Visualization with Matplotlib
Task: Develop Python scripts to create visualizations from the processed data.
Details:
Generate charts such as line graphs and bar charts to represent the weather trends visually.
🚀 6. Flask API Development with Save Feature
Task: Create a Flask API to serve and save the data and visualizations.
Details:
Develop endpoints that perform data processing, return visualizations, and handle data saving requests based on user input (e.g., ?save=true).
Implement SQL operations for INSERT and UPDATE to manage user data and visualizations in the PostgreSQL database.
🌌 7. Serving Images through Flask
Task: Configure Flask to serve images dynamically.
Details:
Implement routes that create and stream images directly to the client without intermediate file storage.
🛠️ 8. Deployment on Render
Task: Deploy the Flask application to Render.
Details:
Prepare the application for deployment, including environment setup and deployment configuration.
🧪 9. Testing and Validation
Task: Ensure the full system works as intended.
Details:
Conduct thorough testing of web scraping, data processing, visualization generation, API responsiveness, and database interactions.
📝 10. Documentation and Final Submission
Task: Document the entire project process and setup.
Details:
Write comprehensive documentation covering how to set up the project, run the API, and interact with the system. Include examples of saving and retrieving data.
