Skip to content

Latest commit

 

History

History

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 

README.md

Divemap LLM Content Worker

This Cloudflare Worker is responsible for serving llms.txt and related markdown documentation files directly from Cloudflare R2 storage, ensuring high availability and zero load on the main application backend for these resources.

Why this exists

  1. Reliability: It serves content even if the main Fly.io backend is down.
  2. Performance: Files are served from Cloudflare's edge network, closer to the user/crawler.
  3. Efficiency: Offloads static file serving from the application server.

How it works

The worker intercepts requests to divemap.gr for specific files:

  • llms.txt
  • dive-sites.md
  • dive-routes.md
  • diving-centers.md
  • dives.md

When a request matches one of these files:

  1. The worker checks the divemap-prod R2 bucket for the corresponding file in the llm_content/ folder.
  2. If found, it serves the file with aggressive caching headers:
    • Cache-Control: public, max-age=86400, stale-while-revalidate=86400, stale-if-error=86400
  3. If not found (or on error), it falls back to fetching from the origin server (the main website).

Directory Structure

The worker project is located in divemap-llm-worker/late-moon-cc3c (the default name generated by Wrangler).

  • src/index.ts: The main worker logic.
  • wrangler.jsonc: Cloudflare configuration (routes, bindings).

Development

Prerequisites

  • Node.js installed
  • npm installed
  • Access to the Cloudflare account

Setup

cd divemap-llm-worker/late-moon-cc3c
npm install

Deployment

To deploy updates to the worker:

cd divemap-llm-worker/late-moon-cc3c
npx wrangler deploy

This command will:

  1. Build the TypeScript code.
  2. Upload the worker to Cloudflare.
  3. Update the route configurations.

Updating Content

The actual content (llms.txt, *.md) is generated by the backend task backend/generate_static_content.py. This script runs daily (or manually) and uploads the fresh content to the divemap-prod R2 bucket. The worker then serves whatever is in that bucket.

To force a content update:

  1. Run the generation script in the backend.
  2. The worker will serve the new content once the cache expires (24h) or immediately if the cache is purged.