Tags: nicnocquee/dataqueue
Tags
Add job dependencies (dependsOn) across Postgres, Redis, demo, and do… …cs (#40) ## Summary Adds optional job prerequisites via `dependsOn`: wait for listed jobs to complete and/or for tag-based “drain” barriers. Implements persistence and scheduling in PostgreSQL and Redis, exposes types and helpers (including batch references for `addJobs`), and documents the feature with a demo and API/usage pages. ## Important changes - New `dependsOn` on job options: `jobIds` (all must complete) and `tags` (active barrier until no matching in-flight work) - PostgreSQL migration adds dependency columns; Postgres and Redis backends enforce prerequisites and cancel dependents when prerequisites fail or are cancelled - `batchDepRef` / resolution for dependencies inside the same `addJobs` batch - Demo: Dependencies page and monitor updates to illustrate behavior ## Other changes - Docs: `job-dependencies` usage page, API cross-links, `llms.txt` / `llms-full.txt` updates - Website copy tweaks; AI rules/skills updated for the new feature - `@nicnocquee/react` types extended where needed ## How to test 1. Run the demo app, open the new Dependencies section from the sidebar, and run the dependency demo; confirm jobs order and blocking match the UI. 2. Add jobs with `dependsOn.jobIds` and `dependsOn.tags` via the library or demo; verify a dependent stays pending until prerequisites complete, and that failure/cancel of a prerequisite cancels pending dependents as documented. 3. With Postgres: apply migrations, enqueue dependent jobs, and confirm rows/constraints behave as expected. 4. With Redis backend: repeat key flows and confirm equivalent behavior. 5. Skim `apps/docs/content/docs/usage/job-dependencies.mdx` and linked API pages; confirm examples and behavior descriptions match the implementation. --------- Co-authored-by: Nico Prananta <[email protected]>
Update MCP command to use scoped package name (#39) ## Summary Updates MCP install and config to use the scoped package name `@nicnocquee/dataqueue` instead of `dataqueue-cli`, so the MCP server command matches the published package and works correctly when users run the install flow. ## Important changes - Default MCP server config now uses `npx @nicnocquee/dataqueue mcp` instead of `npx dataqueue-cli mcp` - Success log message after MCP install shows the new command - Existing MCP configs written by this flow will use the scoped package name ## Other changes - Test expectations and fixtures updated to assert the new command in `install-mcp-command.test.ts` ## How to test 1. From the repo root, run the MCP install (e.g. `npx @nicnocquee/dataqueue mcp install` or the equivalent entry point). 2. Confirm the generated MCP config (e.g. in `~/.cursor/mcp.json` or the path you use) has `args: ['@nicnocquee/dataqueue', 'mcp']` for the dataqueue server. 3. Run `pnpm test` in `packages/dataqueue` and confirm all tests pass. Co-authored-by: Nico Prananta <[email protected]>
Update CLI commands to use package name @nicnocquee/dataqueue (#38) ## Summary Updates all CLI invocations and docs to use the scoped package name `@nicnocquee/dataqueue` instead of `dataqueue-cli`. Adds a `dataqueue` bin alias and adjusts package `files` so the published CLI is consistent and discoverable under the new name. ## Important changes - Docs and code now reference `npx @nicnocquee/dataqueue <command>` (e.g. `install-mcp`, `mcp`, `migrate`) instead of `npx dataqueue-cli` - `package.json`: new bin `dataqueue` (alongside `dataqueue-cli`), and `cli.cjs` added to `files` so the CLI is included in the published package - `install-mcp-command` and its test output the new run command: `npx @nicnocquee/dataqueue mcp` ## Other changes - CLI index, init, install-mcp, install-rules, install-skills, mcp, and migrate docs updated to the new command - Usage docs (quick-start, building-with-AI, cron-jobs) and `llms-full.txt` updated to the new package name ## How to test 1. From repo root, run `pnpm build` in `packages/dataqueue` and confirm it succeeds. 2. Run `npx @nicnocquee/dataqueue --help` (or link the package locally) and confirm the CLI runs. 3. Run the install-mcp flow (or unit tests for it) and confirm the printed command is `npx @nicnocquee/dataqueue mcp`. 4. Skim the updated docs and `llms-full.txt` and confirm every CLI example uses `@nicnocquee/dataqueue`. Co-authored-by: Nico Prananta <[email protected]>
Fix migration env loading when using --envPath (#37) - Added dotenv dependency to load environment variables from a specified file path. - Implemented `loadEnvFromPath` function to handle the `--envPath` argument, ensuring environment variables are loaded before executing commands. - Updated tests to verify the correct invocation of `loadEnvFromPath` when the `--envPath` option is used. These changes improve the flexibility of the DataQueue CLI, allowing users to manage environment configurations more effectively. Co-authored-by: Nico Prananta <[email protected]>
Update comparison documentation and component to include pg-boss (#36) - Modified the comparison documentation to add pg-boss alongside BullMQ and Trigger.dev, providing a more comprehensive overview of job queue options. - Updated the ComparisonRow type and related components to include pg-boss features, ensuring accurate representation in the comparison table. - Enhanced the UI to display pg-boss data in the comparison section, improving clarity for users evaluating job queue solutions. These changes enhance the documentation and user interface, making it easier for users to compare DataQueue with additional job queue options. Co-authored-by: Nico Prananta <[email protected]>
Add Next.js shortcut to quick-start documentation - Introduced a new section in the quick-start guide for Next.js users, detailing a shortcut command to scaffold the project structure with DataQueue. - Included instructions on using the `npx dataqueue-cli init` command, which auto-detects project configurations and sets up necessary files. - Updated the documentation to enhance user experience and streamline the setup process for Next.js projects. These changes improve the onboarding experience for users integrating DataQueue with Next.js.
Add init command support to CLI and enhance tests - Introduced the `init` command in the CLI, routing it to a new `runInitImpl` function for initializing project structure. - Updated CLI usage messages to include the new `init` command for better user guidance. - Added comprehensive tests for the `init` command, ensuring proper functionality and integration with the CLI. - Created a new `init-command.ts` file to handle initialization logic, including file creation and package updates. These changes improve the CLI's usability and extend its functionality, providing users with a streamlined way to set up their projects.
Redis-parity (#25) * Implement waitpoint and step-data support in QueueBackend - Added new methods to the QueueBackend interface for managing job waitpoints and persisting step data, enhancing job processing capabilities. - Implemented waitJob, updateStepData, createWaitpoint, completeWaitpoint, and getWaitpoint methods in the PostgresBackend and RedisBackend classes. - Updated the initJobQueue function to support wait/token features across all backends. - Refactored existing queue functions to utilize the new backend methods, improving code clarity and maintainability. - Enhanced documentation to cover the new waitpoint functionality and its usage. These changes significantly improve the flexibility and robustness of job management in DataQueue. * Update documentation to reflect Redis (beta) support and clarify processor initiation in serverless environments - Updated the documentation to indicate that Redis is in beta as a storage backend for DataQueue. - Clarified the instructions for initiating the processor in serverless environments, including the use of cron jobs for periodic execution. These changes enhance the clarity and accuracy of the documentation, ensuring users are well-informed about the current state of features and best practices. --------- Co-authored-by: Nico Prananta <[email protected]>
Add cron scheduling feature to DataQueue (#22) * Add cron scheduling feature to DataQueue - Introduced a new cron scheduling system allowing users to define recurring jobs using cron expressions. - Implemented backend functionality for adding, listing, pausing, resuming, and removing cron schedules. - Created a user interface for managing cron schedules in the demo application, including a dedicated page and components for adding schedules. - Enhanced documentation to cover the new cron jobs feature, including usage examples and API endpoints. - Updated the pnpm-lock.yaml to include the croner package for cron expression handling. These changes significantly enhance the job scheduling capabilities of DataQueue, providing users with powerful tools for managing recurring tasks. * Update build script in demo package.json to include data migration step - Modified the build script to run `dataqueue-cli migrate` before executing the Next.js build process. This change ensures that any necessary data migrations are applied prior to building the application, improving deployment reliability. * Pass through the env * Update turbo.json to change passThroughEnv to globalPassThroughEnv for improved environment variable handling --------- Co-authored-by: Nico Prananta <[email protected]>
PreviousNext