This directory contains the testing and Quality Assurance infrastructure for oshea, including test execution, analytics, and intelligent monitoring systems. Test scenarios are defined in YAML manifests *.manifest.yaml, while the test runner dynamically instantiates test cases from manifests, applying stubs, mocks, and asserts.

Specifications are defined in structured YAML metadata where tests are organized by increasing scope and integration complexity.

• Level 1 Tests: Module integration scenarios
• Level 2 Tests: Subsystem integration scenarios
• Level 3 Tests: End-to-End CLI scenarios
• Level 4 Tests: End-to-End lifecycle scenarios

CLI applications derive more value from integration-level testing, as meaningful behavior occurs at module boundaries, file system operations, and cross-system coordination rather than isolated function calls.

Together with .mocharc.js, these manifests provide a strategic mechanism of the CI:

• Test levels, ranks, and groups
• Source file mappings
• Command specifications
• Documentation references

The project uses Mocha as its test runner with a hybrid architecture combining traditional integration tests and manifest-driven end-to-end tests. A centralized configuration file, .mocharc.js, defines test groups and paths, making the suite flexible and manageable.

End-to-end tests use declarative YAML manifests that define test scenarios, enabling user-extensible workflows and efficient batch processing of test cases.

Tests can be executed in a variety of ways to target specific groups, levels, or specific test files.

• Run all tests

  npm test

• Run a specific group

  npm test -- --group level3

• Run a specific test file by grepping for its test ID

  npm test -- --grep "3.5.1"

• Run a single test file by path (integration tests)

  npm test -- 'test/runners/integration/config/config-resolver.test.js'

• Run a single test file by path (end-to-end tests)

  node test/runners/end-to-end/e2e-runner.js \
       test/runners/end-to-end/cli/config.manifest.yaml \
       # --show   # Show test output 
       # --debug  # Show intermediate steps

The analytics system provides intelligent insights into test performance, brittleness, and historical patterns to optimize development workflows.

• qa-analytics.js analyzes test brittleness and failure patterns

  node test/analytics/qa-analytics.js [--limit 20] [--min-runs 5]

• test-watcher.js for smart test selection based on source changes

  node test/analytics/test-watcher.js

• log-failures-reporter.js for tracking test history
• run-last-fails.js to re-run previously failed tests

  npm run test:last-fails

Test analytics data is stored in ~/.local/share/oshea/test-analytics/test-results.json tracks

• success/failure counts per test
• volatility classification (stable/flaky/unstable)
• temporal tracking (first seen, last run)
• performance metrics (duration, error patterns)

All formation-era test documentation and deprecated tools are preserved in the archive.

• Archive Index: Complete historical documentation
• Legacy QA Tools: Deprecated analysis scripts
• Original Checklists: Markdown-based test checklists
• Test Generation Priority Order: Test roll out implementation strategy
• Next-Generation Testing Framework toward a YAML manifest system