RAG Evaluation library for Java.

Rage4J provides tools to evaluate and measure the quality of language model outputs using various metrics like correctness, relevance, faithfulness, and semantic similarity. It integrates with LangChain4j and supports fluent test assertions for RAG pipelines.

Modules:

• rage4j - Core evaluation library with evaluators and model classes
• rage4j-assert - Fluent assertion library for RAG evaluation in tests
• rage4j-persist - Persistence module for saving evaluation results (JSONL format)
• rage4j-persist-junit5 - JUnit 5 extension for automatic persistence lifecycle

Add the dependency to your pom.xml:

<dependency>
    <groupId>dev.rage4j</groupId>
    <artifactId>rage4j</artifactId>
    <version>1.1.1-SNAPSHOT</version>
</dependency>

For fluent test assertions:

<dependency>
    <groupId>dev.rage4j</groupId>
    <artifactId>rage-assert</artifactId>
    <version>1.1.1-SNAPSHOT</version>
    <scope>test</scope>
</dependency>

Sample sample = Sample.builder()
    .withQuestion("What is the capital of France?")
    .withAnswer("Paris is the capital of France.")
    .withGroundTruth("Paris")
    .build();

Evaluator evaluator = new AnswerCorrectnessEvaluator(chatModel);
Evaluation result = evaluator.evaluate(sample);
System.out.println(result.getName() + ": " + result.getValue());

RageAssert rageAssert = new OpenAiLLMBuilder().fromApiKey(apiKey);

rageAssert.given()
    .question("What is the capital of France?")
    .groundTruth("Paris")
    .context("Paris is the capital of France.")
    .when()
    .answer("Paris is the capital of France.")
    .then()
    .assertFaithfulness(0.7)
    .then()
    .assertAnswerCorrectness(0.8);

Visit our documentation on Github Pages: Visit Docs

• Java 21
• Maven (wrapper included: ./mvnw)

For development, use the code formatter (./mvnw formatter:format) and install the EditorConfig extension (IntelliJ has built-in support).

richard 💻	Michael Hainz 💻	Vladislav Knyshov 💻	Markus Herhoffer 📆	Aamin Gem 💻	DrBilip 💻	Andreas Dinauer 💻
Klaus-Martin Fink 💻	andresCh01 💻