---
title: "Quickstart for Model Validation (R)"
author: "ValidMind"
date: "2026-03-18"
output: html_document
---

# Quickstart for Model Validation

Learn the basics of using ValidMind to validate models as part of a model validation workflow using R. This notebook uses the ValidMind R package (a `reticulate` wrapper around the Python library) to generate a draft of a validation report for a binary classification model.

We will:

1. Import a sample dataset and preprocess it, then initialize datasets for use with ValidMind
2. Independently verify data quality tests performed on datasets by model development
3. Train a champion model for evaluation
4. Run model evaluation tests with the ValidMind Library

## Setting up

The Python path is auto-configured via the `VALIDMIND_PYTHON` environment variable.
If not set, it falls back to the system Python. For local development, create a
`.Renviron` file in the project root with `VALIDMIND_PYTHON=.venv/bin/python`.

```{r setup, include=FALSE}
library(reticulate)

python_version <- Sys.getenv("VALIDMIND_PYTHON", Sys.which("python"))
if (nchar(python_version) > 0 && !startsWith(python_version, "/")) {
  python_version <- file.path(getwd(), python_version)
}
use_python(python_version, required = TRUE)

library(validmind)
library(dplyr)
library(caTools)
library(knitr)

knitr::opts_chunk$set(warning = FALSE, message = FALSE)
```

## Initialize the ValidMind Library

Log in to the [ValidMind Platform](https://app.prod.validmind.ai) and register a model:

1. Navigate to **Inventory** and click **+ Register Model**.
2. Assign yourself as a **Validator** (remove yourself from Owner and Developer roles).
3. Under **Documents > Validation**, select the `Generic Validation Report` template.
4. Go to **Getting Started**, select `Validation` from the **DOCUMENT** drop-down, and copy the code snippet.

Replace the placeholder values below with your own credentials:

```{r}
vm_r <- vm(
  api_host = "https://app.prod.validmind.ai/api/v1/tracking",
  api_key = "<your_api_key_here>",
  api_secret = "<your_api_secret_here>",
  model = "<your_model_id_here>",
  document = "validation-report"
)
```

## Preview the validation report template

Verify the connection and see the validation report structure:

```{r}
py_print(vm_r$preview_template())
```

## Identify available tests

List the tasks and tags available in the ValidMind test library:

```{r}
vm_r$tests$list_tasks_and_tags()
```

List all data quality tests for classification:

```{r}
vm_r$tests$list_tests(tags = list("data_quality"), task = "classification")
```

## Load the demo dataset

We use the Bank Customer Churn dataset for this demonstration:

```{r}
customer_churn <- reticulate::import(
  "validmind.datasets.classification.customer_churn"
)

cat(sprintf(
  paste0(
    "Loaded demo dataset with:\n\n\t- Target column: '%s'",
    "\n\t- Class labels: %s\n"
  ),
  customer_churn$target_column,
  paste(
    names(customer_churn$class_labels),
    customer_churn$class_labels,
    sep = ": ", collapse = ", "
  )
))

data <- customer_churn$load_data()
head(data)
```

## Preprocess the raw dataset

Handle categorical variables using one-hot encoding and remove unnecessary columns:

```{r}
# load_data() already drops RowNumber, CustomerId, Surname
# One-hot encode categorical variables
geo_dummies <- model.matrix(~ Geography - 1, data = data)
gender_dummies <- model.matrix(~ Gender - 1, data = data)
data_processed <- data %>% select(-Geography, -Gender)
data_processed <- cbind(data_processed, geo_dummies, gender_dummies)
```

### Split the dataset

Split into training (60%), validation (20%), and test (20%) sets:

```{r}
set.seed(42)

# First split: 80% train+validation, 20% test
target_col <- customer_churn$target_column
split1 <- sample.split(data_processed[[target_col]], SplitRatio = 0.8)
train_val_data <- subset(data_processed, split1 == TRUE)
test_data <- subset(data_processed, split1 == FALSE)

# Second split: 75% train, 25% validation (of the 80% = 60/20 overall)
split2 <- sample.split(train_val_data[[target_col]], SplitRatio = 0.75)
train_data <- subset(train_val_data, split2 == TRUE)
validation_data <- subset(train_val_data, split2 == FALSE)
```

### Separate features and targets

```{r}
x_train <- train_data %>% select(-all_of(target_col))
y_train <- train_data[[target_col]]
```

## Initialize the ValidMind datasets

```{r}
vm_raw_dataset <- vm_r$init_dataset(
  dataset = data,
  input_id = "raw_dataset",
  target_column = customer_churn$target_column,
  class_labels = customer_churn$class_labels
)

vm_train_ds <- vm_r$init_dataset(
  dataset = train_data,
  input_id = "train_dataset",
  target_column = customer_churn$target_column
)

vm_validation_ds <- vm_r$init_dataset(
  dataset = validation_data,
  input_id = "validation_dataset",
  target_column = customer_churn$target_column
)

vm_test_ds <- vm_r$init_dataset(
  dataset = test_data,
  input_id = "test_dataset",
  target_column = customer_churn$target_column
)
```

## Run data quality tests

### Run an individual data quality test

Run the ClassImbalance test on the raw dataset and log it to the platform:

```{r}
vm_r$tests$run_test(
  test_id = "validmind.data_validation.ClassImbalance",
  inputs = list(dataset = vm_raw_dataset)
)$log()
```

### Run data comparison tests

Compare class imbalance across dataset splits:

```{r}
comparison_tests <- list(
  "validmind.data_validation.ClassImbalance:train_vs_validation" = list(
    input_grid = list(dataset = list("train_dataset", "validation_dataset"))
  ),
  "validmind.data_validation.ClassImbalance:train_vs_test" = list(
    input_grid = list(dataset = list("train_dataset", "test_dataset"))
  )
)

for (test_name in names(comparison_tests)) {
  cat(paste0("Running: ", test_name, "\n"))
  config <- comparison_tests[[test_name]]
  tryCatch({
    vm_r$tests$run_test(
      test_name,
      input_grid = config$input_grid
    )$log()
  }, error = function(e) {
    cat(paste0("Error running test ", test_name, ": ", e$message, "\n"))
  })
}
```

## Train the champion model

Train a logistic regression (GLM) to serve as the champion model:

```{r}
formula <- as.formula(paste(target_col, "~ ."))
model <- glm(formula, data = train_data, family = binomial)
summary(model)
```

## Initialize the model object

Save the R model and initialize it with ValidMind:

```{r}
model_path <- save_model(model)

vm_xgboost <- vm_r$init_r_model(
  model_path = model_path,
  input_id = "xgboost_champion"
)
```

### Assign predictions

Link model predictions to the training and testing datasets:

```{r}
vm_train_ds$assign_predictions(model = vm_xgboost)
vm_test_ds$assign_predictions(model = vm_xgboost)
```

## Run model evaluation tests

### Run model performance tests

List available model performance tests:

```{r}
vm_r$tests$list_tests(tags = list("model_performance"), task = "classification")
```

Run and log performance tests:

```{r}
performance_tests <- c(
  "validmind.model_validation.sklearn.ClassifierPerformance:xgboost_champion",
  "validmind.model_validation.sklearn.ConfusionMatrix:xgboost_champion",
  "validmind.model_validation.sklearn.ROCCurve:xgboost_champion"
)

for (test in performance_tests) {
  cat(paste0("Running: ", test, "\n"))
  vm_r$tests$run_test(
    test,
    inputs = list(dataset = vm_test_ds, model = vm_xgboost)
  )$log()
}
```

### Run diagnostic tests

Assess the model for overfitting:

```{r}
vm_r$tests$run_test(
  test_id = paste0(
    "validmind.model_validation.sklearn.OverfitDiagnosis",
    ":xgboost_champion"
  ),
  input_grid = list(
    datasets = list(list(vm_train_ds, vm_test_ds)),
    model = list(vm_xgboost)
  )
)$log()
```

Test robustness:

```{r}
vm_r$tests$run_test(
  test_id = paste0(
    "validmind.model_validation.sklearn.RobustnessDiagnosis",
    ":xgboost_champion"
  ),
  input_grid = list(
    datasets = list(list(vm_train_ds, vm_test_ds)),
    model = list(vm_xgboost)
  )
)$log()
```

### Run feature importance tests

Note: `PermutationFeatureImportance` and `SHAPGlobalImportance` are not supported for R models.

```{r}
vm_r$tests$run_test(
  "validmind.model_validation.FeaturesAUC:xgboost_champion",
  inputs = list(dataset = vm_test_ds, model = vm_xgboost)
)$log()
```

## Next steps

Head to the [ValidMind Platform](https://app.prod.validmind.ai) to view the validation report:

1. Navigate to **Inventory** and select your model.
2. Click **Validation** under Documents.
3. Include your logged test results as evidence, create risk assessment notes, and assess compliance.

---

*Copyright 2023-2026 ValidMind Inc. All rights reserved.*
*Refer to [LICENSE](https://github.com/validmind/validmind-library/blob/main/LICENSE) for details.*
*SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial*
