1 ValidMind Library Python API – ValidMind

ValidMind Library Python API

The ValidMind Library is a suite of developer tools and methods designed to automate the documentation and validation of your models.

Designed to be model agnostic, the ValidMind Library provides all the standard functionality without requiring you to rewrite any functions as long as your model is built in Python.

With a rich array of documentation tools and test suites, from documenting descriptions of your datasets to testing your models for weak spots and overfit areas, the ValidMind Library helps you automate model documentation by feeding the ValidMind Platform with documentation artifacts and test results.

To install the ValidMind Library:

pip install validmind

To initialize the ValidMind Library, paste the code snippet with the model identifier credentials directly into your development source code, replacing this example with your own:

import validmind as vm

vm.init(
  api_host = "https://app.prod.validmind.ai/api/v1/tracking/tracking",
  api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  api_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  model = "<model-identifier>"
)

After you have pasted the code snippet into your development source code and executed the code, the Python Library API will register with ValidMind. You can now use the ValidMind Library to document and test your models, and to upload to the ValidMind Platform.

__version__

2.12.5

init

definit(api_key:Optional[str]=None,api_secret:Optional[str]=None,api_host:Optional[str]=None,model:Optional[str]=None,monitoring:bool=False,generate_descriptions:Optional[bool]=None,document:Optional[str]=None):

Initializes the API client instances and calls the /ping endpoint to ensure the provided credentials are valid and we can connect to the ValidMind API.

If the API key and secret are not provided, the client will attempt to retrieve them from the environment variables VM_API_KEY and VM_API_SECRET.

Arguments

  • model (str, optional): The model CUID. Defaults to None.
  • api_key (str, optional): The API key. Defaults to None.
  • api_secret (str, optional): The API secret. Defaults to None.
  • api_host (str, optional): The API host. Defaults to None.
  • monitoring (bool, optional): The ongoing monitoring flag. Defaults to False.
  • generate_descriptions (bool, optional): Whether to use GenAI to generate test result descriptions. Defaults to True.
  • document (str): The name of the document. Omitting this argument is deprecated.

Raises

  • ValueError: If the API key and secret are not provided

init_dataset

definit_dataset(dataset:Union[pd.DataFrame, pl.DataFrame, np.ndarray, torch.utils.data.tensordataset],model:Optional[validmind.vm_models.VMModel]=None,index:Optional[Any]=None,index_name:Optional[str]=None,date_time_index:bool=False,columns:Optional[List[str]]=None,text_column:Optional[str]=None,target_column:Optional[str]=None,feature_columns:Optional[List[str]]=None,extra_columns:Optional[Dict[str, Any]]=None,class_labels:Optional[Dict[str, Any]]=None,type:Optional[str]=None,input_id:Optional[str]=None,copy_data:bool=True,__log:bool=True) → validmind.vm_models.VMDataset:

Initializes a VM Dataset, which can then be passed to other functions that can perform additional analysis and tests on the data. This function also ensures we are reading a valid dataset type.

The following dataset types are supported:

  • Pandas DataFrame
  • Polars DataFrame
  • Numpy ndarray
  • Torch TensorDataset

Arguments

  • dataset: Dataset from various Python libraries.
  • model (VMModel): ValidMind model object.
  • index (Any): Index for the dataset.
  • index_name (str): Name of the index column.
  • date_time_index (bool): Whether the index is a datetime index.
  • columns (List[str]): List of column names.
  • text_column (str): Name of the text column.
  • target_column (str): The name of the target column in the dataset.
  • feature_columns (List[str]): A list of names of feature columns in the dataset.
  • extra_columns (Dict[str, Any]): A dictionary containing the names of the prediction_column and group_by_columns in the dataset.
  • class_labels (Dict[str, Any]): A list of class labels for classification problems.
  • type (str): The type of dataset (one of DATASET_TYPES) - DEPRECATED.
  • input_id (str): The input ID for the dataset (e.g. "my_dataset"). By default, this will be set to dataset but if you are passing this dataset as a test input using some other key than dataset, then you should set this to the same key.
  • copy_data (bool, optional): Whether to copy the data. Defaults to True.
  • __log (bool, optional): Whether to log the input. Defaults to True.

Returns

  • A VM Dataset instance.

Raises

  • ValueError: If the dataset type is not supported.

init_model

definit_model(model:Optional[object]=None,input_id:str='model',attributes:Optional[Dict[str, Any]]=None,predict_fn:Optional[Callable]=None,__log:bool=True,**kwargs:Any) → validmind.vm_models.VMModel:

Initializes a VM Model, which can then be passed to other functions that can perform additional analysis and tests on the data. This function also ensures we are creating a model supported libraries.

Arguments

  • model: A trained model or VMModel instance.
  • input_id (str): The input ID for the model (e.g. "my_model"). By default, this will be set to model but if you are passing this model as a test input using some other key than model, then you should set this to the same key.
  • attributes (dict): A dictionary of model attributes.
  • predict_fn (callable): A function that takes an input and returns a prediction.
  • **kwargs: Additional arguments to pass to the model.

Returns

  • A VM Model instance.

Raises

  • ValueError: If the model type is not supported.

init_r_model

definit_r_model(model_path:str,input_id:str='model') → validmind.vm_models.VMModel:

Initialize a VM Model from an R model.

The model must first be saved to an .RData file using the R package's save_model() function. This function then uses rpy2 to load the model into Python for testing and validation.

Arguments

  • model_path (str): The path to the R model saved as an .RData file.
  • input_id (str, optional): The input ID for the model. Defaults to "model".

Returns

  • A VM Model instance.

get_test_suite

defget_test_suite(test_suite_id:Optional[str]=None,section:Optional[str]=None,*args:Any,**kwargs:Any) → validmind.vm_models.TestSuite:

Gets a TestSuite object for the current project or a specific test suite.

This function provides an interface to retrieve the TestSuite instance for the current project or a specific TestSuite instance identified by test_suite_id. The project Test Suite will contain sections for every section in the project's documentation template and these Test Suite Sections will contain all the tests associated with that template section.

Arguments

  • test_suite_id (str, optional): The test suite name. If not passed, then the project's test suite will be returned. Defaults to None.
  • section (str, optional): The section of the documentation template from which to retrieve the test suite. This only applies if test_suite_id is None. Defaults to None.
  • args: Additional arguments to pass to the TestSuite.
  • kwargs: Additional keyword arguments to pass to the TestSuite.

Returns

  • The TestSuite instance.

log_metric

deflog_metric(key:str,value:Union[int, float],inputs:Optional[List[str]]=None,params:Optional[Dict[str, Any]]=None,recorded_at:Optional[str]=None,thresholds:Optional[Dict[str, Any]]=None,passed:Optional[bool]=None):

Logs a unit metric.

Unit metrics are key-value pairs where the key is the metric name and the value is a scalar (int or float). These key-value pairs are associated with the currently selected model (inventory model in the ValidMind Platform) and keys can be logged to over time to create a history of the metric. On the ValidMind Platform, these metrics will be used to create plots/visualizations for documentation and dashboards etc.

Arguments

  • key (str): The metric key
  • value (Union[int, float]): The metric value (scalar)
  • inputs (List[str]): List of input IDs
  • params (Dict[str, Any]): Parameters used to generate the metric
  • recorded_at (str): Timestamp when the metric was recorded
  • thresholds (Dict[str, Any]): Thresholds for the metric
  • passed (bool): Whether the metric passed validation thresholds

preview_template

defpreview_template():

Preview the documentation template for the current project.

This function will display the documentation template for the current project. If the project has not been initialized, then an error will be raised.

Raises

  • ValueError: If the project has not been initialized.

reload

defreload():

Reconnect to the ValidMind API and reload the project configuration.

run_documentation_tests

defrun_documentation_tests(section:Optional[str]=None,send:bool=True,fail_fast:bool=False,inputs:Optional[Dict[str, Any]]=None,config:Optional[Dict[str, Any]]=None,**kwargs:Any) → Union[validmind.vm_models.TestSuite, Dict[str, validmind.vm_models.TestSuite]]:

Collect and run all the tests associated with a template.

This function will analyze the current project's documentation template and collect all the tests associated with it into a test suite. It will then run the test suite, log the results to the ValidMind API, and display them to the user.

Arguments

  • section (str or list, optional): The section(s) to preview. Defaults to None.
  • send (bool, optional): Whether to send the results to the ValidMind API. Defaults to True.
  • fail_fast (bool, optional): Whether to stop running tests after the first failure. Defaults to False.
  • inputs (dict): A dictionary of test inputs to pass to the TestSuite.
  • config: A dictionary of test parameters to override the defaults.
  • **kwargs: backwards compatibility for passing in test inputs using keyword arguments.

Returns

  • TestSuite or dict: The completed TestSuite instance or a dictionary of TestSuites if section is a list.

Raises

  • ValueError: If the project has not been initialized.

run_test_suite

defrun_test_suite(test_suite_id:str,send:bool=True,fail_fast:bool=False,config:Optional[Dict[str, Any]]=None,inputs:Optional[Dict[str, Any]]=None,**kwargs:Any) → validmind.vm_models.TestSuite:

High Level function for running a test suite.

This function provides a high level interface for running a test suite. A test suite is a collection of tests. This function will automatically find the correct test suite class based on the test_suite_id, initialize each of the tests, and run them.

Arguments

  • test_suite_id (str): The test suite name. For example, 'classifier_full_suite'.
  • config (dict, optional): A dictionary of parameters to pass to the tests in the test suite. Defaults to None.
  • send (bool, optional): Whether to post the test results to the API. send=False is useful for testing. Defaults to True.
  • fail_fast (bool, optional): Whether to stop running tests after the first failure. Defaults to False.
  • inputs (dict, optional): A dictionary of test inputs to pass to the TestSuite, such as model, dataset models, etc. These inputs will be accessible by any test in the test suite. See the test documentation or vm.describe_test() for more details on the inputs required for each. Defaults to None.
  • **kwargs: backwards compatibility for passing in test inputs using keyword arguments.

Returns

  • The TestSuite instance.

Raises

  • ValueError: If the test suite name is not found or if there is an error initializing the test suite.

tags

deftags(*tags:str):

Decorator for specifying tags for a test.

Arguments

  • *tags: The tags to apply to the test.

tasks

deftasks(*tasks:str):

Decorator for specifying the task types that a test is designed for.

Arguments

  • *tasks: The task types that the test is designed for.

test

deftest(func_or_id:Union[Callable[..., Any], str, None]):

Decorator for creating and registering custom tests

This decorator registers the function it wraps as a test function within ValidMind under the provided ID. Once decorated, the function can be run using the validmind.tests.run_test function.

The function can take two different types of arguments:

  • Inputs: ValidMind model or dataset (or list of models/datasets). These arguments must use the following names: model, models, dataset, datasets.
  • Parameters: Any additional keyword arguments of any type (must have a default value) that can have any name.

The function should return one of the following types:

  • Table: Either a list of dictionaries or a pandas DataFrame
  • Plot: Either a matplotlib figure or a plotly figure
  • Scalar: A single number (int or float)
  • Boolean: A single boolean value indicating whether the test passed or failed

The function may also include a docstring. This docstring will be used and logged as the metric's description.

Arguments

  • func_or_id (Union[Callable[..., Any], str, None]): Either the function to decorate or the test ID. If None, the function name is used.

Returns

  • The decorated function.

scorer_decorator

defscorer(func_or_id:Union[Callable[..., Any], str, None]=None) → Callable[[validmind.vm_models.F], validmind.vm_models.F]:

Decorator for creating and registering custom scorers

This decorator registers the function it wraps as a scorer function within ValidMind under the provided ID. Once decorated, the function can be run using the validmind.scorers.run_scorer function.

The scorer ID can be provided in three ways:

  1. Explicit ID: @scorer("validmind.scorers.classification.BrierScore")
  2. Auto-generated from path: @scorer() - automatically generates ID from file path
  3. Function name only: @scorer - uses function name with validmind.scorers prefix

The function can take two different types of arguments:

  • Inputs: ValidMind model or dataset (or list of models/datasets). These arguments must use the following names: model, models, dataset, datasets.
  • Parameters: Any additional keyword arguments of any type (must have a default value) that can have any name.

The function should return one of the following types:

  • Table: Either a list of dictionaries or a pandas DataFrame
  • Plot: Either a matplotlib figure or a plotly figure
  • Scalar: A single number (int or float)
  • Boolean: A single boolean value indicating whether the test passed or failed
  • List: A list of values (for row-level metrics) or a list of dictionaries with consistent keys
  • Any other type: The output will be stored as raw data for use by calling code

When returning a list of dictionaries:

  • All dictionaries must have the same keys
  • The list length must match the number of rows in the dataset
  • Each dictionary key will become a separate column when using assign_scores
  • Column naming follows the pattern: {model_id}{metric_name}{dict_key}

Note: Scorer outputs are not logged to the backend and are intended for use by other parts of the system (e.g., assign_scores method).

The function may also include a docstring. This docstring will be used and logged as the scorer's description.

Arguments

  • func_or_id (Union[Callable[..., Any], str, None], optional): Either the function to decorate or the scorer ID. If None or empty string, the ID is auto-generated from the file path. Defaults to None.

Returns

  • The decorated function.

log_text

deflog_text(content_id:str,text:str,_json:Optional[Dict[str, Any]]=None) → str:

Logs free-form text to ValidMind API.

Arguments

  • content_id (str): Unique content identifier for the text.
  • text (str): The text to log. Will be converted to HTML with MathML support.
  • _json (dict, optional): Additional metadata to associate with the text. Defaults to None.

Returns

  • HTML string containing the logged text in an accordion format.

Raises

  • ValueError: If content_id or text are empty or not strings.
  • Exception: If the API call fails.

log_test_result

deflog_test_result(result:Dict[str, Any],section_id:str=None,position:int=None,unsafe:bool=False,config:Dict[str, bool]=None) → Dict[str, Any]:

Logs test results information.

Arguments

  • result (dict): A dictionary representing the test result.
  • section_id (str): The section ID add a test driven block to the documentation.
  • position (int): The position in the section to add the test driven block.
  • unsafe (bool): If True, log the result even if it contains sensitive data.
  • config (Dict[str, bool]): Configuration options for displaying the test result.

Returns

  • The response from the API.

experimental_agent

Agent interface for all text generation tasks

RawData

classRawData:

Holds raw data for a test result.

RawData

RawData(log:bool=False,**kwargs:Any)

Create a new RawData object.

Arguments

  • log (bool): If True, log the raw data to ValidMind.
  • **kwargs: Keyword arguments to set as attributes, such as RawData(log=True, dataset_duplicates=df_duplicates).

inspect

definspect(self,show:bool=True) → Optional[Dict[str, Any]]:

Inspect the raw data.

Arguments

  • show (bool): If True, print the raw data. If False, return it.

Returns

  • If True, print the raw data and return None. If False, return the raw data dictionary.

serialize

defserialize(self) → Dict[str, Any]:

Serialize the raw data to a dictionary

Returns

  • The serialized raw data