Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
09a6957
Edits up until Releases
validbeck Oct 10, 2024
c47ceb2
Edits for 2024-aug-13 releases
validbeck Oct 10, 2024
2f0473a
Adjusted figcaption to center text
validbeck Oct 10, 2024
6e45c75
Edits for 2024-jul-22 releases
validbeck Oct 10, 2024
8d17df9
Edits for 2024-jun-10 releases
validbeck Oct 10, 2024
f3d0dec
Edits to overview guides
validbeck Oct 14, 2024
7cb9494
Releases tweaks up to June 10 2024
validbeck Oct 14, 2024
e3f4012
Edits to march 27 2024 wip
validbeck Oct 14, 2024
745cb50
Final tweaks to march 27 2024
validbeck Oct 14, 2024
6002563
Edited feb 14 2024
validbeck Oct 14, 2024
3a2ffb1
Edited jan 26 2024
validbeck Oct 14, 2024
f4d3035
Edited jan 18 2024
validbeck Oct 14, 2024
60ec26e
May 22 2024 releases WIP
validbeck Oct 14, 2024
a1660e6
Edited may 22 2024
validbeck Oct 14, 2024
5a6141b
Added _how-to-upgrade.qmd single-source
validbeck Oct 14, 2024
1e09edb
Added _how-to-upgrade.qmd to newer releases missing it
validbeck Oct 14, 2024
11a223a
Edited nov 09 2023
validbeck Oct 14, 2024
6a0ddfd
Edited oct 25 2023
validbeck Oct 14, 2024
deef0be
WIP Sep 27 2023
validbeck Oct 15, 2024
2babaf0
Edited Sep 27 2023
validbeck Oct 15, 2024
5622182
Tweaks to variable in buttons & links
validbeck Oct 15, 2024
e7763db
Edited aug 15 2023
validbeck Oct 15, 2024
e6e8a41
Edited jul 24 2023
validbeck Oct 15, 2024
43c5d12
Edited jun 22 2023
validbeck Oct 15, 2024
1aee690
Edited may 30 2023
validbeck Oct 15, 2024
86cf3db
Visual tweaks
validbeck Oct 16, 2024
d91e05b
More cleanup
validbeck Oct 16, 2024
cf27de1
Some more tweaks
validbeck Oct 16, 2024
733a21d
Final? edits
validbeck Oct 16, 2024
97aca81
Fixed merge conflicts
validbeck Oct 16, 2024
504b4de
Feedback from Nik
validbeck Oct 17, 2024
60053cd
Maybe the python docs need to be updated on remote?
validbeck Oct 17, 2024
32ee29d
Nope, just a docs-demo preview issue
validbeck Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions site/about/overview-model-documentation.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,16 @@ filters:
- tachyons
aliases:
- ../guide/overview-model-documentation.html
listing:
id: quickstart
type: grid
grid-columns: 2
# image-height: 100%
contents:
- path: ../get-started/quickstart.qmd
title: "QuickStart"
description: "Our QuickStart is the quickest and easiest way to try out our product features."
fields: [title, description, reading-time]
---

<!---
Expand All @@ -15,17 +25,19 @@ Overview & updated graphic
### Closed beta: LLM testing & documentation
--->

The ValidMind Developer Framework streamlines the process of documenting various types of models. The {{< var vm.developer >}} automates the documentation process, ensuring that your model documentation and testing aligns with regulatory and compliance standards.
The {{< var validmind.developer >}} streamlines the process of documenting various types of models. The {{< var vm.developer >}} automates the documentation process, ensuring that your model documentation and testing aligns with regulatory and compliance standards.

::: {.prereq}

## {{< fa code >}} The {{< var validmind.developer >}}

The {{< var validmind.developer >}} is a {{< var vm.developer >}} and documentation engine designed to streamline the process of documenting various types of models, including traditional statistical models, legacy systems, artificial intelligence/machine learning models, and large language models (LLMs). It offers model developers a systematic approach to documenting and testing risk models with repeatability and consistency, ensuring alignment with regulatory and compliance standards.
The {{< var validmind.developer >}} is a {{< var vm.developer >}} and documentation engine designed to streamline the process of documenting various types of models, including traditional statistical models, legacy systems, artificial intelligence/machine learning models, and large language models (LLMs).

It offers model developers a systematic approach to documenting and testing risk models with repeatability and consistency, ensuring alignment with regulatory and compliance standards.

<!-- Using the variable in alt text messes up the image display -->

![](overview-developer-framework.png){width=80% fig-align="center" fig-alt="An image showing the two main components of ValidMind. The developer framework that integrates with your existing developer environment, and the ValidMind AI risk platform."}
![The two main components of ValidMind. The developer framework that integrates with your existing developer environment, and the ValidMind AI risk platform.](overview-developer-framework.png){width=80% fig-align="center" fig-alt="An image showing the two main components of ValidMind. The developer framework that integrates with your existing developer environment, and the ValidMind AI risk platform."}

The {{< var validmind.developer >}} consists of a client-side library, API integration for models and testing, and validation tests that streamline the model development process. Implemented as a series of independent libraries in Python and R, our {{< var vm.developer >}} ensures compatibility and flexibility with diverse sets of developer environments and requirements.

Expand Down Expand Up @@ -117,7 +129,7 @@ ValidMind imports the following artifacts into the documentation via our API end
- Output from test and test suites that have been run.
- Images, plots, visuals that were generated as part of extracting metrics and running tests.

![](fine-print/overview-api-integration.png){fig-alt="A representation of artifacts imported into the documentation via our API"}
![Artifacts imported into the documentation via our SaaS API](fine-print/overview-api-integration.png){width=90% fig-alt="A representation of artifacts imported into the documentation via our SaaS API"}

::: {.callout-important title="ValidMind does NOT:"}
- Send any personal identifiable information (PII) when generating documentation reports.
Expand All @@ -126,7 +138,9 @@ ValidMind imports the following artifacts into the documentation via our API end

## Ready to try out ValidMind?

Our [QuickStart](/get-started/quickstart.qmd) is the quickest and easiest way to try out our product features.
:::{#quickstart}
Comment thread
validbeck marked this conversation as resolved.
:::



<!-- FOOTNOTES -->
Expand Down
66 changes: 46 additions & 20 deletions site/about/overview-model-risk-management.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,16 @@ filters:
- tachyons
aliases:
- ../guide/overview-model-risk-management.html
listing:
id: quickstart
type: grid
grid-columns: 2
# image-height: 100%
contents:
- path: ../get-started/quickstart.qmd
title: "QuickStart"
description: "Our QuickStart is the quickest and easiest way to try out our product features."
fields: [title, description, reading-time]
---

<!---
Expand All @@ -26,17 +36,17 @@ The {{< var validmind.platform >}} provides a comprehensive suite of tools, guid

<!-- Using the variable in alt text messes up the image display -->

![](overview-platform-ui.png){width=80% fig-align="center" fig-alt="An image showing the two main components of ValidMind. The developer framework that integrates with your existing developer environment, and the ValidMind AI risk platform."}
![The two main components of ValidMind. The developer framework that integrates with your existing developer environment, and the ValidMind AI risk platform.](overview-platform-ui.png){width=80% fig-align="center" fig-alt="An image showing the two main components of ValidMind. The developer framework that integrates with your existing developer environment, and the ValidMind AI risk platform."}

The {{< var vm.platform >}} employs a multi-tenant architecture, hosting the cloud-based user interface, APIs, databases, and internal services. The design ensures efficient resource utilization and offers a highly scalable solution for organizations of varying sizes.

With the {{< var vm.platform >}}, you can:

- **Track your model inventory** — Manage the model lifecycle, track the workflow status for models, plan for upcoming validation dates, and more.
- **Work on validation initiatives** — Collaborate with developers and validators to review documentation, add findings, keep track of review statuses, and generate validation reports.
- **Configure workflows** — Set up ValidMind to follow your existing model risk management processes, manage statuses for different parts of the workflow, and get an end-to-end view of workflows and who is involved.
- **Use, create, or edit tests, test suites, and templates** — Create and/or configure required validation tests, test suites, and documentation templates for specific model use cases, tailoring it to your own specific needs.
- **Integrate with your stack** — Import and export model documentation and validation reports.
- **Track your model inventory**[^1] — Manage the model lifecycle, track the workflow status for models, plan for upcoming validation dates, and more.
- **Work on validation initiatives**[^2] — Collaborate with developers and validators to review documentation, add findings, keep track of review statuses, and generate validation reports.
- **Configure workflows**[^3] — Set up ValidMind to follow your existing model risk management processes, manage statuses for different parts of the workflow, and get an end-to-end view of workflows and who is involved.
- **Use, create, or edit tests, test suites, and templates**[^4] — Create and/or configure required validation tests, test suites, and documentation templates for specific model use cases, tailoring it to your own specific needs.
- **Integrate with your stack**[^5] — Import and export model documentation and validation reports.

:::

Expand All @@ -51,18 +61,21 @@ Examples of regulations or policies include:
::: {.w-75-ns}

### SR 11-7: Guidance on Model Risk Management
The Supervisory Guidance on model risk management issued by the Board of Governors of the Federal Reserve System and the Office of the Comptroller of the Currency in the United States in 2011. It provides comprehensive guidance to financial institutions on developing and maintaining a robust model risk management framework, covering aspects like model development, implementation, use, and validation. SR 11-7 is widely recognized and has become a benchmark in the industry for model risk management practices.

The Supervisory Guidance on model risk management issued by the Board of Governors of the Federal Reserve System and the Office of the Comptroller of the Currency in the United States in 2011. It provides comprehensive guidance to financial institutions on developing and maintaining a robust model risk management framework, covering aspects like model development, implementation, use, and validation.

:::

::: {.w-20-ns .content-center}

![](federal-reserve.svg){width=70% fig-alt="Board of Governers Federal Reserve logo"}
![Board of Governors Federal Reserve logo](federal-reserve.svg){width=70% fig-alt="Board of Governors Federal Reserve logo"}

:::

::::

SR 11-7 is widely recognized and has become a benchmark in the industry for model risk management practices.

SR 11-7 outlines these core requirements:

Model Risk Management
Expand All @@ -89,18 +102,22 @@ The regulation also mandates ongoing monitoring and periodic reviews to ensure m

::: {.w-20-ns .content-center}

![](bank-of-england.svg){fig-alt="Bank of England logo"}
![Bank of England logo](bank-of-england.svg){fig-alt="Bank of England logo"}

:::

::: {.w-75-ns}

### SS1/23 – Model Risk Management Principles for Banks
A policy issued by the Prudential Regulation Authority (PRA) in the UK. It encapsulates the final model risk management principles following feedback on the earlier consultation paper CP6/22. The statement provides guidelines for banks in the UK on managing model risk effectively, with particular emphasis on strategic planning and technical capabilities. It outlines principles and amendments, like clarifications on model complexity factors, senior management function responsibilities, and inclusion of dynamic adjustments in model change management, aiming to standardize MRM practices across UK banks and foster the safe adoption of emerging technologies, such as machine learning, artificial intelligence, and large language models (LLMs).

A policy issued by the Prudential Regulation Authority (PRA) in the UK. It encapsulates the final model risk management principles following feedback on the earlier consultation paper CP6/22. The statement provides guidelines for banks in the UK on managing model risk effectively, with particular emphasis on strategic planning and technical capabilities.

:::

::::

It outlines principles and amendments, like clarifications on model complexity factors, senior management function responsibilities, and inclusion of dynamic adjustments in model change management, aiming to standardize MRM practices across UK banks and foster the safe adoption of emerging technologies, such as machine learning, artificial intelligence, and large language models (LLMs).

SS1/23 outlines these core principles:

Model Identification and Model Risk Classification
Expand Down Expand Up @@ -163,7 +180,7 @@ ValidMind, as a robust tool for implementing Model Risk Management (MRM) best pr

:::: {.flex .flex-wrap .justify-around}

::: {.w-50-ns}
::: {.w-40-ns}

First line of defense — model developers
: ValidMind offers a suite of tools for model developers, facilitating thorough documentation and rigorous testing of models, aligning with the regulatory expectations of both SR 11-7 and SS1/23, particularly for models under regulatory purview.
Expand All @@ -177,14 +194,11 @@ Third line of defense — auditors
Model inventory
: The Model Inventory feature encapsulates a centralized repository for all models, aiding in streamlined tracking, management, and monitoring, simplifying compliance with the inventory mandates specified in SR 11-7 and SS1/23.

Lifecycle management and custom workflows
: ValidMind’s capabilities extend to effective model lifecycle management through configurable workflows. This structured approach to managing model risks across various lifecycle stages significantly aids in meeting the rigorous management and oversight expectations set by SR 11-7 and SS1/23.

:::

::: {.w-40-ns}

![](/assets/img/validmind-demo-compact.png){width=90% fig-alt="An image showing the main ValidMind UI main dashboard"}
::: {.w-50-ns}
Lifecycle management and custom workflows
: ValidMind’s capabilities extend to effective model lifecycle management through configurable workflows. This structured approach to managing model risks across various lifecycle stages significantly aids in meeting the rigorous management and oversight expectations set by SR 11-7 and SS1/23.

Model documentation automation
: By automating model documentation through configurable templates and test plans, ValidMind ensures consistent and accurate documentation capture, directly aligning with the documentation standards stipulated in these regulatory guidelines.
Expand All @@ -198,10 +212,22 @@ Communication and tracking

::::



By integrating these features, ValidMind provides a comprehensive platform that not only simplifies the path to compliance with SR 11-7 and SS1/23 but also embeds a culture of rigorous and transparent model risk management within the organization.

## Ready to try out ValidMind?

Our [QuickStart](/get-started/quickstart.qmd) is the quickest and easiest way to try out our product features.
:::{#quickstart}
:::


<!-- FOOTNOTES -->

[^1]: [Working with the model inventory](/guide/model-inventory/working-with-model-inventory.qmd)

[^2]: [Preparing validation reports](/guide/model-validation/preparing-validation-reports.qmd)

[^3]: [Working with model workflows](/guide/model-workflows/working-with-model-workflows.qmd)

[^4]: [Working with documentation templates](/guide/model-documentation/working-with-documentation-templates.qmd)

[^5]: [Export documentation](/guide/model-documentation/export-documentation.qmd)
45 changes: 23 additions & 22 deletions site/about/overview.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -48,39 +48,33 @@ Content & Workflow management to track models and manage risks, governance, and

:::: {.flex .flex-wrap .justify-around}

::: {.w-50-ns}
::: {.w-60-ns}

The ValidMind platform is a suite of tools helping developers, data scientists and risk & compliance stakeholders identify potential risks in their AI and large language models, and generate robust, high-quality model documentation that meets regulatory requirements.
The ValidMind platform is a suite of tools helping developers, data scientists and risk and compliance stakeholders identify potential risks in their AI and large language models, and generate robust, high-quality model documentation that meets regulatory requirements.

The platform is adept at handling many use cases, including models compatible with the Hugging Face Transformers API, and GPT 3.5, GPT 4, and hosted LLama2 and Falcon-based models (focused on text classification and text summarization use cases).

:::

::: {.w-40-ns}

<!-- Using the variable in alt text messes up the image display -->

![](/assets/img/validmind-demo-compact.png){width=85% fig-alt="An image showing the main ValidMind Platform UI main dashboard"}

:::

::::

In addition to LLMs, ValidMind can also handle testing and documentation generation for a wide variety of models, including:

:::: {.flex .flex-wrap .justify-around}

::: {.w-50-ns}

- Traditional machine learning models (ML), such as tree-based models and neural networks
- Natural language processing models (NLP)
- Traditional statistical models, such as OLS regression, logistic regression, time series, etc.
- And many more model types

:::

::: {.w-50-ns}
::: {.w-40-ns}

<!-- Using the variable in alt text messes up the image display -->

::: {.image-container}
![](/assets/img/admin-diagram.png)
![](/assets/img/developer-diagram.png)
![](/assets/img/validator-diagram.png)
![](/assets/img/admin-diagram.png)
![](/assets/img/developer-diagram.png)
:::

- Traditional statistical models, such as OLS regression, logistic regression, time series, etc.
- And many more model types
:::

::::
Expand All @@ -94,8 +88,15 @@ We do all of this while making it easy to align with regulatory guidelines on mo
:::{#validmind-features}
:::

## Ready to try out ValidMind?
::: {.prereq}

## {{< fa hand-point-right >}} Ready to try out ValidMind?

Our [QuickStart](/get-started/quickstart.qmd) is the quickest and easiest way to try out our product features.

:::

<!-- ::: {.callout title="Ready to try out ValidMind?"}
Our [QuickStart](/get-started/quickstart.qmd) is the quickest and easiest way to try out our product features.
::: -->

25 changes: 19 additions & 6 deletions site/get-started/get-started.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,20 @@ title: "Get started"
date: last-modified
aliases:
- ../guide/get-started.html
listing:
id: next-steps
type: grid
grid-columns: 2
# image-height: 100%
sort: none
contents:
- path: quickstart.qmd
title: "QuickStart"
description: "The fastest way to explore what ValidMind can offer is with our QuickStart — Try out the {{< var validmind.developer >}} and explore the {{< var validmind.platform >}}."
- path: platform/next-steps.qmd
title: "Next steps"
description: "If you have already tried the QuickStart, more how-to instructions and links to our FAQs can be found under our next steps."
fields: [title, description, reading-time]
---

ValidMind is a platform designed to streamline the management of risk for AI models, including those used in machine learning (ML), natural language processing (NLP), and large language models (LLMs). ValidMind offers tools that cater to both model developers and validators, simplifying key aspects of model risk management.
Expand Down Expand Up @@ -76,19 +90,18 @@ In production
Periodic review and revalidation
: As part of regular performance monitoring or change management, you follow a process similar to that seen in the **Initial validation** step.

## Next steps
::: {.prereq}

The fastest way to explore what ValidMind can offer is with our [QuickStart](quickstart.qmd) which walks you through:
## {{< fa caret-right >}} What's next?

- Trying out the {{< var validmind.developer >}}
- Exploring the {{< var validmind.platform >}}

If you have already tried the QuickStart, more how-to instructions and links to our FAQs can be found under [Next steps](platform/next-steps.qmd).
:::{#next-steps}
:::

::: {.callout title="Need an account for ValidMind?"}
Signing up is FREE — {{< var link.register >}}
:::

:::


<!-- FOOTNOTES -->
Expand Down
6 changes: 5 additions & 1 deletion site/guide/configuration/manage-your-profile.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,9 @@ Personalize your user experience and ensure you have access to the necessary too
- Access keys

#### Onboarding
The {{< var vm.platform >}} provides an interactive onboarding journey right in the interface. You can toggle the onboarding wizard on or off at any time, as well as restart the onboarding tasks after you've completed them:
The {{< var vm.platform >}} provides an interactive onboarding journey right in the interface.

You can toggle the onboarding wizard on or off at any time, as well as restart the onboarding tasks after you've completed them:

Getting started guide
: - Toggle [**getting started guide**]{.smallcaps} on to display the interactive onboarding guide.
Expand All @@ -47,6 +49,8 @@ Under User Interface Preferences, you can switch between the dark and light mode
- Toggle [**enable dark mode**]{.smallcaps} on to use dark mode.
- Toggle [**enable dark mode**]{.smallcaps} off to use light mode.

The selected theme setting is saved to your user profile and applied across all devices where you log in.

#### Access keys

View your API and secret keys under Access Keys:
Expand Down
Loading