This project is under heavy active development. We are planning to release a stable version of the framework in the coming weeks.
In the meantime, join our Slack Community
Learn more about our approach and early results:
This project trains Sparse Autoencoders (SAEs) on the internal activations of an AI agent to understand why it selects specific tools. Given a user request like "Search our docs for API limits," the agent must choose between tools (e.g., internal_search vs web_search). We extract the model's hidden representations at the moment of that decision, decompose them into interpretable features using a JumpReLU SAE, and validate the resulting explanations through automated fuzzing and causal ablation experiments.
The key insight: train the SAE on raw activations (not difference vectors), then use contrastive pairs post-hoc to identify which learned features correspond to specific tool-selection decisions. This preserves the SAE's general feature dictionary while enabling targeted analysis of decision-relevant features.
We welcome contributions! Whether you're fixing a bug, improving documentation, or proposing a new feature, your help is appreciated.
- Report Bugs - Open an issue with steps to reproduce
- Improve Docs - Documentation PRs are always welcome
- Submit Features - Open an issue to discuss your idea before submitting a PR
- Share Feedback - Start a discussion
- Slack - Join our community to ask questions and connect with other contributors
- Contributors - See CONTRIBUTORS.md for the list of people who have contributed
Copyright (c) 2026 Dataiku SAS
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.