Interpretability & model internals tooling
Interpretability & model internals tooling covers libraries and platforms for inspecting and steering LLM internals: activation hooks, sparse autoencoders, circuit analysis, feature steering.
In brief
Interpretability & model internals tooling, in brief.
What it isInterpretability & model internals tooling covers libraries and platforms for inspecting and steering LLM internals: activation hooks, sparse autoencoders, circuit analysis, feature steering.
Where it sitsOne of 16 categories in the Safety and trust layer. Safety and trust covers the tools that check, test and control AI systems: evaluations, guardrails, security, red-teaming, monitoring, and the platforms that handle governance and compliance.
What we track38 products, including activation-steering (conditional steering), Activault (activation storage for interpretability), Automatic Circuit Discovery (ACDC), AxBench (benchmark for steering/interpretability methods) and BertViz (attention visualization).
The products
38 products in Interpretability & model internals tooling.
Each name opens a search for that product in our index.
ProductMakerIts own site
NDIF - National Deep Inference Fabric (remote access to large open models' internals)NSF / Northeastern Universityndif.us
From our census of 2026-11-08. A product listed here is one that belongs in this category; the search answer says whether we have checked it.
Questions
Interpretability & model internals tooling, answered.
What is Interpretability & model internals tooling in AI infrastructure?Interpretability & model internals tooling covers libraries and platforms for inspecting and steering LLM internals: activation hooks, sparse autoencoders, circuit analysis, feature steering.
Which layer of AI infrastructure is Interpretability & model internals tooling in?Interpretability & model internals tooling is in the Safety and trust layer. Safety and trust covers the tools that check, test and control AI systems: evaluations, guardrails, security, red-teaming, monitoring, and the platforms that handle governance and compliance.
Which products are in Interpretability & model internals tooling?We track 38 products in Interpretability & model internals tooling, including activation-steering (conditional steering), Activault (activation storage for interpretability), Automatic Circuit Discovery (ACDC), AxBench (benchmark for steering/interpretability methods), BertViz (attention visualization), Captum, circuit-tracer (attribution graphs) and CircuitsVis. The full list is on this page.
Safety and trust
The other categories in Safety and trust.
AI agent management platformsAI governance & compliance platformsAI incident & risk databasesAI securityAI-generated content & deepfake detectionBias, fairness & explainabilityConfidential AI & private inferenceContent provenanceEvalsGuardrailsLLM & agent observabilityModeration & guard classifiersPrompt managementRed-teamingVoice & conversational agent simulation testing