Benchmarks

Tests whether AI-generated medical recommendations are safe for patients. Built from real clinical consultations across many medical specialties, each AI response is reviewed by board-certified specialists for potential harm. This is the core safety benchmark of the MAST suite.

Evaluates AI diagnostic reasoning on complex clinical cases drawn from a century of New England Journal of Medicine Clinicopathological Conferences. The benchmark spans multiple tasks including differential diagnosis, next-test selection, literature search, and image interpretation.

Tests AI agents on 100 long-horizon physician tasks adapted from real consultations across 21 specialties. Agents work in an EHR environment with real patient records over standard FHIR APIs, scored against 670 execution-verified checkpoints.

Challenging cases measuring probabilistic clinical reasoning under uncertainty. The benchmark uses a Script Concordance Testing methodology, a format that assesses how clinicians adjust their diagnostic or therapeutic judgments in response to new, uncertain information.

Tests AI agents on multi-step tasks in a simulated hospital electronic health record. Tasks require agents to search patient data, follow conditional logic, and take actions such as ordering medications or scheduling labs.

Evaluates how accurately AI models interpret chest X-ray images and generate clinically appropriate findings, testing both radiology report generation and visual question answering. ReXrank Mini is a curated subset of the full ReXrank benchmark, built on a multi-site chest radiograph dataset from the Harvard Rajpurkar Lab.

Tests multimodal AI models on skin-lesion assessment, combining the Diverse Dermatology Images (DDI) and MRA-MIDAS datasets of clinical and dermoscopic images with histopathologic confirmation.