All articles
Knowledge infrastructure

MedChemBench: what a domain benchmark can and cannot prove

OpenAI's MedChemBench tests whether models can read molecular structures and experimental evidence. What enterprise teams should check before trusting a vendor-run domain benchmark.

CogLake Redaktion · 21 November 2026 · 5 min

This article was automatically generated with the assistance of AI.

A benchmark is not a product decision. On November 20, 2026, OpenAI published MedChemBench, a 99-problem evaluation of how models connect molecular structures with experimental evidence. The interesting part for enterprise buyers is not the leaderboard but the design: partial credit, withheld results, and a vendor that states plainly what its own benchmark cannot show.

What the benchmark measures

MedChemBench draws on 117 recent papers and patent publications and grades 761 distinct molecular structures. Each task presents the context a researcher would have at a decision point in a drug programme — biological target, goals, compounds studied so far, selected experimental findings — and asks the model to reconstruct structures, explain evidence, rank compounds or predict measured properties.

The seven task categories are unevenly weighted. Integrated lead selection accounts for 48 of the 99 problems and SAR forecasting for 26, while ADME and delivery (8), molecular design and enumeration (6), safety and liabilities (4), retrosynthesis (4) and assay biology and mechanism (3) are smaller slices.

Grading combines deterministic checks of molecular graphs and numerical values with rubric-based assessment of scientific reasoning. Predictions earn partial credit based on how close they come to the measured value, and measurements supplied in the prompt must be reproduced correctly.

The headline numbers

ModelShare of available credit at xhigh
GPT-6.2 Astra84.5%
GPT-6 Astra83.3%
GPT-6.1 Sol82.0%
GPT-5.6 Sol58.2%

Across all eleven evaluated models at the shared xhigh setting, the paper reports a range from 35.02% to 83.33%. The highest mean score, 83.83%, came from GPT-6 Astra at maximum effort.

One inconsistency is worth flagging: the announcement lists 84.5% for GPT-6.2 Astra, while its own overall paragraph gives 83.3% for Astra at xhigh — the figure the paper attributes to GPT-6 Astra. The ranking of the top group is stable; the exact tenth of a point is not. A vendor table that disagrees with itself is a prompt to run your own sample.

Where the scores come apart

The aggregate hides the useful part. On reconstructing molecular structures, GPT-6 Astra earned 95.4% of the available credit at xhigh. On selecting compounds and predicting numerical properties it earned roughly half.

The worked example in the announcement is instructive. Asked to assemble a compound while preserving an 18-atom ring, GPT-6 Astra succeeded in all ten responses, while GPT-5.6 Sol produced a 15-atom ring and GPT-5.6 Luna returned two disconnected fragments. A model that misreads the structure is reasoning about a different molecule — and its explanation will still read plausibly.

What the vendor concedes

OpenAI states that the results do not establish whether models can discover drugs or improve laboratory decisions. Prior exposure to the published sources cannot be ruled out; each response was a single attempt without tools, feedback or new experiments; the setup is retrospective. For a vendor to name these limits is a good sign, and it is also the correct basis for reading the numbers: this measures reasoning over historical evidence, not laboratory performance.

Five questions for procurement

  1. Who built and graded it? Here: the model vendor, on published literature. Ask for an independent replication or run a slice yourself.
  2. Is the test set contaminated? Published papers can sit in training data. Prefer benchmarks with private holdouts.
  3. How is partial credit assigned? "Closeness to the measured value" can reward a plausible range over a correct decision.
  4. What does a point cost? Performance is plotted against output tokens, and higher reasoning effort improved scores. Budget latency and cost per accepted answer, not per call.
  5. Who verifies? If a chemist must re-check every structure, the saving lies in drafting and comparison, not in decisions.

A pilot that costs a week, not a quarter

Take 20 to 30 decisions your own experts made in the past year where the outcome is known. Give the model only the evidence available at the time. Score structure, explanation and selection separately, and time the expert review. That produces a number your chemists and your auditors can both defend — which no leaderboard can.

This article is based on vendor documentation and the published paper; we ran no independent tests. It is not medical, pharmaceutical or investment advice.

Sources

Sources accessed on 21 November 2026.

Turn company knowledge into governed context.

See how CogLake fits your sources, permissions, and AI interfaces.

Book a walkthrough