AI Models Struggled With Constitutional Coding Tasks
Researchers found that ChatGPT-5 and Grok-3 inconsistently matched human accuracy in comparative legal coding as of fall 2025.
Updated on Oct. 14, 2026 in Artificial Intelligence
Live Poll
Do you trust artificial intelligence to handle complex professional tasks as accurately as human experts?
A study published in October 2026 evaluated how effectively top-tier AI models could replicate human-coded constitutional law datasets. Researchers tested ChatGPT-5 and Grok-3 on seven variables across four datasets, revealing inconsistent performance compared to established human standards.
Why it matters
The research aimed to determine if artificial intelligence could automate the labor-intensive process of legal data collection. Identifying the limitations of these models helps legal scholars understand the current risks of using AI for analytical research.
Models were tested against 106 presidential systems and 43 first-generation constitutional rights. ChatGPT-5 operated at a 1.0 temperature setting, while Grok-3 used a 0.7 temperature setting for prompt responses.
The players
ChatGPT-5
This is a large language model developed by OpenAI that researchers used to test automated comparative legal coding capabilities.
Grok-3
This is a large language model developed by xAI that was benchmarked alongside ChatGPT-5 in the comparative constitutional law study.
The details
The models were tested without direct access to constitutional texts, relying instead on internal training data to code seven variables across four comparative law datasets. Analysis showed a tendency for the models to produce false positives and a bias toward selecting the most common category rather than the accurate one.
Timeline
The Comparative Constitution Making Dataset covers the period from 1900 to 2015.
The Executive Term Limit Evasion Dataset tracks data from 2000 to 2019.
Models evaluated were considered state-of-the-art in fall 2025.
Findings from the research were published in October 2026.
The Tech Race
This research highlights the shift from using AI for basic language generation to complex specialized academic tasks. It positions current AI as a tool that still requires human oversight when dealing with the high-precision requirements of legal record-keeping.
Legal professionals and researchers should exercise caution, as these models exhibit biases that may lead to errors in data-intensive tasks. Users relying on AI for complex analytical summaries should verify outputs against primary source documentation to ensure accuracy.
The takeaway
Artificial intelligence shows significant promise in legal research but currently lacks the reliability required for independent comparative law analysis. Scholars should prioritize human-verified datasets while using AI primarily for preliminary drafting or organizational assistance.
Further reading
For more information on the current capabilities and limitations of machine learning, visit our Artificial Intelligence section.
Source note: This article includes information reported by Verfassungsblog.
Live Poll
Do you trust artificial intelligence to handle complex professional tasks as accurately as human experts?
