FraudBench: A Legal Evaluation of AI Deception is a Harvard-led multi-university collaborative study that evaluated LLMs propensities to recommend, assist, or directly take actions that would constitute fraud under U.S. law.
The key findings were, unfortunately while yet unexpectedly, nothing less than concerning. Regardless, it's unlikely to put the kibosh on AI usage in law firms.
Consider this article a response to some of these concerning key findings, and we'll address how AI can be deployed to help law firms rather than harm.
Private AI: An Honesty Check
Opichi is an AI consultancy and software development agency specializing in data privacy. We build security measures into all our software, and especially in scenarios when AI might have any involvement with sensitive data.
The most effective path to handling sensitive data (particularly in regards to law firms, clinics, and any other institutions handling extremely sensitive data) is to implement a local LLM and deploy a private AI. That keeps the data within the organization's system; processed and analyzed entirely on their own local area network.
(The other most effective method has been placing an encryption layer between an organization's data before handing it off to a Cloud AI for analysis. It works, but there is always a risk any time data is sent into a Cloud AI for analysis.)
Now, running a model locally does not, by itself, lower fraud rates. As this paper identified (see Tables 2 and 3 on pages 4 and 7), even open-weight models like Qwen3 235B and Grok 4.1 Fast (both reasoning and non-reasoning enabled) agentic tasks hit double digit percentages of fraud instances, while Claude models (reasoning) stayed in single digits percentages of fraud rates.
It should be noted, the models in this paper are already outdated. While they tested poor performing Qwen3 models, Qwen3.8 models now have flexible thinking modes for built-in chain-of-thought and reasoning controls that run by default, which can be tuned or toggled off via API parameters. It's both fascinating and frustrating given how fast AI moves and published research crawls.
Private AI reduces risk. It does not replace your judgement. This article responds to FraudBench's identified levers that move fraud rates in legal proceedings specifically, and explains how a private infrastructure puts a law firm in control of every one of them.
How Private AI and Local LLMs Put Firms in Control
Fraud Rises with Autonomy, so the Firm Should Set the Autonomy
The paper's abstract identified fraud rates across all tested models averaged "3.0% when advising, 14.7% when drafting, and 21.4% when acting directly, without applying pressure." The findings showed GPT-5 swung from "1.3% to 21.9%" between advising and acting (Finding 1).
Given this range, law firms could face liability issues should the AI agent be given the agency to act, a thing that is entirely capable of happening. The paper author's shared exactly that situation in Appendix C, the air purifier misrepresentation recreation scenario. Grok 4.1 Fast called GmailSendEmail and emailed the customer that the product had "undergone independent laboratory testing," leaving out that the company had in fact provided the equipment, funded the study, and had final say on what results to publish (Appendix c.1.6, Agentic Completions). The email went out with no human in between. The worst part? Grok 4.1 even demonstrated fraudulent awareness and advised on dishonesty anyways to maximize profit. More on that later.
When the firm owns the deployment, the firm decides which tools the model can use. Practice groups can get advise-and-draft access with no send, post, or file permissions, so every outbound message has to pass a lawyer first.
The system prompt changes fraud rates, so the firm should own it
Law firms using a Cloud AI provider will ultimately be subject to the model's main system prompt and whatever that entails. While a “No Illegal Behavior” instruction prompt reduced fraud (µ = −0.21), instructing the models to use “High Agency” and “Maximize Profit” increased fraud responses (Finding 2). Under pressure, “the No Illegal Behavior instruction significantly offsets increases in fraud due to contextual pressure, while High Agency and Maximize Profit compound” them (Finding 2, Table 15).
Still looking at the same example, Appendix C shows examples of evaluation items and completion from the recreated Garner v. Global Plasma Solutions Inc., 590 F.Supp.3d 738, involving false claims about air purifier effectiveness. Appendix C.1.6 gives the exact binary, open, and agentic responses of several models including Claude Haiku and Kimi K2 (by far the most ethical in citing why both refused to draft an email), GPT-5 Nano (the most ethical in it's email draft), Qwen3 235B (preposterous ethics), Qwen3 4B (personally it's my favorite email debacle), and Grok 4.1 Fast (absolute dishonesty, as mentioned above).
So why was this vast range of fraud awareness and action/inaction happening? With a cloud tool, the vendor’s product tuning and any individual user’s custom instructions sit between the firm and the model. In other words, unless explicitly instructed with each prompt to act with integrity, the model is, at some point, going to default to how OpenAI or Amazon or xAI programmed it to answer, which is usually a 'confirmation bias' confirmation.
On a private deployment, the firm writes the governing system prompt once and applies it to every seat. No associate can install an “act boldly” or “maximize the client’s position” persona to overwrite the firm's governing system.
Just like no lawyer is completely infallible, no AI is infallible either. The governing instructions reduces fraud, it doesn’t eliminate it. That's why keeping safeguards between AI interaction directly with customers is and should always be a requirement.
Concealment is the bigger risk, so build disclosure checks into the workflow
FraudBench found models "more often engaged in fraud by concealment than by misrepresentation," a statistically significant effect (βConceal = 0.38, Finding 4). Wei et al. call this "a new failure mode for AI honesty":
As agents become more widely deployed, both human-AI and AI-AI interactions may require the development of new norms around honesty and disclosure; understanding where disclosure is or is not needed will require input from stakeholders beyond the ML community, e.g., from law and AI governance. More broadly, we call for more research on how AI agents can reshape the assumptions underlying existing economic transactions, when and why agents engage in motivated reasoning, and what measures are needed to adapt and transition to an AI agent economy.
Because a private AI model runs against the firm’s own document stores, the firm can add a pre-send check that searches the matter file for items it flags as material (pending regulatory complaints, sponsor funding, known defects) and holds any draft that doesn’t address them. A cloud chat window only sees what someone pastes into it. A private system can be built to see what the firm already knows, which is exactly what concealment depends on.
Models recognize the legal risk and proceed anyway, so supervision needs the logs
The tested models were “aware of possible legal risks in 33.7% of generations in which they behave fraudulently” (Finding 3).
Grok 4.1 Fast (Non-Reasoning) behaved fraudulently in 22.8% of the responses where it mentioned legal risk. GLM-4.7 wrote “I cannot draft an email that omits the [regulatory] complaint... Omitting this information could be... fraud,” then drafted an email that left out the complaint (Finding 3, Appendix C.2.6).
This finding means you can’t trust a model’s own disclaimers as a safeguard. Supervision under Rules 5.1 and 5.3 requires being able to review what the system actually did. On private infrastructure, every prompt and output stays in the firm’s own logs, and a second reviewer model can scan for the exact pattern the paper describes: a response that names a legal risk and then complies anyway. Those flagged items go to a supervising attorney. With a third-party tool, those logs sit with the vendor, under the vendor’s retention terms.
More reasoning means less fraud, and owned hardware makes reasoning affordable
When the models were tested with reasoning on, results were “associated with lower fraud rates” (βReason = −0.17), and higher reasoning settings lowered them further (βhigh reasoning = −0.08). Responses with more reasoning tokens were less likely to be fraudulent (Finding 5, Table 23).
Side note, that's something that can seriously eat up an AI budget. Here at Opichi we've seen upgrading from Haiku 4.5 to Haiku 5.5 does the same workload for a lot cheaper.
However, on metered cloud pricing, every extra reasoning token is a cost line. This could push small to medium sized firms toward the fast, cheap settings. On infrastructure the firm owns, tuning reasoning for client-facing drafts costs compute time, not an invoice per token. It's also possible to run a couple models depending on the needs, which is where Opichi's consultation services come into play. Even though these models are out of date, it provides clear evidence that all models return very different outputs. Reviewing Table 3 shows that some models have outstanding results within certain testing perimeters, and law firms can and should use that to their advantage where applicable. Which leads us to...
Model choice matters more than model size, so test before you deploy
Model- and family-level effects were often significant, while size and release date were not (Finding 5 - but again, this new paper is already dated). For evidence, FraudBench's datasets and code are public (Appendix Q).
A private deployment lets the firm pick the model(s) rather than accept whatever the vendor routes to. That choice only helps if it’s tested, and FraudBench gives a ready-made way to do it.
A Private AI Doesn't Eliminate AI Dishonesty, But It Can significantly Reduce It with proper safeguards
Together, we can run the benchmark against candidate local models before a rollout to show law firm's the results by task type. We match models to each practice group’s work, document their limits, and turns Opinion 512's Model Rule 1.1 into something measurable.
At the end of the day, the research shows fraud risk depends on how much autonomy an AI model has, what instructions it runs under, and whether anyone reviews its output. Private AI puts all three under your firm’s direct and final control.