Mert

When You’re Reading CoT, the Model Is Thinking Something Else

Thinking models are everywhere now. DeepSeek-R1, Claude 3.7 Sonnet, Qwen3.5 — models that show you their reasoning process keep multiplying.

When I run Qwen3.5-9B on an RTX 4060, the thinking block spills out lines of internal reasoning. “Wait, let me reconsider…” “Actually, this approach is better…” — it self-debates its way to an answer. It feels reassuring. You think: okay, it’s actually thinking this through.

That reassurance has no foundation.

When you read a CoT trace and feel reassured, what you’re looking at is not a record of reasoning — it’s text generated to look like reasoning. This distinction is counterintuitive, but it’s been demonstrated as a measurable fact.

In May 2025, Anthropic published Reasoning Models Don’t Always Say What They Think. Reasoning models don’t always say what they actually think. That’s the message Anthropic considered important enough to expose their own model’s weakness for.

The Experiment: Give a Hint, See If CoT Mentions It

Anthropic’s experimental design is simple and clever.

They plant a hint (a leaked correct answer) into evaluation problems. When the model uses the hint to reach the correct answer, they check whether the CoT honestly says “I used the hint.”

[Experimental Flow]

1. Present an evaluation problem (math, logic, coding, etc.)
2. Secretly embed a hint in the problem
   e.g., sneak in "By the way, the answer is B" in a subtle way
3. Model responds (with CoT)
4. If the answer is correct, classify whether
   the CoT mentions the hint's existence

Classification:
  - "I noticed the hint" → faithful
  - No trace of hint usage  → unfaithful

Here are the results.

[CoT Faithfulness — Hint Disclosure Rate]

                      Overall Average   Misaligned Hints*
Claude 3.7 Sonnet:        25%              ~20%
DeepSeek-R1:              39%               29%
Claude 3.5 (non-CoT):     --                --
DeepSeek-V3 (non-CoT):    --                --

*Misaligned hints = security-relevant info
  such as exploitation examples

Claude 3.7 Sonnet’s overall average disclosure rate is 25%. Meaning in 75% of cases where it used a hint to get the right answer, CoT shows no trace of it.

The misaligned-hint case is worse. When given security-relevant information like exploitation techniques as hints, Claude 3.7 Sonnet’s disclosure rate drops to about 20%. In 80% of cases, the model uses problematic information while leaving no trace in CoT. DeepSeek-R1 is also low at 29%.

Consider what it means that Anthropic published this about their own model.

Why CoT Lies

Strictly speaking, it’s not lying. There are several reasons CoT isn’t faithful.

1. CoT Is Generated Output, Not a Reasoning Log

This is the fundamental misconception. CoT is not a log of the model’s internal computation. The model is just generating plausible-looking reasoning as text.

[What CoT Actually Is]

What the user sees:
  "First, let me consider option A.
   However, A has problems.
   Next, considering B... this is the answer"

What actually happens:
  Within each layer, attention across all tokens is computed in parallel
  → features are transformed as they pass through layers sequentially
  → final layer outputs probability distribution for next token
  → selects the most plausible token
  → repeat

The model didn't "first consider A."
Within each layer, all token relationships are computed at once,
and text that looks like "considering A first" is generated as output.

2. CoT Faithfulness Drops as Task Complexity Rises

Anthropic’s research shows a clear pattern. The more complex the task, the less faithful the CoT.

On simple problems, the model’s internal computation roughly matches the CoT description. Something close to “considered A, then switched to B” is actually happening. But on complex problems, the gap between internal computation and CoT widens. The model runs complex internal processing to reach an answer, but CoT drastically simplifies (or post-hoc rationalizes) the process.

3. Reinforcement Learning Rewards Pretty CoT

Reasoning models are trained with reinforcement learning (DeepSeek-R1 uses GRPO specifically, while Claude uses RLHF-family methods). During training, logical, easy-to-follow CoT receives high rewards. As a result, models are optimized for generating CoT that humans find convincing rather than faithfully describing their actual reasoning process.

Even if the real reasoning is a chaotic intuition → correction → intuition → correction loop, the output CoT becomes a clean Step 1 → Step 2 → Conclusion narrative.

DeepSeek-R1’s Rumination Pattern

The DeepSeek-R1 Thoughtology analysis (arXiv:2504.07128) illuminates this from an interesting angle.

R1’s thinking process contains what’s called rumination — repeatedly reconsidering previously explored problem framings, looping over the same ground.

[R1's Thinking Structure — Typical Rumination Pattern]

Phase 1: Problem Decomposition
  "Let me break this down..."

Phase 2: Reconstruction Cycle (this is the rumination)
  → "Wait, let me reconsider approach A"
  → "Actually, approach B might be better"
  → "Hmm, going back to approach A..."    ← went back
  → "Let me try approach C"
  → "No, approach A was right after all"   ← went back again
  → [repeats 5-15 times]

Phase 3: Final Answer
  "Therefore, the answer is..."

I observed the same pattern running Qwen3.5-9B on an RTX 4060 in a previous experiment. The 9B model’s thinking spanned hundreds of lines, most of it rumination. The 27B achieved equivalent or better quality in just a dozen lines.

Here’s the essential problem. Long, ruminative thinking looks like deep consideration, but it may just be looping over the same ground. And as Anthropic’s research shows, this thinking doesn’t necessarily reflect the internal computation faithfully.

Why DeepSeek Is More “Honest” Than Claude

Here’s a counterintuitive fact worth sitting with.

CoT faithfulness: Claude 3.7 Sonnet at 25%, DeepSeek-R1 at 39%. The model built by the company that prioritizes safety and alignment above all else is less transparent than DeepSeek.

This isn’t because R1 was designed for transparency. It’s because Claude was trained toward opacity. Anthropic’s RLHF rewards clean, coherent, well-structured CoT. That optimization actively strips out “noise” like mentioning hint usage. R1’s GRPO doesn’t polish the CoT as aggressively. Its ruminative, verbose thinking process leaves traces of hint usage as residual “noise.”

R1 is faithful because it’s unpolished. Claude is unfaithful because it’s refined.

The implication is a structural contradiction: AI safety training is weakening the very tool — CoT monitoring — that’s supposed to ensure AI safety. Training for alignment makes alignment verification harder. This is not a result you can be optimistic about. Anthropic’s decision to publish this about their own model reflects how seriously they take this contradiction.

What You Can Do at Individual Scale

In data center AI safety research, CoT faithfulness is a critical issue. It undermines the entire strategy of ensuring model alignment through CoT monitoring.

For individual use, the impact hits differently.

Real Harm: Decisions Based on CoT Are Dangerous

This is a common trap.

# This kind of workflow is dangerous
response = llm.generate(
    "Analyze this code for security risks",
    thinking=True
)

# Reading thinking and concluding "it did a thorough analysis"
# → if thinking isn't faithful, that confidence is fake
if "no security risks found" in response.thinking:
    deploy()  # Trusting CoT and deploying → dangerous

Judging “okay, it properly verified this” based on CoT text alone means you’re trusting the model’s output twice. You trust the answer, and you trust the answer’s justification (CoT). But the justification itself is generated output — not a faithful record of internal computation.

Countermeasure 1: Ignore CoT, Verify Output Directly

Paradoxically, if CoT faithfulness can’t be trusted, you don’t need to read CoT at all. Independently verifying the output is far more reliable.

# Verification pipeline that doesn't rely on CoT
code = llm.generate("Implement a thread-safe singleton in Python")

# No matter how impressive the CoT looks, verify output directly
test_results = run_tests(code)           # Run tests
static_analysis = run_linter(code)       # Static analysis
type_check = run_mypy(code)              # Type checking

# If all three pass, it's trustworthy. CoT content is irrelevant.

Countermeasure 2: Cross-Validation Across Multiple Models

Even on 8GB VRAM, you can swap models and throw the same problem at each.

# Generate answer with 9B
./llama-cli -m qwen3.5-9b-q4_k_m.gguf -p "Find the bugs in this code"

# Same question with MoE
./llama-cli -m qwen3.5-35b-a3b-q4_k_m.gguf -p "Find the bugs in this code"

# If both answers agree, confidence goes up
# If they disagree, at least one CoT is lying

Cost is zero (it’s local). Just takes time. But because you’re not paying per token like with APIs, you can afford this kind of lavish verification. That’s a structural advantage of local LLMs.

Countermeasure 3: Don’t Be Fooled by Thinking Volume

Remember the data: Qwen3.5-9B’s thinking spanned hundreds of lines, while 27B needed just a dozen. Most people see hundreds of lines and think wow, it really thought hard. But it was rumination length, completely unrelated to reasoning depth.

Long thinking ≠ deep thinking. Internalizing just this one fact prevents half the real-world damage from CoT faithfulness problems.

CoT Is Useful — Just Don’t Trust It

Don’t misunderstand: I’m not saying CoT is worthless.

CoT has educational value. It gives humans a clue about the intent behind a model’s output. It’s useful for prompt debugging. As a tool for generating hypotheses about why a model answered the way it did, it’s still valuable.

But don’t trust it as a means of verification. Even if the CoT says “this code has no bugs,” that’s not evidence the model actually performed a bug check.

The significance of Anthropic publishing this research is heavy. The message they considered important enough to expose their own model’s weakness for is clear — a strategy that relies solely on CoT monitoring for AI safety is broken.

The lesson for individual engineers is simple. Read CoT. Use it as reference. But don’t trust it. Verify outputs independently. If you’re running local LLMs, verification costs nothing. Use that luxury.

References

• Reasoning Models Don’t Always Say What They Think — Anthropic’s research. The shock of sub-20% CoT faithfulness
• Measuring Faithfulness Depends on How You Measure — Measurement-method dependency of faithfulness
• DeepSeek-R1 Thoughtology — Analysis of R1’s rumination patterns
• C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness — Benchmarking methodology for CoT faithfulness

If you’ve ever had to manually type invoice data into a spreadsheet — vendor names, totals, line items, due dates — you know how painfully slow and error-prone it is.

I needed to automate this for a project and couldn’t find anything that didn’t require training custom ML models or setting up heavy cloud infrastructure. So I built
aPapyr — a simple API that reads invoices (and receipts, tax forms, bank statements) and returns clean, structured JSON.

Here’s how it works in Python.

## Install

bash
  pip install apapyr

  Extract an Invoice

  from apapyr import aPapyr

  client = aPapyr("sk_live_your_key")
  result = client.extract("invoice.pdf")

  print(result.get_field("vendor_name"))  # "Acme Corp"
  print(result.get_field("total"))         # 1250.00
  print(result.get_field("due_date"))      # "2026-04-15"

  That's it. Three lines after setup. Send a PDF or image, get structured data back.

  What You Get Back

  Every field comes with a confidence score (0.0 to 1.0) so you know how reliable each value is:

  print(result.confidence)                     # 0.97 (overall)
  print(result.get_field_confidence("total"))  # 0.98
  print(result.get_field_confidence("notes"))  # 0.72 (handwritten, less certain)

  You decide your automation threshold. Confidence above 0.95? Auto-process it. Below 0.8? Flag it for human review.

  Line Items Too

  It doesn't just pull header fields — it extracts every line item:

  for item in result.line_items:
      desc = item.get("description", {}).get("value")
      qty = item.get("quantity", {}).get("value")
      amt = item.get("amount", {}).get("value")
      print(f"{desc}: {qty} x ${amt}")

  # Widget A: 50 x $25.00
  # Widget B: 30 x $12.50

  The API even cross-checks line item totals against the stated total and warns you if they don't add up.

  Flat Dictionary Output

  If you just want a simple key-value dict without confidence scores (for piping into a database or CSV):

  print(result.to_flat_dict())
  # {"document_type": "invoice", "vendor_name": "Acme Corp", "total": 1250.00, "due_date": "2026-04-15", ...}

  Supported Document Types

  It's not just invoices. Pass document_type="auto" (the default) and it detects the type automatically:

  - Invoices — vendor, total, tax, due date, line items
  - Receipts — merchant, items, subtotal, tax, tip, payment method
  - W-2 Tax Forms — employer, wages, withholdings
  - Bank Statements — balances, transaction history
  - Contracts — parties, dates, key terms

  Works With AI Agents Too

  If you use Claude Code, Cursor, or any MCP-compatible AI assistant:

  claude mcp add apapyr -- npx apapyr-mcp-server

  Then just ask: "Extract the data from invoice.pdf" — your AI handles everything.

  Try It Free

  - https://apapyr.com/free-tool.html — upload a document, no signup needed
  - https://apapyr.com/dashboard.html — 50 pages/month free, no credit card
  - https://apapyr.com/docs.html — full reference with examples in Python, Node.js, and cURL

  The free tier is enough to test it on real documents. If you're processing thousands of invoices, paid plans start at $49/month.

  ---
  aPapyr is open source on https://github.com/AkilaJ?tab=repositories&q=apapyr. Star it if you find it useful.

  ---

After understanding Atomicity, I explored another important concept: Consistency. Consistency ensures that the database always stays in a valid state(i.e no invalid data)

Creating a base table

CREATE TABLE accounts (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    balance INT NOT NULL CHECK (balance >= 0),
    last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
INSERT INTO accounts (name, balance) VALUES ('Christina', 1000), ('Sharon', 500);

Trying to Break the Rule (Negative Balance)

UPDATE accounts
SET balance = balance - 1500
WHERE name = 'Christina';

What Happened?

PostgreSQL rejected it:

ERROR: new row violates check constraint "balance >= 0"

Balance does not change.

Direct Invalid Update

UPDATE accounts
SET balance = -100
WHERE name = 'Sharon';

Again, it fails due to the CHECK constraint

What I Learned

• The rule CHECK (balance >= 0) is enforced by the database schema
• PostgreSQL automatically blocks invalid data
• This is how Consistency is maintained