Work
Evaluation Format

Structured Knowledge Notation

A benchmark proving a token-efficient extraction format (SKN) that encodes confidence, causality, gaps, and risk inline beats plain JSON at the RAG extraction layer.

RoleSolo build
Timeline2025
StackPython · Ollama · qwen2.5:7b · DuckDuckGo Search · Bootstrap CI
Structured Knowledge Notation — project visual
01Thesis

JSON carries facts but zero epistemic metadata.

AI pipelines extract web-search results into JSON before passing them to a reasoning LLM. JSON has no per-claim confidence, no causal structure, no knowledge gaps, no risk signals, so the reasoning model is left guessing what is reliable and what is missing.

Structured Knowledge Notation is a drop-in replacement for JSON at the extraction layer that encodes confidence, causality, gaps, and risk inline, token-efficiently.

02Approach

Isolate the extraction format as the single variable.

The same DuckDuckGo search results feed six extraction variants, from a JSON baseline through ablations (no gaps, no causal, no risk) to full SKN, then the identical reasoning and evaluation functions run on all of them, so any downstream difference is attributable to the format alone.

v2 moved entirely to local Ollama (`qwen2.5:7b`, no API key), expanded to 50 samples across 7 categories, normalized the hallucination metric (fabricated/total claims), and added bootstrap 95% confidence intervals for statistical rigor.

6extraction variants
50benchmark samples
7question categories
Next projectAlgorithmic Trading System with AI Analysis