ProductExperimental

Datter AI — The Input Gate Before AI Spend

Audit corpus redundancy, select a smaller candidate set, and test whether it still clears a task-specific quality floor before paying to process everything.

Data QualityRAGPythonStreamlit

Runtime and tools

Python
Streamlit

GitHub

Illustrative offline evaluation from a reproducible demo run. It is not a production benchmark, and it does not establish superiority over a matched random-selection baseline.

Pipeline: 7 auditable stages
Decision: Task quality floor
Output: Corpus + audit export

Meet Datter

Proof, not only a compression ratio

Datter treats corpus reduction as a decision that must be checked against the questions the data still needs to answer. The output is a selected corpus plus an audit trail, not an automatic claim that less is always better.

How it works

The seven-stage gate

The README workflow is grouped into five readable decisions here while preserving the complete ingest, chunk, deduplicate, score, select, evaluate, and report pipeline.

01Ingest + chunkRead PDF, text, and Markdown into inspectable units.
02DeduplicateFind exact and near-duplicate material.
03Score + selectPack a candidate set under a token budget.
04EvaluateCheck attached questions or a structural proxy.
05Export + reportShip the corpus, manifest, and audit trail.

Product

Datter estimates how much of a corpus can be removed while an attached task-specific evaluation remains above a chosen quality floor. It is designed to test data usefulness before paying to embed, label, fine-tune, or repeatedly retrieve from everything.

Why it exists

Document corpora often contain duplication, near-duplication, and material with little relevance to the downstream questions. Compression alone is not evidence of usefulness, so Datter couples selection with an explicit evaluation step and an audit trail.

Working prototype

PDF, text, and Markdown ingestion with automated chunking
Exact and near-duplicate auditing
Token-budget selection with optional task-query relevance
An offline retrieval-and-overlap quality proxy
Selected-corpus export and machine-readable audit reports

Build

Python selection and evaluation pipeline
Streamlit local interface
TF-IDF retrieval and token-overlap proxy evaluation
Pytest coverage and reproducible sample corpora

Maturity

What exists now

Experimental

Area	State	Evidence
Upload and automatic scan	Working	Local Streamlit flow for PDF, text, and Markdown.
Deduplication and selection	Working	Baseline heuristic scorer and token-budget selection.
Proof loop	Experimental	Current results use an offline proxy; an optional LLM judge is supported.
Optimised export	Working	Selected chunks, manifest, and Markdown/JSON reports.
External data connectors	Planned	S3, SQL, and Kafka remain interface placeholders.

Working with a large AI corpus?

Discuss data selection, corpus auditing, and how to test usefulness before committing to downstream AI spend.

Discuss Datter AI

Explore the other independent products in Tharm Lab.

All products