Products
Experimental

Datter AI — The Input Gate Before AI Spend

Audit corpus redundancy, select a smaller candidate set, and test whether it still clears a task-specific quality floor before paying to process everything.

Data QualityRAGPythonStreamlit
GitHub
Datter AI experimental corpus-audit dashboard
Illustrative offline evaluation from a reproducible demo run. It is not a production benchmark, and it does not establish superiority over a matched random-selection baseline.
At a glance
Pipeline
7 auditable stages
Decision
Task quality floor
Output
Corpus + audit export
Meet Datter

Proof, not only a compression ratio

Datter treats corpus reduction as a decision that must be checked against the questions the data still needs to answer. The output is a selected corpus plus an audit trail, not an automatic claim that less is always better.

Datter AI brain mark
How it works

The seven-stage gate

The README workflow is grouped into five readable decisions here while preserving the complete ingest, chunk, deduplicate, score, select, evaluate, and report pipeline.

  1. 01

    Ingest + chunk

    Read PDF, text, and Markdown into inspectable units.

  2. 02

    Deduplicate

    Find exact and near-duplicate material.

  3. 03

    Score + select

    Pack a candidate set under a token budget.

  4. 04

    Evaluate

    Check attached questions or a structural proxy.

  5. 05

    Export + report

    Ship the corpus, manifest, and audit trail.

Product

Datter estimates how much of a corpus can be removed while an attached task-specific evaluation remains above a chosen quality floor. It is designed to test data usefulness before paying to embed, label, fine-tune, or repeatedly retrieve from everything.

Why it exists

Document corpora often contain duplication, near-duplication, and material with little relevance to the downstream questions. Compression alone is not evidence of usefulness, so Datter couples selection with an explicit evaluation step and an audit trail.

Working prototype
  • PDF, text, and Markdown ingestion with automated chunking
  • Exact and near-duplicate auditing
  • Token-budget selection with optional task-query relevance
  • An offline retrieval-and-overlap quality proxy
  • Selected-corpus export and machine-readable audit reports
Build
  • Python selection and evaluation pipeline
  • Streamlit local interface
  • TF-IDF retrieval and token-overlap proxy evaluation
  • Pytest coverage and reproducible sample corpora
Product status

Upload and automatic scan

Working

Local Streamlit flow for PDF, text, and Markdown.

Deduplication and selection

Working

Baseline heuristic scorer and token-budget selection.

Proof loop

Experimental

Current results use an offline proxy; an optional LLM judge is supported.

Optimised export

Working

Selected chunks, manifest, and Markdown/JSON reports.

External data connectors

Planned

S3, SQL, and Kafka remain interface placeholders.

Evaluation limitation — current figures are illustrative offline proxies. Production use would require representative queries, a validated judge, and comparison against matched baselines, including random selection.