The Digesta decision trees

Read from the code on 4 November 2026 · figures measured against the production PostgreSQL

Diogene is a source of DOCUMENTS; the schede are a source of DATA. Two different things that were living in the same place. Diogene gives us 3.884.843 documents — filings from electronic case files, each with a file — while the 1.091.881 schede are the case-file record cards: no PDF, no pages, nothing to read and nothing to extract. Their job is to hold the documents together and carry their metadata.

As long as the schede sat in documents they inflated every total («11.906.322 documents» included 1,09 million that are not documents at all) — and Diogene, conversely, was invisible: its documents sit under the sources merito_tribunale and merito_gdp, which look like the lower-court database and are not. The source column names the register, not who gave them to us.

Diogene by register: dgn:cc 2.357.766, dgn:gdp 1.443.430, dgn:esm 83.647.

Four decisions in a row. Each has exactly one job, and the output of one is the input of the next:

* the perimeter is not closed yet: 1.623.869 is what cataloguing carries away as done so far. 737.931 invisibles remain, and until they have a type there is no telling whether they are primary — that is tree 2's job.

Tree 1 · the CHOICEWhich documents we look at — and no others

It decides one thing only: in or out. Out of 10.814.441 catalogued documents it takes 2.934.278 by four rules — and that is the subcorpus, what this alpha version works on. It does not decide what they are, nor how to read them: that is where it ends.

You cannot exclude by class something that has no class. Keeping the invisibles out would mean never finding out what they are: they would not enter, so they would not be read, so they would never get a type. They come in, and tree 2 will say whether they are primary. Also in are spese, altro and veicolo, which looked like procedural paperwork and in fact carry invoices, payment demands, attachment orders and vehicle registry extracts.

Tree 2 · CATALOGUINGWhat type each document is — and which are PRIMARY

It decides the TYPE of every document in the subcorpus, and out of that comes the PERIMETER: the primary documents. A hearing record, a power of attorney, a certified email carry none of the fields we extract — they are the paperwork of procedure, not of the case. Cataloguing them is precisely how we keep them out of the perimeter, and therefore out of pre-OCR. In come the 2.934.278 of the subcorpus, out comes a type for each, and the type says which are primary.

The perimeter — who passes (1.185.653 by class): memoria 345.008, provvedimento 267.633, introduttivo 132.217, ctu_perizia 119.111, spese 77.735, assicurativo 71.372, comparsa 67.924, medica 56.394, sinistro 24.657, veicolo 16.251, altro 7.351 — plus the 438.216 that pass only by the type the source declares (sentenze 206.789, ordinanze 182.023, decreti 42.154). The two routes make 1.623.869, which is exactly what sits in lavoro_ocr today.

Of these, 1.269.602 already have text and 354.267 do not. The real work is the second figure: the source's rulings almost all have text (0 ordinanze and 0 decreti without), whereas medica (42.979 of 56.394) and sinistro (16.313 of 24.657) are almost all still to be read.

Who does not pass (482.494): tecnico 135.676 (envelopes, certifications, filing notes), procura 88.031, corrispondenza 80.531, hearing records 70.834, notifica 44.933, F24 tax forms 27.840, photographs 27.180, registry data 6.490. Reading them would cost money and yield not one field.

The 275.666 without a type are neither in nor out: until they are catalogued, the perimeter is not closed. Nearly all of them are waiting for text — and text is what pre-OCR produces, which is the next tree. It is the one point where the two chase each other.

From here the journey is a single one: the primary documents enter pre-OCR, from there the ones that need it go to OCR, and the text that comes out goes to extraction.

The first question is not «is it an envelope?»: it is «do we already know it?». A document that has a class, an envelope or a name that speaks is read as it is, and the five sources vote. A document we know nothing about goes first to discovery: we find out what the file really is, we split off the text layer that already exists, we unwrap wherever there is an envelope, and only over the pages that remain does tesseract run. At that point text exists — and that document is catalogued from its text like every other, entering through the testo source and from there to the vote. It skips nothing: it arrives late at the same table.

Five independent sources, each proposing a class with a strength; then they vote. Text is one of the five, and that is why 1,4 million documents have a class despite never having had a single character of text.

Discovery: four steps, and back in line with everyone else

The filename, the envelope and Diogene's own label are read as they come. For a document with none of the three, text has to be brought into existence, and that takes four steps. The 16-byte guardian says what the file really is, because anything at all can arrive behind a .pdf name. The PDF's text layer is split off: the pages that already have text cost nothing. Unwrapping opens the envelopes where there are any — and it is a loop, not a step, because what comes out can itself be an envelope. And tesseract reads only the pages left over, on CPU we already own: cataloguing needs 3.500 characters, which is to say the first page.

Once the four steps are done text exists, and from there on that document is no longer a special case: it enters through the testo source like any document that had text from the start, and the vote treats it identically. That is why testo is no longer the «expensive» source in the drawing: in discovery the price is paid by the CPU, not the GPU.

And after the vote, low confidence is discarded

The 89.984 at low confidence end there: they do not enter the filter and they go no further. A contradicted class, or one resting on a weak clue, mapping onto the empty tuple would not get a field wrong — it would lose the whole document. They are left out deliberately, not by oversight.

Level 2 — the Qwen model (197.235 documents) and the CPU classifier (69.656) — is not a step after the vote: it is one of the ways a class gets decided, and once it has spoken its verdict is already inside the high or the medium band.

Text is not free the way the other three sources are

The filename, the envelope and Diogene's label are read as they come. Text is not: first the file has to actually be a PDF. The 16-byte guardian runs ALWAYS, not only when the type is missing — anything at all arrives behind a .pdf name, and the stored type can be wrong: 7.716 documents out of 97.783 were recorded as scanned and were PCT XML envelopes, JPEG photographs, zips, mail messages. As long as the check ran only on an empty type, those files left for the GPU as scans and came back without a character.

And a container — certified email, ZIP, XML envelope, CAdES — is not a document: it gets unwrapped, the main document that comes out is re-typed and re-enters from the beginning. It is a loop, not a step: unwrapping can itself produce something to unwrap.

The two hand-written disambiguations

Between the text and the filing-type recogniser there are two cases the vote alone would get wrong, and they sit in the code as explicit exceptions: a judgment delivered orally into the record (art. 281-sexies) is a ruling even though the header says «hearing record», and a party's filing that cites a decree is not a decree.

Tree 3 · READINGHow to read the primary documents

Every primary document enters: the whole Alpha 2 Perimeter, as tree 2 passes it. There is no choosing by type here any more — the filter of tree 2 has already decided what is primary.

The estimate uses figures measured on the perimeter: 11,9 pages per document to be read, 52% of pages escalating to the GPU after the tesseract gate, 0,59 pages/s per pod and 0,59 $/h per pod. A document that already has text costs nothing: it just goes into extraction.

Pre-OCR does not read: it decides. First it discards what is not a document, then it looks page by page for text that is already there, and only what is left goes to the GPU. It costs tenths of a second of CPU.

The 354.267 without text are this tree's work. They have no text anywhere: without this pass they stay documents that can be neither catalogued better nor extracted. The other 1.269.602 already have text and can go to extraction without spending a minute of GPU.

In here there is nothing left to settle on the text layer: 99% is genuine scan, and obviously so — these are precisely the documents that have no text. The pages go to the tesseract gate, and only those tesseract cannot read well enough escalate to the GPU:

Tree 4 · EXTRACTIONWhich fields to ask for, given the type

Fed by what tree 3 PASSED: every primary document it has read. The same output feeds tree 5, and the two run side by side. Running now: the pilot, 2,000 documents from tree 3's PASSED, stratified by kind of document, on the base qwen38 model with no adapter — it measures the prompt before it is frozen for tree 6. The boxes below count this run.

Here the class really earns its keep: it decides which fields to ask for. You do not ask a medico-legal report for the operative part of a judgment, and you do not ask a hearing record how the accident happened. Asking everything of everyone is what produced 89% empty values in the 3 November pass — and you pay for it twice, in prompt and in empty answers written up as though the model had failed.

The catalogue albero4.v1 merges three lists into one: the judgment catalogue (sib.v4, 167 fields), the 27 case-file questions and Diogene's 127 fields, of which 55 had no counterpart in ours. The draft had 225 fields across 18 kinds; after the review of 08/11/2026 it was frozen as albero4.v1 and then albero4.v2: 249 fields across 20 kinds of document. Each field carries a judgment reading and a filing reading, and the kinds of document that can contain it. In v2 a missing value is null, and −1 on a liquidated amount means the claim was rejected.

Tree 5 · CHUNKS AND VECTORSCutting the text into pieces, and turning each piece into a vector

Fed by what tree 3 PASSED, the same output that feeds tree 4. Tree 5 runs beside extraction, not after it — neither waits for the other.

A small tree with one job: make the text searchable by meaning. The text is cut into pieces, every piece becomes a vector, and the vectors go to Qdrant. Two gates come first, because a vector is hard to take back once someone has searched with it: a text that turns out to be in fase di oscuramento must not be indexed at all, and a page that holds only a signature is not text — it goes back to be read again.

Every document is chunked and embedded again, from scratch, even when it already has pieces and vectors: on 4,000 documents sampled on 07/11, 2.7% had pieces older than their current text, and cutting is cheap (about 2 ms a document against 313 ms of embedding). The new pieces replace the old ones and the new vectors replace the old points in Qdrant. A document whose write fails halfway has no «done» mark in this run and is simply done again.

Tree 6 · TRAININGTeaching digesta-alpha the extraction tree 4 will ask of it

Not training yet. It starts once the catalogue and tree 4's prompt are frozen after the 2,000-document pilot: every example must look exactly like what tree 4 will send. Steps 1–2 are done: 0 labels on 0 documents collected, the gate frozen. The pilot is running now, and every answer it gets back is kept in tree 4's exact prompt format: 0 so far. No training example exists yet, so that counter stays at zero.

The data to teach from exists already; nothing here waits for tree 4 to run. What changes is its shape: every source is rewritten as one tree 4 call — the fields that kind of document can carry, the window, the photo blocks — and the answer tree 4 expects back: value, confidence and verbatim passage for each field, plus the note on the document. The blind half of the gold set is never shown to the model: it is the gate.

The thresholds, and why they are what they are

Three things the tree does that its name does not reveal

The text layer of mixed documents is saved regardless. In ridotto mode the pages that do not go to the engine have text, and that text is written straight away as a partial reading: quality forced to «failed» and pagine_fallite telling the next tier which pages are missing. Without this the document would leave the queue and its text would be re-extracted every time — measured on 31 October, 81.314 pages across 2.930 documents.

Photographs do not go to OCR. Since 25 October: on figure-pages 46,7% of requests came back 5xx against 1,7% for normal pages, and of the 83% that did come back the reading was rejected. They stay recorded under figure, and that is the vision queue.

A document sent whole «as a fallback» is distinguishable. When the PDF will not open or exceeds the page ceiling, the mode is intero but per_pagina is false and motivo says why. Without that distinction it was indistinguishable from a document that genuinely is all scan — and they are 8%.