As Filed, Part Two

Part one ended with the extractor about half built and a promise that the next post would cover how it's put together and how well it scores, so this is that post, written from a slightly awkward position where the numbers are real but still moving week to week. Everything below is as of September 5. The short version is that there are now 214 financial maintenance covenants sitting in a table, pulled out of 162 credit agreements, each one backed by a verbatim quote that appears in exactly one place in the filed document, and the whole thing cost about $23 of model time to produce. It scores 0.87 against my label set, which sounds fine until you learn that the label set was drafted by the same family of model and that I haven't reviewed it yet. I'll get to why I decided that was acceptable, because I think it's the more interesting part.

Counting the corpus again

In part one I said the corpus was 552 documents, of which 77 were credit agreements, and while both numbers were technically correct the second one turned out to be close to meaningless. The classification came from the document's title, and titles in EDGAR exhibits are unreliable in both directions: when I sorted by length, 33 of the 77 "credit agreements" were cover pages and exhibit indexes under 10,000 characters, while 149 of the 344 "amendments" were full amended-and-restated agreements running well over 100,000 characters, with the entire covenant package inside them. The real count of documents that actually contain an agreement is 162, and getting there took one more correction, because 22 documents that looked like agreements by every structural test (length, numbered sections, a definitions article with hundreds of defined terms) were guarantee, collateral, intercreditor, and pledge agreements. Those have the shape of a credit agreement and no maintenance covenants at all, and I only found them because the extractor, when handed them, mostly declined to say anything (on seven of them it asserted covenant-lite, which is wrong in a way it had no means of knowing; a security agreement is covenant-lite in the sense that a cookbook is).

I'll mention one wrong turn here because it says something about how much weight a document's caption can bear. The first version of the classifier required the caption to say "credit agreement", which sounds reasonable and threw out 88 documents captioned "Amendment No. 3" or similar that attach the whole restated agreement as an exhibit. Those 88 had yielded 110 covenants in the batch run, more than the 69 documents that were captioned as credit agreements had (98), so the caption is useful for saying what a document isn't and not much else.

The segmentation from part one was at the article level, and that turned out to be the wrong grain for this problem. A financial covenant isn't an article; it's a numbered section inside the negative covenants article, "Section 7.11 Financial Covenants" or thereabouts, and in maybe a quarter of agreements the heading names the ratio instead of the concept ("Leverage Ratio", "Fixed Charge Coverage Ratio"), which means routing by heading needs a vocabulary rather than a phrase. So the segmenter now works at the section level under both numbering conventions the corpus uses (the "SECTION 7.11" style and the bare "7.11 Heading" style, and some documents mix them), and it also indexes every defined term to the character span of its definition, which matters later. Across the full agreements, 95% have ten or more numbered sections, 66% have a section whose heading names a covenant, and 97% have at least fifty defined terms indexed. My first pass at this kept the table-of-contents entry rather than the body heading on six of the first seven documents I checked, for a reason that was obvious in hindsight: a table of contents is by construction the longest ascending run of section numbers in a document, so a rule that preferred long runs was a rule that preferred the table of contents. What fixed it was a density test, on the theory that a heading with another heading within 250 characters on either side is a list rather than a section, and that turned out to hold across the corpus.

How it reads a document

The one rule I'd keep if I had to throw everything else away is that the model has to quote the document. Every covenant it returns has to carry a quote of at least twenty characters that occurs exactly once in the frozen text of the filing, or it's rejected; if a quote fails, the model gets one chance to repair it, with the nearest matching passage shown back to it, and a second failure drops the covenant and logs why. That's the whole defense against the model inventing something. It can't put a covenant into the table, it can only point at one, and the pointer (the quote becomes a character span, which becomes the citation) is re-checked every time the table is rebuilt. Across five runs and 302 model calls, the number of quotes that failed to locate after a repair attempt is zero, which I'd like to say is because the prompt is good but is mostly because the model is good at copying text when you insist on it.

The model doesn't see the whole agreement if it can be avoided, partly for cost and partly because a 290,000-character document is a bad place to ask a precise question. For each document the routing tries, in order: the sections the segmenter typed as financial covenants; failing that, the negative covenants article; failing that, every sentence containing a covenant verb ("shall not permit", "will at no time exceed", "make or commit to make") alongside a ratio term, each with 3,000 characters of context on either side; and only when all of that fails, the whole document in chunks. On the full run, 29 of 184 documents fell all the way through to the last option, and they accounted for most of the bill.

Definitions go in with the text, which is where the defined-term index earns its keep. If the candidate spans mention "Consolidated First Lien Net Leverage Ratio", the definition of that term comes along from the index (up to ten definitions per document, ranked by how often the term shows up in the spans and whether it's the object of a covenant verb). This ranking wasn't in the first version, which took the first six definitions it happened to find, and on at least one document that left out the one definition that mattered. More on that below.

What comes back is a label in a fixed schema: covenant type from a closed list, the defined term, the numeric threshold, the ratio text as written, the test frequency, the springing trigger if there is one, a step schedule, the quote, and a free-text note. One model does the extraction and a second, smaller one independently reads a sample of the same documents as a check, for reasons I'll get to.

Scoring, and the problem with the labels

Part one said the labels would be mine, and they aren't, at least not yet. Thirty-one documents were labeled by the model in a single pass at the end of August, with every quote span-validated in the same way, and I set aside seven blind holdouts that no model has seen. I haven't had the hours to go through the thirty-one properly. Rather than let that stall everything, the labels are treated as provisional gold, the scorer was frozen, and the extractor is scored against them anyway, with the understanding that my review will improve the score when it happens and doesn't gate the work in the meantime.

The obvious objection is that a model grading a model can agree with itself for bad reasons, and that's true. What makes it less bad than it sounds is that the extractor reads the document differently from the labeler (it has the sections and the ranked definitions routed in, and a much narrower question), so when the two disagree, the document itself usually settles it. So far the label set has been wrong outright once: Caesars' 2022 agreement carries an $850 million minimum liquidity covenant during a covenant relief period that the labels missed entirely, and the label schema didn't even have a type for it, which it now does. Two other labels were stale rather than wrong, typed "other" for a monthly EBITDAR test and a cash-on-hand floor because the schema had no better value when they were written, and were retyped once it did.

Here's how the score moved on the same ten documents, using a second version of the scorer that I'll explain in a second:

Two of the run-one misses are worth spelling out because they're the kind of error that would look completely fine on a spreadsheet. The Caesars covenant is written on the "Total Leverage Ratio", which sounds like a gross number; the definition says Total Net Debt over EBITDA, and the model typed it gross from the name because the definition never reached it (the routing had attached the first six definitions it found and this wasn't among them). Fixing it took two things, a sentence in the prompt saying the definition wins over the name, and the ranking so that the right definition is actually attached. The second miss was Fluor's 2004 agreement, where the model returned a $400 million cap on subsidiary debt as a maintenance covenant. It's a basket, a limit on incurring a category of debt, and it belongs with the negative covenants rather than the financial ones; that's a distinction a credit analyst makes without thinking about it and a model has to be told explicitly.

The scorer versioning is because I didn't want to quietly loosen the grading to make the number go up. The original scorer matches a covenant by type and by overlap of the quoted spans, and it was marking as misses things that were plainly the same covenant quoted from a different part of the same sentence (the label quoted the "shall not permit" head, the extractor quoted the threshold clause), along with defined-term strings that differed by a parenthetical and step schedules that differed by date format. The second version matches on the same sentence, normalizes the term and date strings, and collapses step schedules to their change points before comparing, since the model returns every test date and threshold while the labels only recorded the dates the threshold changes (I kept the full schedule, it being what the document says). Under the original scorer run 3 is 0.80. Both numbers are reported in every run; neither is hidden.

One more thing the runs showed, which I hadn't expected to matter at this scale: adaptive reasoning isn't deterministic, and on identical inputs one covenant's type flipped between two runs, which at fifteen covenants is 0.07 of F1 on its own. The more stable number is the score on the twenty-nine labeled documents inside the full run, and that also comes out at 0.87.

What the full run looked like

The 184 documents went through the batch API in a single submission: 231 calls, about three minutes of wall clock, no errors, and $17.82 against an estimate of $10 that I'd made from the ten-document runs (29 documents fell through to whole-document reads, and the model wrote more than I'd assumed, mostly step schedules). The second model then read twenty of the same documents for $0.61 and agreed with the first on the covenant set in eighteen of them, with eleven field-level disagreements and none at all about whether a covenant existed. Total model spend on the project to date is $22.85, which I mention because I'd assumed going in that this part would be the expensive one.

Ninety-six of these are tested quarterly, 53 are springing, 43 apply at all times, and the remainder are monthly, annual, or something the schema doesn't have a name for yet.

Close

The 19 documents with no valid output split two ways. Fifteen of them weren't credit agreements at all, as above, and the other four are routing misses, where the sentence windows surfaced reporting covenants or baskets instead of the covenant article; those four are the next thing I'd fix on the reading side. Over the 162 documents that actually are agreements, 158 produced either a covenant or a covenant-lite finding, which is a better number than I expected and probably a bit flattering, since "produced a finding" and "produced the right finding" are different things and only 29 of the 162 have labels to check against.

On cost, the per-agreement figure works out to around ten cents and half that through the batch API, so the whole corpus is a one-time expense of under fifty dollars and a document only ever needs reading once per version. What makes it cheap is the routing rather than the discount; a whole-document read costs five to eight times what a routed one does, and the batch pricing sits on top of that.

Where it's still wrong

Mostly amendments, which I think is the bigger of the problems left. A sentence like "Section 7.11 is hereby amended by replacing '4.50 to 1.00' with '5.00 to 1.00'" doesn't restate a covenant, it edits one, and reading it correctly means resolving the edit against the parent agreement, which nothing here does yet. The full amended-and-restated agreements are handled because they carry the whole text; the short amendments, which is where the actual step-downs over a facility's life tend to live, are not, and I suspect that's the harder half of the problem rather than the tail of it.

Then the four routing misses, and two documents whose covenant section the segmenter still can't find: one puts the covenants in an unnumbered annex, and the other rewrote the section as a lettered clause inside an amendment, so the only candidate heading is the table-of-contents entry. The second model, reading on its own, made two errors the first didn't (a threshold in millions where the schema wants dollars, and a "Consolidated Secured Debt Ratio" typed as other rather than senior secured leverage), both of which are prompt fixes that are now in.

And the labels are still model-drafted. Every score in this post is against a label set I haven't read, and the corrections made to it so far were made because the document text confirmed them rather than because I did.

What's usable now

Locally, the same app from part one has grown a covenants register with the whole table in it, filterable by type, frequency, confidence, and review status, along with a per-company view that lists each agreement's covenants with a link into the document at the exact character, and a labeling workbench that loads the extractor's draft into a form, validates every quote against the text, and saves the label. The workbench is how the review will happen, whenever I get to it. There's also an MCP server over the same tables, so a coding assistant on my machine can ask for a company's covenants, its filed statements as of a date, or the exact text behind a citation, and get the answer straight from the parquet with the source table and license attached.

Close

Close

The project is still private. Part three will be a hosted version of the application, with the statements and the covenant table behind it, once the amendments are handled and the labels have had my review.