Structure and quotations

chaff reads a numbered document, such as a contract or a specification, as a tree of addresses. As a tree, breaks like a reference to a section that is not there, or a skipped number, can be checked by machine. The same tree checks whether the passages an AI quotes are really in the source.

Seeing a document as a tree

tree prints every address, with what sits at each. Plain .txt is read as well as Markdown, because contracts often come as text without headings.

On a contract it prints:

$ npx chaffjs tree contract.txt
(doc :language "en" :path "contract.txt" :line 1
  (definition :form "bare" :placement "inline" :term "the Client" :line 3)
  (definition :form "bare" :placement "inline" :term "the Supplier" :line 3)
  (article "1" :heading "Purpose" :label "Section 1" :line 5
    (reference :label "Section 3" :numbering "section" :target "3" :line 6))
  (article "2" :heading "Definitions" :label "Section 2" :line 8
    (definition :term "Deliverables" :line 9))
  (article "3" :heading "Services" :label "Section 3" :line 11
    (obligation :marker "shall" :type "must" :line 12)
    (item "3.a" :label "(a)" :line 13)
    (item "3.b" :label "(b)" :line 14)
    (item "3.c" :label "(c)" :line 15))
  (article "4" :heading "Fees" :label "Section 4" :line 17
    (article "4.1" :heading "The Client shall pay a fee of $50,000." :label "4.1" :line 18
      (obligation :marker "shall" :type "must" :line 18)
      (quantity :unit "$" :value 50000 :line 18))
    (article "4.2" :heading "The Client shall pay within 30 days of accepting the Deliverables." :label "4.2" :line 19
      (obligation :marker "shall" :type "must" :line 19)
      (quantity :unit "days" :value 30 :line 19))
    (article "4.3" :heading "A late payment bears interest of 3% a year." :label "4.3" :line 20
      (quantity :unit "%" :value 3 :line 20)))
  (article "5" :heading "Subcontracting" :label "Section 5" :line 22
    (obligation :marker "must not" :type "must-not" :line 23))
  (article "6" :heading "Term" :label "Section 6" :line 25
    (date :value "2024-04-01" :line 26)
    (date :value "2025-03-31" :line 26)
    (article "6.2" :heading "Either party may renew it for 1 year by giving 3 months' notice." :label "6.2" :line 27
      (obligation :marker "may" :type "may" :line 27)
      (quantity :unit "year" :value 1 :line 27)
      (quantity :unit "months" :value 3 :line 27)))
  (article "7" :heading "Termination" :label "Section 7" :line 29
    (obligation :marker "may" :type "may" :line 30)
    (reference :label "Section 4.2" :numbering "section" :target "4.2" :line 30)
    (reference :label "Section 5" :numbering "section" :target "5" :line 30)))

article is a section and item a subsection or a lettered item. Inside them are definitions (definition), references (reference), obligations (obligation), quantities (quantity) and dates (date). :line is the line in the source.

Reading an address

An address is the numbers joined by dots. The number as written stays in :label.

A reference carries an address of the same shape: “Section 4.2” above is :target "4.2". So what a reference points at can be looked up in the tree.

Finding structural breaks

These rules read the tree.

All three run with no settings. This is a short contract, contract.txt, after some sections were deleted.

SERVICES AGREEMENT

Section 1 Purpose
This Agreement covers the services described in Section 3.

Section 4 Fees
4.1 The Client shall pay a fee of $50,000.
4.2 The Client shall pay within 30 days of accepting the Deliverables.

Section 7 Termination
The Client may terminate this Agreement if the Supplier breaches Section 9.

$ npx chaffjs contract.txt --compact

contract.txt   blog/tech · English   genre from the default
   Looks like: Contract and terms. Try --genre legal/contract

  4:49    error   "Section 3" (address 3) is not in this document
                  dangling-reference
  6:1     error   "Section 4" follows "Section 1" (expected number 2)
                  numbering-gap
  10:1    error   "Section 7" follows "Section 4" (expected number 5)
                  numbering-gap
  11:66   error   "Section 9" (address 9) is not in this document
                  dangling-reference

4 findings, 103 rules not run

The line under the first one suggests checking it as a contract. npx chaffjs contract.txt --genre legal/contract reports the same four.

A reference into another document, such as “section 9 of the Companies Act 2006”, is not looked up here. Numbers are compared only among siblings under the same parent. duplicate-definition does not decide whether the two definitions disagree.

A contract taken out of a PDF often has its numbered lines indented too deep, or run into the text. Sometimes the text shows many clause numbers but the tree read almost none. Then the three rules are listed as not run, with both counts, instead of reporting nothing.

A worked example: tidying an amendment to an Act

You can try it on a real Act. Sections 30 to 32 of the Arbitration Act 1996 are copied here, with two mistakes of the kind made while drafting an amendment:

• Subsection (2) of section 31 was deleted, so the subsections run (1), (3), (4), (5).
• In section 31(5), “section 32” was mistyped as “section 35”.

draft.txt reads:

Section 30 Competence of tribunal to rule on its own jurisdiction.
(1) Unless otherwise agreed by the parties, the arbitral tribunal may rule on its own substantive jurisdiction, that is, as to—
(a) whether there is a valid arbitration agreement,
(b) whether the tribunal is properly constituted, and
(c) what matters have been submitted to arbitration in accordance with the arbitration agreement.
(2) Any such ruling may be challenged by any available arbitral process of appeal or review or in accordance with the provisions of this Part.

Section 31 Objection to substantive jurisdiction of tribunal.
(1) An objection that the arbitral tribunal lacks substantive jurisdiction at the outset of the proceedings must be raised by a party not later than the time he takes the first step in the proceedings to contest the merits of any matter in relation to which he challenges the tribunal’s jurisdiction.
(3) The arbitral tribunal may admit an objection later than the time specified in subsection (1) or (2) if it considers the delay justified.
(4) Where an objection is duly taken to the tribunal’s substantive jurisdiction and the tribunal has power to rule on its own jurisdiction, it may—
(a) rule on the matter in an award as to jurisdiction, or
(b) deal with the objection in its award on the merits.
(5) The tribunal may in any case, and shall if the parties so agree, stay proceedings whilst an application is made to the court under section 35 (determination of preliminary point of jurisdiction).

Section 32 Determination of preliminary point of jurisdiction.
(1) The court may, on the application of a party to arbitral proceedings (upon notice to the other parties), determine any question as to the substantive jurisdiction of the tribunal.
(1A) An application under this section must not be considered to the extent that it is in respect of a question on which the tribunal has already ruled.
(2) An application under this section shall not be considered unless—
(a) it is made with the agreement in writing of all the other parties to the proceedings, or
(b) it is made with the permission of the tribunal...
(3) ................................
(4) Unless otherwise agreed by the parties, the arbitral tribunal may continue the arbitral proceedings and make an award while an application to the court under this section is pending.
(5) Unless the court gives leave, no appeal lies from a decision of the court whether either condition specified in subsection (2) is met.
(6) The decision of the court on the question of jurisdiction shall be treated as a judgment of the court for the purposes of an appeal.

Run it with nothing added.

$ npx chaffjs draft.txt --compact

draft.txt   blog/tech · English   genre from the default

  9:1     info    This sentence runs 51 words (limit 40)
                  max-sentence-length
  10:1    error   "(3)" follows "(1)" (expected number 2)
                  numbering-gap
  14:136  error   "section 35" (address 35) is not in this document
                  dangling-reference

3 findings, 103 rules not run

The two error lines come from the two changes. Line 10 is the skipped subsection, and line 14 is the reference to a section that is not there. The info lines mark places to reread for readability; long sentences are normal in legislation. To check only the structure of an Act, read the error lines.

Run it as a statute, with --genre legal/statute, and the limits are a statute’s: only the two error lines are left.

Undo the two changes and run fixed.txt: the error lines are gone.

$ npx chaffjs fixed.txt --compact

fixed.txt   blog/tech · English   genre from the default

  9:1     info    This sentence runs 51 words (limit 40)
                  max-sentence-length

1 finding, 103 rules not run

An Act in force should have no such breaks. The chaff repository runs the structure rules over the full text of three UK Acts and four Japanese statutes. A test checks that none of them gets a finding, and yarn corpus does the same measurement locally.

In English, chaff reads the other members of a list (“sections 44 and 45”) and inserted subsections (“(1A)”). It does not yet read a reference to a subsection without its section (“subsection (1)”).

The extract is from the Arbitration Act 1996 (legislation.gov.uk) and contains public sector information licensed under the Open Government Licence v3.0.

Checking an answer’s quotations

When an AI reads a document and answers, it quotes: “Section 4.2 says this”. cite checks that each quotation really is at that address in the source.

npx chaffjs cite contract.txt claims.json

The quotations are a JSON array of pairs: an address (address) and the quoted text (quote).

[{ "address": "4.2", "quote": "within 30 days of accepting" }]

Spaces and line breaks are ignored, and characters are compared after NFKC normalisation. So a quotation that differs only in spacing, or in full-width digits, still matches.

$ npx chaffjs cite contract.txt claims.json
✓ 4.2 "within 30 days of accepting": matches

A quotation with a changed number, or one taken from somewhere else, fails. When it is somewhere else, you are told the real address.

$ npx chaffjs cite contract.txt claims.json
✗ 4.2 "within 60 days of accepting": the quotation is nowhere in the source
✗ 7 "shall pay a fee of $50,000": the quotation is not at 7 but at 4.1 (line 18)

If any quotation fails, the run ends with exit code 1, so an answer’s quotations can be checked like a unit test.

When you do not know where a quotation is, leave address out or empty. It matches if the words are anywhere in the source, and you are told the address and line where they are. This also works for a source with no addresses, such as an article without headings.

$ npx chaffjs cite contract.txt claims.json
✓ (anywhere) "within 30 days of accepting": matches (4.2, line 19)

A quotation with no source

Before cite can check a quotation, someone has to know where the words came from. When a document gives words to someone (“X said …”, “according to X”) and names no source, there is nothing to check them against. quote-without-source points at such a quotation: in quotation marks, given to someone, in a paragraph with no link, no footnote, no numbered citation and no year in brackets. A block quotation (>) with a dash line, ”— Name”, at its end or right after it is given to someone too. A year or a link on that line (”— Name, Title (1975)”) counts as its source. It runs in blogs, papers, manuals, legal documents, literature and speeches.

# Notes from the platform meetup

The keynote opened with a number. Dana Reyes, who runs the on-call rotation at a payments company, said "most outages start with a change nobody reviewed".

The second talk was about rollbacks. Its slides are at https://example.com/rollbacks, and its claim, "a rollback you have never run is not a rollback", is on slide 4.

The closing slide quoted an old line:

> A change you cannot undo is a change you have not finished.
>
> — Kim Lee

$ npx chaffjs meetup.md --compact

meetup.md   blog/tech · English   genre from the default

  3:106   info    "most outages start with a change nobody reviewed" is given to someone, but no source (a link, a footnote, a reference) is given
                  quote-without-source
  9:3     info    "A change you cannot undo is a change you have not finished." is given to someone, but no source (a link, a footnote, a reference) is given
                  quote-without-source

2 findings, 71 rules not run

The second quotation has a link in its paragraph, so it is not reported. The block quotation names only who said it, so it is. chaff does not open the link to see whether the words are there. With the source saved as a file, cite checks that.

cite --scaffold writes the quotations that give no source as a quotes.json for you to fill in:

$ npx chaffjs cite --scaffold meetup.md
[
  {
    "source": "",
    "address": "",
    "quote": "most outages start with a change nobody reviewed"
  },
  {
    "source": "",
    "address": "",
    "quote": "A change you cannot undo is a change you have not finished."
  }
]

Save it, and write in each source the file the words come from, relative to the quotes file ("source": "talk-transcript.md"). address can stay empty: the words are then looked for anywhere in that file. When every quotation names its source, give cite the quotes file alone:

npx chaffjs cite quotes.json

Each line of the result starts with the source it checked. With a source on the command line as well, a quotation that names its own is an error, so the two are never mixed up.

A source on the web

When the source is a web page, give its address instead of a file:

npx chaffjs cite --url https://example.com/talk quotes.json

A source in the quotes file may be an http:// or https:// address too. chaff fetches the page once and reads it as chaff tree reads a file. HTML is turned into Markdown first, so the page’s headings become addresses (h1.2); a page with no headings still works with an empty address. After the fetch nothing depends on the network: the same page gives the same result. A page that cannot be fetched within 30 seconds, answers with an HTTP error, is larger than 20 MB, or is not text (a PDF, an image) ends the run with exit code 1 and says which address failed.

What chaff does, and what it does not

chaff only decides whether something is broken. It never rewrites the document. Reading the document and answering is the AI’s side; chaff checks before and after.

What to read next

• A worked example on internal rules, with the structural checks turned on, is in Internal rules and regulations.
• Checking a RAG answer’s quotations with cite inside an eval is in Using chaff for AI evals.
• tree and cite are listed with the other commands in Commands.