Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Real Tasks

This chapter solves six everyday problems with a grammar each: a log file, semicolon-separated values, dates written in several ways, a small programming language, Markdown-like text, and the grammar inside a specification. Every solution has two parts, a grammar that turns the text into XML and a query that uses the XML.

Log files

log.ixml:

log: line++-#a, -#a?.
line: date, -" ", time, -" ", level, -" ", message.
@date: digit, digit, digit, digit, "-", digit, digit, "-", digit, digit.
@time: digit, digit, ":", digit, digit, ":", digit, digit.
@level: "INFO"; "WARN"; "ERROR".
message: ~[#a]*.
-digit: ["0"-"9"].

A log is lines separated by line feeds, with an optional line feed at the end (-#a?). The date, time and level become attributes of their line, and message takes everything up to the end of the line, ~[#a]*. The grammar describes this log:

app.log:

2026-10-20 09:15:02 INFO server started
2026-10-20 09:17:45 WARN disk 91% full
2026-10-20 09:21:13 ERROR disk full
2026-10-20 09:21:14 INFO cleanup started
2026-10-20 09:24:50 INFO cleanup done
2026-10-20 10:02:31 ERROR connection refused

Once the log is a document, grouping and filtering are ordinary XQuery:

levels.xq:

(: Counts the lines of each level in app.log and lists the errors. :)
let $parse := invisible-xml(unparsed-text("log.ixml"))
let $log := $parse(unparsed-text("app.log"))
return string-join((
  for $line in $log//line
  group by $level := $line/@level
  order by count($line) descending, $level
  return `{$level}: {count($line)}`,
  $log//line[@level = "ERROR"] ! `{@time} {message}`
), "
")

$ sxml xquery --method=text levels.xq
INFO: 3
ERROR: 2
WARN: 1
09:21:13 disk full
10:02:31 connection refused

Real logs are rarely this tidy. Two habits keep the grammar useful: make the part you care about precise (the date, the level) and let the rest be "anything up to the end of the line", and test the grammar on the oddest lines first.

Semicolon-separated values

For plain CSV, parse-csv (JSON, CSV, HTML and Invisible XML) is quicker. A grammar earns its place when the format is a variant that a CSV parser does not know. This one separates fields with semicolons and lets a quoted field hold semicolons and doubled quotes:

prices.ixml:

{Semicolon-separated values: a header line, then rows. A field in
 double quotes may hold semicolons; "" in it is one quote.}
table: header, -#a, (row, -#a)+.
header: field++-";".
row: field++-";".
field: quoted; plain.
-plain: ~[";"; '"'; #a]*.
-quoted: -'"', (~['"']; -'"', '"')*, -'"'.

prices.txt:

item;price;note
Coffee;3.50;"hot; large"
Tea;2.10;
"Juice ""fresh""";4.20;cold

$ sxml xquery --param=grammar=prices.ixml --param=input=prices.txt ixml.xq
<table>
  <header>
    <field>item</field>
    <field>price</field>
    <field>note</field>
  </header>
  <row>
    <field>Coffee</field>
    <field>3.50</field>
    <field>hot; large</field>
  </row>
  <row>
    <field>Tea</field>
    <field>2.10</field>
    <field/>
  </row>
  <row>
    <field>Juice "fresh"</field>
    <field>4.20</field>
    <field>cold</field>
  </row>
</table>

Note the first rule: every row ends with a line feed, (row, -#a)+. A first try, row++-#a, -#a?, reads the last line feed in two ways, as the end of the file or as a separator before a row with one empty field, and the result is ambiguous. Saying exactly where each line feed belongs removes the doubt.

From the table to JSON is one query, with the header naming the fields:

prices-json.xq:

(: Turns prices.txt into JSON: one object per row, keyed by the header. :)
declare option output:method "json";
declare option output:indent "yes";
let $table := invisible-xml(unparsed-text("prices.ixml"))(unparsed-text("prices.txt"))
let $names := $table//header/field ! string()
return array {
  for $row in $table//row
  return map:merge(
    for $field at $position in $row/field
    return map:entry($names[$position],
      if ($field castable as xs:decimal) then xs:decimal($field) else string($field)))
}

$ sxml xquery prices-json.xq
[
  {
    "item": "Coffee",
    "price": 3.5,
    "note": "hot; large"
  },
  {
    "item": "Tea",
    "price": 2.1,
    "note": ""
  },
  {
    "item": "Juice \"fresh\"",
    "price": 4.2,
    "note": "cold"
  }
]

Dates

People write dates in many ways. One grammar can read several forms and give them all the same XML, so the query after it sees one form only:

dates.ixml:

ixml version "1.1".
{Dates in three forms, one per line, all made into year, month and day.}
dates: (date, -#a)+.
date: iso; written; american.
-iso: year, -"-", month, -"-", day.
-written: day, -" ", month-name, -" ", year.
-american: month, -"/", day, -"/", year.
year: digit, digit, digit, digit.
month: digit, digit; +"0", digit.
day: digit, digit; +"0", digit.
month-name>month:
    -"January", +"01"; -"February", +"02"; -"March", +"03";
    -"April", +"04"; -"May", +"05"; -"June", +"06";
    -"July", +"07"; -"August", +"08"; -"September", +"09";
    -"October", +"10"; -"November", +"11"; -"December", +"12".
-digit: ["0"-"9"].

dates.txt:

2026-10-20
20 October 2026
7/4/1976
1 May 1886

Three features do the work. The three forms are hidden alternatives of date, so each gives year, month and day directly. The month names are hidden strings with an insertion of the month's number, and >month renames month-name to month, so a written month and a numbered one are the same element. And +"0", digit pads a one-digit day or month. The result is in the first chapter; here a query sorts the dates:

sort-dates.xq:

(: Reads dates.txt as xs:date values and lists them in order. :)
let $dates := invisible-xml(unparsed-text("dates.ixml"))(unparsed-text("dates.txt"))
for $date in $dates//date
let $value := xs:date(`{$date/year}-{$date/month}-{$date/day}`)
order by $value
return format-date($value, "[D] [MNn] [Y], a [FNn]")

$ sxml xquery --method=adaptive sort-dates.xq
"1 May 1886, a Saturday"
"4 July 1976, a Sunday"
"20 October 2026, a Tuesday"
"20 October 2026, a Tuesday"

The first and second lines of dates.txt were the same day in two forms.

A small programming language

A grammar can describe a language with statements, names and expressions. In this one, let names a value, print prints one, and * and / bind tighter than + and -:

calc.ixml:

ixml version "1.1".
{A small language: statements end with ";", "let" names a value,
 "print" prints one. Products bind tighter than sums.}
program: s, (statement, s, -";", s)*.
-statement: let; print.
let: -"let", S, @name, s, -"=", s, expression.
print: -"print", S, expression.
-expression: sum; -term.
sum: expression, s, @add>op, s, term.
-term: product; -factor.
product: term, s, @multiply>op, s, factor.
-factor: number; variable; -"(", s, expression, s, -")".
number: ["0"-"9"]+.
variable: @name.
@name: ["a"-"z"]+.
add: ["+-"].
multiply: ["*/"].
-s: -[" "; #a]*.
-S: -[" "; #a]+.

Precedence comes from the shape of the rules: a sum adds terms, a term is a product or a factor, and a product multiplies factors, so 1 + 2 * 3 can only be read with the product inside the sum. Because sum begins with expression and product with term, both are left-recursive, and 8 - 2 - 1 groups as (8 - 2) - 1. @add>op and @multiply>op make both kinds of operator an attribute op. s is optional space and S required space.

area.calc:

let width = 6;
let height = 7;
print width * height;
print (width + height) * 2;

$ sxml xquery --param=grammar=calc.ixml --param=input=area.calc ixml.xq
<program>
  <let name="width">
    <number>6</number>
  </let>
  <let name="height">
    <number>7</number>
  </let>
  <print>
    <product op="*">
      <variable name="width"/>
      <variable name="height"/>
    </product>
  </print>
  <print>
    <product op="*">
      <sum op="+">
        <variable name="width"/>
        <variable name="height"/>
      </sum>
      <number>2</number>
    </product>
  </print>
</program>

The XML is the program's syntax tree, and an interpreter is a recursive function over it:

run.xq:

(: Runs a program of the small language of calc.ixml, in the file $file:
   each print statement gives one integer. :)
declare variable $file external;

declare function local:value($node as element(), $names as map(*)) as xs:integer {
  typeswitch ($node)
    case element(number) return xs:integer($node)
    case element(variable) return $names(string($node/@name))
    default return
      let $left := local:value($node/*[1], $names)
      let $right := local:value($node/*[2], $names)
      return switch (string($node/@op))
        case "+" return $left + $right
        case "-" return $left - $right
        case "*" return $left * $right
        default return $left idiv $right
};

declare function local:run($statements as element()*, $names as map(*)) as xs:integer* {
  if (empty($statements)) then () else
  let $statement := head($statements)
  return
    if ($statement instance of element(let)) then
      local:run(tail($statements),
        map:put($names, string($statement/@name), local:value($statement/*, $names)))
    else (
      local:value($statement/*, $names),
      local:run(tail($statements), $names)
    )
};

let $program := invisible-xml(unparsed-text("calc.ixml"))(unparsed-text($file))
return local:run($program/program/*, {})

$ sxml xquery --method=adaptive --param=file=area.calc run.xq
42
26

The same tree could be checked for names used before they are set, pretty-printed, or translated into another language with XSLT.

Markdown-like text

The last grammar reads a little of Markdown: headings, lists and paragraphs separated by blank lines, with emphasis and code inside. Its names are HTML's, so the XML it makes is HTML:

markdown.ixml:

ixml version "1.1".
{A little Markdown: headings, lists and paragraphs, separated by blank
 lines, with *emphasis* and `code` inside. The names are HTML's.}
document>body: block++(-#a, -#a+), -#a*.
-block: h1; h2; ul; p.
h1: -"# ", inline.
h2: -"## ", inline.
ul: li++-#a.
li: -"- ", inline.
p: line++(-#a, +" ").
-line: ~["#-*`"; #a], inline?.
-inline: (char; em; code)+.
-char: ~["*`"; #a].
em: -"*", ~["*"; #a]+, -"*".
code: -"`", ~["`"; #a]+, -"`".

notes.md:

# Release notes

Version 2 reads *Invisible XML* grammars
and parses `CSV` files.

## Changes

- faster parsing
- better *error* messages

$ sxml xquery --param=grammar=markdown.ixml --param=input=notes.md ixml.xq
<body>
  <h1>Release notes</h1>
  <p>Version 2 reads <em>Invisible XML</em> grammars and parses <code>CSV</code> files.</p>
  <h2>Changes</h2>
  <ul>
    <li>faster parsing</li>
    <li>better <em>error</em> messages</li>
  </ul>
</body>

Two details make this grammar unambiguous. The characters of a line are single hidden characters, -char, so a run of them is one text node and there is only one way to split it. And a paragraph line may not start with # or -, so a heading or a list item can never also be read as a paragraph. The line feed inside a paragraph becomes a space with -#a, +" ".

The grammar in a specification

Specifications define their syntax in ABNF (RFC 5234), and the grammar sits in the document as text. The Vane draft (Transforming with XSLT and the chapters before it use it) has the syntax of its messages in a sourcecode element; vane.xml here is a link to the draft:

$ sxml xpath --context-item=vane.xml --method=text '//sourcecode[@type = "abnf"]'

message        = rollout-update / kill-update / status-request
rollout-update = "ROLLOUT" SP flag-key SP percentage CRLF
kill-update    = "KILL" SP flag-key CRLF
status-request = "STATUS" SP flag-key CRLF
flag-key       = 1*32(ALPHA / DIGIT / "-")
percentage     = 1*3DIGIT

To a program, those lines are one string. A small grammar for ABNF rules turns them into XML: rule names, alternatives separated by /, sequences of elements, repetitions such as 1*32 and 1*3, quoted strings and groups in parentheses:

abnf.ixml:

ixml version "1.1".
{The rules of ABNF (RFC 5234): one rule on each line, a name, "=" and
 alternatives separated by "/", each a sequence of elements, each
 element with an optional repetition.}
rules: -#a*, rule++(-#a+), -#a*.
rule: @name, s, -"=", s, alternatives.
-alternatives: alternative++(s, -"/", s).
alternative: element++S.
-element: repeat; -item.
repeat: (@min, -"*", @max?; -"*", @max; @min), -item.
-item: ref; string; group.
ref: @name.
string: -'"', ~['"']*, -'"'.
group: -"(", s, alternatives, s, -")".
@name: ["A"-"Z"; "a"-"z"], ["A"-"Z"; "a"-"z"; "0"-"9"; "-"]*.
@min: ["0"-"9"]+.
@max: ["0"-"9"]+.
-s: -" "*.
-S: -" "+.

rule++(-#a+) is one or more rules separated by line feeds, and element++S one or more elements separated by spaces. A repetition has a minimum, a maximum or both around the *, or just a number, which ABNF reads as an exact count. XPath extracts the text, and the grammar parses it:

$ sxml xquery --context-item=vane.xml --expression='declare option output:indent "yes"; invisible-xml(unparsed-text("abnf.ixml"))(string(//sourcecode[@type = "abnf"]))/rules/rule[@name = ("flag-key", "percentage")]'
<rule name="flag-key">
  <alternative>
    <repeat min="1" max="32">
      <group>
        <alternative>
          <ref name="ALPHA"/>
        </alternative>
        <alternative>
          <ref name="DIGIT"/>
        </alternative>
        <alternative>
          <string>-</string>
        </alternative>
      </group>
    </repeat>
  </alternative>
</rule>
<rule name="percentage">
  <alternative>
    <repeat min="1" max="3">
      <ref name="DIGIT"/>
    </repeat>
  </alternative>
</rule>

Once the grammar is XML, questions about it are paths. Which rules does each rule use, and which names does the draft use without defining them?

$ sxml xquery --context-item=vane.xml --method=adaptive --expression='
    let $rules := invisible-xml(unparsed-text("abnf.ixml"))(string(//sourcecode[@type = "abnf"]))//rule
    return (
      for $rule in $rules
      return `{$rule/@name} uses {string-join(distinct-values($rule//ref/@name), ", ")}`,
      `undefined: {string-join(distinct-values($rules//ref/@name[not(. = $rules/@name)]), ", ")}`
    )'
"message uses rollout-update, kill-update, status-request"
"rollout-update uses SP, flag-key, percentage, CRLF"
"kill-update uses SP, flag-key, CRLF"
"status-request uses SP, flag-key, CRLF"
"flag-key uses ALPHA, DIGIT"
"percentage uses DIGIT"
"undefined: SP, CRLF, ALPHA, DIGIT"

The four undefined names are the core rules of RFC 5234 (Appendix B.1), which every ABNF grammar may use, so the draft is complete. A misspelled rule name, flag_key for flag-key, would show up in the same list, and a broken rule fails to parse with the line and column of the mistake:

$ sxml xpath 'invisible-xml(unparsed-text("abnf.ixml"))("limit = percentage / ")'
<failed xmlns:ixml="http://invisiblexml.org/NS" ixml:state="failed"><line>1</line><column>22</column><position>22</position><end-of-input/><expected>"("</expected><expected>"*"</expected><expected>#20</expected><expected>'"'</expected><expected>["0"-"9"]</expected><expected>["A"-"Z"; "a"-"z"]</expected></failed>