sxml in JavaScript

The @tinyactors/sxml package runs the sxml library in Node.js and in browsers, compiled to WebAssembly, with a small TypeScript wrapper and no dependencies. It speaks the same languages as the C library and the sxml command, and gives the same results. The examples are in docs/book/examples/javascript.

Getting the package

The package is in the sxml repository, in bindings/javascript. It is not on npm yet; its README explains how to build it from the repository. It needs Node.js 22.18 or later, or any current browser.

It comes with one WebAssembly module, sxml.wasm, which has everything the sxml command has but EXPath File: XML parsing, XPath 4.0 with the whole function library, XQuery 4.0, XSLT 4.0, XSD 1.1, Full Text, HTML, Invisible XML, EXPath Binary, collations, time zones, localization and serialization. load() loads it from beside the package's JavaScript; load({ wasm }) loads it from elsewhere.

Processors and values

As in C, everything starts with a processor. parse makes a document, compile compiles an expression once, and evaluate runs it with a context and variables. The examples read the Vane draft of Finding Things with XPath and the traces of Project: Comparing Traces Before and After a Change; the example directory links to both files.

hello.mjs:

import { readFileSync } from 'node:fs';
import { load } from '@tinyactors/sxml';

// Load the module, which has every language and feature.
const sxml = await load();
const processor = sxml.createProcessor();

const draft = processor.parse(readFileSync('vane.xml', 'utf8'));

// Compile once...
const anchors = processor.compile('//t[bcp14 = $keyword]/@anchor ! string()', {
  variables: ['keyword'],
});

// ...and evaluate as often as you like.
for (const keyword of ['MUST', 'MAY']) {
  const found = anchors.evaluate({ context: draft, variables: { keyword } });
  console.log(`${keyword}:`, found.toString().split('\n'));
}

// Results serialize by any method; maps and arrays become JSON.
const summary = processor.evaluate(
  'array { for $keyword in distinct-values(//bcp14) return { "keyword": $keyword, "count": count(//bcp14[. = $keyword]) } }',
  { context: draft });
console.log(summary.serialize({ method: 'json' }));

processor.dispose();

$ node hello.mjs
MUST: [ '"req-1"', '"req-2"', '"req-4"', '"req-6"', '"req-7"' ]
MAY: [ '"req-9"', '"req-11"' ]
[{"keyword":"MUST","count":5},{"keyword":"SHOULD","count":3},{"keyword":"MUST NOT","count":1},{"keyword":"MAY","count":2},{"keyword":"SHOULD NOT","count":1}]

toString() writes a value by the adaptive method, as the sxml command does, and serialize takes the parameters of fn:serialize.

XQuery, XSLT and XSD

compileQuery, compileStylesheet and compileSchema compile the other languages, again once, to run many times. JavaScript values become XPath values: strings are strings, numbers are doubles, objects are maps and arrays are arrays. So a query can take a whole OTLP file, parsed by JSON.parse, as a variable:

languages.mjs:

import { readFileSync } from 'node:fs';
import { load } from '@tinyactors/sxml';

// The one module has XQuery, XSLT and XSD too.
const sxml = await load();
const processor = sxml.createProcessor();

// XQuery over OTLP JSON: JavaScript objects and arrays become maps and
// arrays, so the parsed file can be a variable.
const query = processor.compileQuery(`
  declare option output:method "json";
  declare variable $request external;
  array {
    for member $resource in $request?resourceSpans
    let $service := $resource?resource?attributes?*[?key = "service.name"]?value?stringValue
    return { "service": $service, "spans": count($resource?scopeSpans?*?spans?*) }
  }`);
const request = JSON.parse(readFileSync('before.json', 'utf8'));
console.log(query.run({ variables: { request } }));

// XSLT: compile a stylesheet once, transform many documents.
const draft = processor.parse(readFileSync('vane.xml', 'utf8'));
const stylesheet = processor.compileStylesheet(`
<xsl:stylesheet version="4.0" expand-text="yes"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="text"/>
  <xsl:template match="/"><xsl:apply-templates select="//section[t/bcp14 = 'MUST']"/></xsl:template>
  <xsl:template match="section">{name}: {count(t[bcp14 = 'MUST'])} MUST&#10;</xsl:template>
</xsl:stylesheet>`);
process.stdout.write(stylesheet.transform({ source: draft }).output);

// XSD: an invalid document is a result, not an exception.
const schema = processor.compileSchema(`
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="bcp14">
    <xs:simpleType>
      <xs:restriction base="xs:token">
        <xs:enumeration value="MUST"/><xs:enumeration value="SHOULD"/><xs:enumeration value="MAY"/>
      </xs:restriction>
    </xs:simpleType>
  </xs:element>
</xs:schema>`);
const report = schema.validate('<bcp14>MIGHT</bcp14>');
console.log(report.validity, report.reports.map((each) => each.code));

processor.dispose();

$ node languages.mjs
[{"service":"storefront","spans":1},{"service":"vane","spans":12},{"service":"vane-console","spans":1}]
Flag States: 2 MUST
Rollout: 1 MUST
Status: 1 MUST
Pause: 1 MUST
invalid [ 'cvc-enumeration-valid' ]

The Vane service recorded twelve of the fourteen spans; the backend SDK in the storefront and the console recorded one each, their calls to it.

transform also takes an initial mode or template, stylesheet parameters, and returns the result documents of xsl:result-document. An updating query hands each changed tree, and each document fn:put stores, to callbacks you give it.

Resources and errors

A processor's resolver answers every request for a resource. It gets the kind of resource and its absolute URI, and returns a string, bytes, or undefined when there is no such resource. json-doc and unparsed-text ask for the kind text. In a browser, the resolver can answer from fetch results you loaded before, from IndexedDB, or from memory. Failures throw an SxmlError with the error code and location:

resources.mjs:

import { readFileSync } from 'node:fs';
import { load, SxmlError } from '@tinyactors/sxml';

const sxml = await load();

// The resolver is the only way an expression reaches anything outside.
// This one serves one trace file, read once, under a URI of our choice.
const files = new Map([
  ['https://traces.example.org/staging/before.json', readFileSync('before.json', 'utf8')],
]);
const processor = sxml.createProcessor({
  resolver: ({ kind, absoluteUri }) => (kind === 'text' ? files.get(absoluteUri) : undefined),
});

const services = processor.evaluate(
  'json-doc("before.json")?resourceSpans?*?resource?attributes?*[?key = "service.name"]?value?stringValue',
  { baseUri: 'https://traces.example.org/staging/' });
console.log(services.toString().split('\n'));

// Errors carry the code, the description and the location.
try {
  processor.evaluate('json-doc("after.json")', { baseUri: 'https://traces.example.org/staging/' });
} catch (error) {
  if (!(error instanceof SxmlError)) throw error;
  console.log(error.code, error.lineNumber, error.columnNumber);
}

processor.dispose();

$ node resources.mjs
[ '"storefront"', '"vane"', '"vane-console"' ]
err:FOUT1170 1 1

Languages from tree-sitter's WebAssembly files

sxml:parse-code (see Source Code as XML) knows a dozen languages. tree-sitter has grammars for hundreds more, and tree-sitter build --wasm compiles one to a WebAssembly language file, tree-sitter-rust.wasm for Rust; grammar repositories often ship that file with their releases. sxml.loadCodeLanguage links such a file into the running sxml.wasm, and createProcessor({ codeLanguages }) gives the languages to a processor:

const sxml = await load();
const rust = await sxml.loadCodeLanguage({
  wasm: fetch('/grammars/tree-sitter-rust.wasm'),
  highlights: await (await fetch('/grammars/rust/highlights.scm')).text(),
});
const processor = sxml.createProcessor({ codeLanguages: [rust] });
processor.evaluate('sxml:parse-code("fn main() {}", "rust")//function_item/identifier ! string()');

• wasm is the file, as for load({ wasm }): a URL or path, the bytes, a WebAssembly.Module or a fetch() response.
• highlights, locals and injections are the texts of the grammar's queries/highlights.scm, locals.scm and injections.scm, and injectionRegex is the injection-regex of its tree-sitter.json. The package reads no files of its own: in Node.js read them with readFile, in a browser fetch them. Without queries, the language parses but has no highlights.
• name is the language's name in sxml:parse-code. By default it is the NAME of the file's function tree_sitter_NAME. A language with the name of a built-in one replaces it in the processors it is given to.
• The grammar must be generated for tree-sitter's language ABI 13 to 15 (tree-sitter 0.20 and later). A file that is not a tree-sitter language file, one that needs other WebAssembly libraries, or one whose external scanner calls C functions outside tree-sitter's list for language files, is refused, and the error says why.
• A loaded language stays in memory as long as its sxml; it can be given to any number of processors.

To try it without a network, compile the JSON grammar from sxml's own sources as tree-sitter build --wasm does, with the WASI SDK that also builds sxml.wasm:

code-language.mjs:

import { readFile } from 'node:fs/promises';
import { load } from '@tinyactors/sxml';

const sxml = await load();
const wasm = await readFile('tree-sitter-json.wasm');

// Link the language file into sxml.wasm, with a highlights query of our own.
const json = await sxml.loadCodeLanguage({
  name: 'my-json',
  wasm,
  highlights: '(number) @number\n[(true) (false)] @boolean\n',
});
console.log(json.name, json.abiVersion);

// A processor knows the languages it is given, beside the built-in ones.
const processor = sxml.createProcessor({ codeLanguages: [json] });
const tree = processor.evaluate('sxml:parse-code($code, "my-json")', {
  variables: { code: '[1, true]' },
});
console.log(tree.serialize({ method: 'xml' }));

// A query that does not fit the grammar is an error.
try {
  await sxml.loadCodeLanguage({ wasm, highlights: '(float) @number' });
} catch (error) {
  console.log(error.message);
}

processor.dispose();

$ "$WASI_SDK_PREFIX/bin/clang" --target=wasm32-wasip1 -fPIC -shared -Os -nostdlib \
    -fvisibility=hidden -Wl,--no-entry -Wl,--allow-undefined -Wl,--export=tree_sitter_json \
    -o tree-sitter-json.wasm "$SXML/third_party/tree-sitter-grammars/json/src/parser.c"
$ node code-language.mjs
my-json 14
<document xmlns:sxml="urn:sxml"><array>[<number highlight="number">1</number>, <true highlight="boolean">true</true>]</array></document>
sxml: the highlights query does not fit the grammar at byte 1

Large outputs and inputs

The methods that return a string hold the whole output twice: once in the module's memory, which never shrinks again, and once as the string. For a large output, give a sink instead. value.serializeTo(sink), query.runTo(sink) and transform({ output }) write the output a chunk at a time, 64 KiB by default (createProcessor({ outputChunkByteCount })). A sink is a function that gets each chunk as a Uint8Array, or an object with write, and optionally close and abort. Each chunk is copied out of the module, which reuses its buffer for the next one. transform({ resultDocumentOutput }) chooses a sink for each xsl:result-document by its URI.

On the input side, a resolver may answer an XML document with an iterable of chunks, such as a generator. sxml reads it as it parses. When a stylesheet's mode is streamable, transform({ streamedSource }) reads the principal source this way and serializes the result as it is made, so neither the source nor the result is ever whole in memory. Here a generator makes an archive of 200,000 traces in the normal form of Spans as a Tree, and a streamable mode picks the slow requests out of it, as in Streaming Large Trace Files:

streaming.mjs:

import { closeSync, openSync, writeSync } from 'node:fs';
import { load, textSink } from '@tinyactors/sxml';

const sxml = await load();

// A large trace archive that never exists whole: a generator of chunks
// in the normal form of the traces project, read as it is parsed.
function* archive(count) {
  yield '<traces>';
  for (let start = 1; start <= count; start += 1000) {
    let text = '';
    for (let trace = start; trace < start + 1000 && trace <= count; trace += 1) {
      const query = 5 + ((trace * 7919) % 61);
      text += `<trace id="${trace}" route="GET /api/getFlags">`
        + `<span service="storefront" kind="client" duration="${query + 12}" peer="vane">`
        + `<span service="vane" kind="server" duration="${query + 10}">`
        + `<span service="vane" kind="client" duration="${query}" peer="postgresql" db="postgresql"/>`
        + '</span></span></trace>';
    }
    yield text;
  }
  yield '</traces>';
}
const processor = sxml.createProcessor({
  resolver: ({ absoluteUri }) => (absoluteUri === 'urn:archive' ? archive(200_000) : undefined),
});

// A streamable mode reads the source as it arrives; the output goes to a
// file a chunk at a time, written as each chunk is made.
const stylesheet = processor.compileStylesheet(`
<xsl:stylesheet version="3.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:mode streamable="yes"/>
  <xsl:output omit-xml-declaration="yes"/>
  <xsl:template match="/traces">
    <slow>
      <xsl:for-each select="trace/span[@duration > 70]">
        <trace id="{../@id}">≥ 70 ms</trace>
      </xsl:for-each>
    </slow>
  </xsl:template>
</xsl:stylesheet>`);
const file = openSync('slow.xml', 'w');
let byteCount = 0;
stylesheet.transform({
  streamedSource: 'urn:archive',
  output: (chunk) => {
    byteCount += chunk.byteLength;
    writeSync(file, chunk);
  },
});
closeSync(file);
console.log(byteCount, 'bytes;', (sxml.exports.memory.buffer.byteLength / 2 ** 20).toFixed(0), 'MiB of module memory');

// Text a chunk at a time: a character split between two chunks is decoded
// once both arrived.
const sizes = sxml.createProcessor({ outputChunkByteCount: 4 });
const value = sizes.evaluate('"≥ 70 ms: 12 €"');
const pieces = [];
value.serializeTo(textSink((text) => pieces.push(text)), { method: 'text' });
console.log(pieces);

$ node streaming.mjs
813465 bytes; 10 MiB of module memory
[ '≥ ', '70 m', 's: 1', '2 ', '€' ]

The evaluation runs synchronously inside WebAssembly, so a sink cannot make it wait. A sink that consumes each chunk at once, like writeSync to a file, a hash, or a parser fed by a decoder, keeps memory bounded by one chunk. Sinks that queue chunks hold them in JavaScript memory until the call returns. That includes writableStreamSink for a WHATWG WritableStream, nodeStreamSink for a Node.js stream and outputReadableStream. They still save the copy in the module's memory and the string. textSink decodes UTF-8 in TextDecoder's stream mode, so a character split between two chunks comes out whole.

A long stream takes many evaluation steps. Every evaluation stops after a hundred million of them ("The step limit" in "Performance and Memory"), but a streamed document grants 500 more for each element it reads, so a streamed transformation of a long input runs within the bound, while a loop that reads nothing more still stops. For other work that needs more, or to keep untrusted queries shorter, set the bound with createProcessor({ maximumEvaluationStepCount }).

Memory

Processors, compiled expressions and values hold memory inside the WebAssembly module. Call dispose() when you are done with one; disposing a processor frees everything it made. Whatever is not disposed is freed when JavaScript's garbage collector collects it.

The package's README, bindings/javascript/README.md, describes the whole API.