RUSTSEC-2026-0311 Stack overflow on deeply nested LaTeX input

The parser and layout engine in affected versions of latex-rust recurse once per level of nesting with no limit. Deeply nested input, such as \frac{1}{…}, braces, \sqrt{…}, \left(, or superscripts nested hundreds or thousands of levels deep, exhausts the call stack.

A stack overflow in Rust aborts the process. It is not a panic, so it cannot be caught with catch_unwind. A program that renders LaTeX from untrusted input can therefore be terminated by a short string. With a release build on an 8 MiB stack, 700 nested \frac{1}{…} (about 7 KB of input) abort version 1.0.4, and 10,000 nested braces (about 20 KB) abort it as well. The threshold is lower on smaller stacks and in debug builds; on a 2 MiB thread in a debug build, about 35 nested fractions are enough.

let mut src = String::from("1");
for _ in 0..700 {
    src = format!("\\frac{{1}}{{{src}}}");
}
let _ = latex_rust::parse(&src); // stack overflow, process aborts

The fixed versions count nesting depth in the parser and in layout and refuse input that nests more than 32 levels. parse, parse_with_colors, and the latex_to_* functions return ParseError::Malformed("input nests deeper than 32 levels"), and layout returns Error::Unsupported { what: "tree nests deeper than 32 levels" } for a tree built by hand. The count is of parser recursion levels, and a braced argument costs two, so the limit admits 15 nested \frac, \sqrt, or x^{…}, and 31 nested groups, \left…\right pairs, or environments. Input past the limit is refused within about 100 KiB of stack in an optimised build.

Version 1.0.5 fixes this with no API changes and is a drop-in replacement for 1.0.4; upgrade with cargo update -p latex-rust. It also includes the 2.0.0 rendering fixes, so rendered output differs from 1.0.4. Versions 2.0.0 and 2.0.1 contain the same fix, with the same limit and messages, together with breaking API changes relative to 1.x, and also let callers change the limit with ParseOptions and layout_with_max_depth.