The history of luaTEX 2006–2009 / v 0.50 - Pragma ADE

do
end
local function nextbyte(f)
return f:read(1)
end
function io.bytes(f)
return nextbyte, f
end
f = io.open("somefile.dat")
for b in io.bytes(f) do
do_something(b)
end
f:close()
Of course in practice one will need to integrate this into one of the reader callback, but
the principle stays the same. In case you wonder if calling functions for each byte is fast
enough . . . it's more than fast enough for normal purposes, especially if we keep in mind
that other tasks like reading of, preparing of and dealing with fonts of processing token
lists take way more time. You can be sore that when half a second runtime is spent on
reading a le, processing may take minutes. If one wants to sqeeze more performance
out of this part, it's always an option to write special libraries for that, but this is beyond
standard LuaTEX. We found out that the speed of loading data from les in Lua is mostly
related to the small size of Lua's le buffer. Reading data stored in tables is extremely fast,
and even faster when precompiled into bytecode.
tables
When Taco and I were experimenting with the callbacks that intercept tokens and nodes,
we wondered what the impact would be on performance. Although in MkIV we allocate
quite some memory due to font handling, we were pretty sure that handling TEX's internal
lists also could have their impact. Data related to fonts is not always subjected to garbage
collection, simply because it's to be available permanently. List processing on the other
hand involves a lot of temporary allocated tables. During a run a real huge amount of tokens
passes the machinery. When digested, they become nodes. For testing we normally
use this document (with the name mk.tex) and at the time of writing this, it has some 48
pages.
This document is of moderately complexity, but not as complex as the documents that
I normally process; they have with lots of graphics, layers, structural elements, maybe a
bit of xml parsing, etc. Nevertheless, we're talking of some 24 million tokens entering the
60 How about performance