[Watson1978]

read_from_str() measured what was left of the input with strlen() every time it filled the 4KB buffer, so a String or StringIO document was walked once per 4KB and Ox.sax_parse was quadratic in its length. A 50 MB document took 8.5 s where the same document read from a File took 0.20 s. The remaining length is now carried in the buffer and taken once when the reader is set up, so the refill is a memcpy and nothing else. That one strlen() also keeps the rule the old code had, that the document ends at the first NUL, which RSTRING_LEN() would not.

Events are unchanged over a 3180 case corpus that puts every length either side of the 4092, 8184 and 12276 byte refill boundaries in text, in a name, in an attribute value, in a comment, in a CDATA section and in an instruction, read as a String, a StringIO, a StringIO with a non-zero pos and a pipe. The same corpus is clean under AddressSanitizer.

Ruby 4.0.6, gcc 16 -O3, best of three per row after a warm-up. Reading a File is untouched and is shown as the reference, which is also why the 0.2 MB row is the noise floor rather than a change.

   0.2 MB  String  0.0006 -> 0.0008 s   StringIO  0.0007 -> 0.0009 s   File  0.0007 -> 0.0009 s
   0.8 MB  String  0.0031 -> 0.0028 s   StringIO  0.0033 -> 0.0030 s   File  0.0030 -> 0.0030 s
   3.1 MB  String  0.0192 -> 0.0111 s   StringIO  0.0202 -> 0.0121 s   File  0.0122 -> 0.0125 s
  12.4 MB  String  0.4064 -> 0.0456 s   StringIO  0.4098 -> 0.0490 s   File  0.0494 -> 0.0508 s
  50.3 MB  String  8.5443 -> 0.1849 s   StringIO  8.7061 -> 0.1988 s   File  0.2036 -> 0.2055 s

Benchmark script

require 'benchmark'
require 'tmpdir'
require 'ox'
require 'stringio'

class Handler < Ox::Sax
  def start_element(name); end
  def end_element(name); end
  def attr(name, value); end
  def text(str); end
end

def gen(nrec)
  s = +"<?xml version=\"1.0\"?>\n<root>\n"
  nrec.times do |i|
    s << "  <record id=\"#{i}\" type=\"sample\" flag=\"true\">\n"
    s << "    <name>Item number #{i}</name>\n"
    s << "    <desc>Some moderately long description text for record #{i}.</desc>\n"
    s << "    <value>#{i * 37}</value>\n"
    s << "  </record>\n"
  end
  s << "</root>\n"
end

handler = Handler.new
path = File.join(Dir.tmpdir, 'ox_bench_sax.xml')

warm = gen(2_000)
10.times { Ox.sax_parse(handler, warm) }

[1_000, 4_000, 16_000, 64_000, 256_000].each do |n|
  doc = gen(n)
  File.write(path, doc)
  best = lambda { |&blk| 3.times.map { Benchmark.realtime(&blk) }.min }
  times = {}
  times['String'] = best.call { Ox.sax_parse(handler, doc) }
  times['StringIO'] = best.call { Ox.sax_parse(handler, StringIO.new(doc)) }
  times['File'] = best.call { File.open(path) { |f| Ox.sax_parse(handler, f) } }
  printf("%6.1f MB  %s\n", doc.bytesize / 1e6, times.map { |k, v| format('%s %8.4f s', k, v) }.join('  '))
end
File.unlink(path)

🤖 Generated with Claude Code

[ohler55]

Thank you