class PDF::Reader::Buffer

A string tokeniser that recognises PDF grammar. When passed an IO stream or a string, repeated calls to token() will return the next token from the source.

This is very low level, and getting the raw tokens is not very useful in itself.

This will usually be used in conjunction with PDF:Reader::Parser, which converts the raw tokens into objects we can work with (strings, ints, arrays, etc)

Constants

CR
CRLF
DIGITS_ONLY

must match whole tokens

FWD_SLASH
ID
LEFT_PAREN

some strings for comparissons. Declaring them here avoids creating new strings that need GC over and over

LESS_THAN
LF
NULL_BYTE
STREAM
TOKEN_DELIMITER
TOKEN_WHITESPACE
TRAILING_BYTECOUNT

Quite a few PDFs have trailing junk. This can be several k of nuls in some cases Allow for this here

WHITE_SPACE

Attributes

pos[R]

Public Class Methods

new(io, opts = {}) click to toggle source

Creates a new buffer.

Params:

io - an IO stream (usually a StringIO) with the raw data to tokenise

options:

:seek - a byte offset to seek to before starting to tokenise
:content_stream - set to true if buffer will be tokenising a
                  content stream. Defaults to false
# File lib/pdf/reader/buffer.rb, line 79
def initialize(io, opts = {})
  @io = io
  @tokens = []
  @in_content_stream = opts[:content_stream]

  @io.seek(opts[:seek]) if opts[:seek]
  @pos = @io.pos
end

Public Instance Methods

empty?() click to toggle source

return true if there are no more tokens left

# File lib/pdf/reader/buffer.rb, line 90
def empty?
  prepare_tokens if @tokens.size < 3

  @tokens.empty?
end
find_first_xref_offset() click to toggle source

return the byte offset where the first XRef table in th source can be found.

# File lib/pdf/reader/buffer.rb, line 144
def find_first_xref_offset
  check_size_is_non_zero
  @io.seek(-TRAILING_BYTECOUNT, IO::SEEK_END) rescue @io.seek(0)
  data = @io.read(TRAILING_BYTECOUNT)

  raise MalformedPDFError, "PDF does not contain EOF marker" if data.nil?

  # the PDF 1.7 spec (section #3.4) says that EOL markers can be either \r, \n, or both.
  lines = data.split(/[\n\r]+/).reverse
  eof_index = lines.index { |l| l.strip[/^%%EOF/] }

  raise MalformedPDFError, "PDF does not contain EOF marker" if eof_index.nil?
  raise MalformedPDFError, "PDF EOF marker does not follow offset" if eof_index >= lines.size-1
  offset = lines[eof_index+1].to_i

  # a byte offset < 0 doesn't make much sense. This is unlikely to happen, but in theory some
  # corrupted PDFs might have a line that looks like a negative int preceding the `%%EOF`
  raise MalformedPDFError, "invalid xref offset" if offset < 0
  offset
end
read(bytes, opts = {}) click to toggle source

return raw bytes from the underlying IO stream.

bytes - the number of bytes to read

options:

:skip_eol - if true, the IO stream is advanced past a CRLF, CR or LF
            that is sitting under the io cursor.
Note:
Skipping a bare CR is not spec-compliant.
This is because the data may start with LF.
However we check for CRLF first, so the ambiguity is avoided.
# File lib/pdf/reader/buffer.rb, line 108
def read(bytes, opts = {})
  reset_pos

  if opts[:skip_eol]
    @io.seek(-1, IO::SEEK_CUR)
    str = @io.read(2)
    if str.nil?
      return nil
    elsif str == CRLF # This MUST be done before checking for CR alone
      # do nothing
    elsif str[0, 1] == LF || str[0, 1] == CR # LF or CR alone
      @io.seek(-1, IO::SEEK_CUR)
    else
      @io.seek(-2, IO::SEEK_CUR)
    end
  end

  bytes = @io.read(bytes)
  save_pos
  bytes
end
token() click to toggle source

return the next token from the source. Returns a string if a token is found, nil if there are no tokens left.

# File lib/pdf/reader/buffer.rb, line 133
def token
  reset_pos
  prepare_tokens if @tokens.size < 3
  merge_indirect_reference
  prepare_tokens if @tokens.size < 3

  @tokens.shift
end

Private Instance Methods

check_size_is_non_zero() click to toggle source
# File lib/pdf/reader/buffer.rb, line 167
def check_size_is_non_zero
  @io.seek(-1, IO::SEEK_END)
  @io.seek(0)
rescue Errno::EINVAL
  raise MalformedPDFError, "PDF file is empty"
end
in_content_stream?() click to toggle source

Returns true if this buffer is parsing a content stream

# File lib/pdf/reader/buffer.rb, line 176
def in_content_stream?
  @in_content_stream ? true : false
end
merge_indirect_reference() click to toggle source

detect a series of 3 tokens that make up an indirect object. If we find them, replace the tokens with a PDF::Reader::Reference instance.

Merging them into a single string was another option, but that would mean code further up the stack would need to check every token to see if it looks like an indirect object. For optimisation reasons, I’d rather avoid that extra check.

It’s incredibly likely that the next 3 tokens in the buffer are NOT an indirect reference, so test for that case first and avoid the relatively expensive regexp checks if possible.

# File lib/pdf/reader/buffer.rb, line 239
def merge_indirect_reference
  return if @tokens.size < 3
  return if @tokens[2] != "R"

  token_one = @tokens[0]
  token_two = @tokens[1]
  if token_one.is_a?(String) && token_two.is_a?(String) && token_one.match(DIGITS_ONLY) && token_two.match(DIGITS_ONLY)
    @tokens[0] = PDF::Reader::Reference.new(token_one.to_i, token_two.to_i)
    @tokens.delete_at(2)
    @tokens.delete_at(1)
  end
end
peek_byte() click to toggle source

peek at the next character in the io stream, leaving the stream position untouched

# File lib/pdf/reader/buffer.rb, line 439
def peek_byte
  byte = @io.getbyte
  @io.seek(-1, IO::SEEK_CUR) if byte
  byte
end
prepare_hex_token() click to toggle source

if we’re currently inside a hex string, read hex nibbles until we find a closing >

# File lib/pdf/reader/buffer.rb, line 302
def prepare_hex_token
  finished = :false
  str = "".dup

  until finished == :true
    byte = @io.getbyte
    if byte.nil?
      finished = :true # unbalanced params
    elsif (48..57).include?(byte) || (65..90).include?(byte) || (97..122).include?(byte)
      str << byte
    elsif byte <= 32
      # ignore it
    else
      @tokens << str if str.size > 0
      @tokens << ">" if byte != 0x3E # '>'
      @tokens << byte.chr
      finished = :true
    end
  end
end
prepare_inline_token() click to toggle source

Extract data between ID and EI If the EI follows white-space the space is dropped from the data The EI must followed by white-space or end of buffer This is to reduce the chance of accidentally matching an embedded EI

# File lib/pdf/reader/buffer.rb, line 256
def prepare_inline_token
  idstart = @io.pos
  prevchr = ''
  eisize = 0 # how many chars in the end marker
  seeking = 'E' # what are we looking for now?
  loop do
    chr = @io.read(1)
    break if chr.nil?
    case seeking
    when 'E'
      if chr == 'E'
        seeking = 'I'
        if WHITE_SPACE.include? prevchr
          eisize = 3 # include whitespace in delimiter, i.e. drop from data
        else # assume the EI immediately follows the data
          eisize = 2 # leave prevchr in data
        end
      end
    when 'I'
      if chr == 'I'
        seeking = ''
      else
        seeking = 'E'
      end
    when ''
      if WHITE_SPACE.include? chr
        eisize += 1 # Drop trailer
        break
      else
        seeking = 'E'
      end
    end
    prevchr = chr.is_a?(String) ? chr : ''
  end
  unless seeking == ''
    raise MalformedPDFError, "EI terminator not found"
  end
  eiend = @io.pos
  @io.seek(idstart, IO::SEEK_SET)
  str = @io.read(eiend - eisize - idstart) # get the ID content
  @tokens << str.freeze if str
end
prepare_literal_token() click to toggle source

if we’re currently inside a literal string we more or less just read bytes until we find the closing ) delimiter. Lots of bytes that would otherwise indicate the start of a new token in regular mode are left untouched when inside a literal string.

The entire literal string will be returned as a single token. It will need further processing to fix things like escaped new lines, but that’s someone else’s problem.

# File lib/pdf/reader/buffer.rb, line 332
def prepare_literal_token
  str = "".dup
  count = 1

  while count > 0
    byte = @io.getbyte
    if byte.nil?
      count = 0 # unbalanced params
    elsif byte == 0x5C
      str << byte << @io.getbyte
    elsif byte == 0x28 # "("
      str << "("
      count += 1
    elsif byte == 0x29 # ")"
      count -= 1
      str << ")" unless count == 0
    else
      str << byte unless count == 0
    end
  end

  @tokens << str if str.size > 0
  @tokens << ")"
end
prepare_regular_token() click to toggle source

Extract the next regular token and stock it in our buffer, ready to be returned.

What each byte means is complex, check out section “3.1.1 Character Set” of the 1.7 spec to read up on it.

# File lib/pdf/reader/buffer.rb, line 362
def prepare_regular_token
  tok = "".dup

  loop do
    byte = @io.getbyte

    case byte
    when nil
      break
    when 0x25
      # comment, ignore everything until the next EOL char
      loop do
        commentbyte = @io.getbyte
        break if commentbyte.nil? || commentbyte == 0x0A || commentbyte == 0x0D
      end
    when *TOKEN_WHITESPACE
      # white space, token finished
      @tokens << tok if tok.size > 0

      #If the token was empty, chomp the rest of the whitespace too
      while TOKEN_WHITESPACE.include?(peek_byte) && tok.size == 0
        @io.getbyte
      end
      tok = "".dup
      break
    when 0x3C
      # opening delimiter '<', start of new token
      @tokens << tok if tok.size > 0
      if peek_byte == 0x3C # check if token is actually '<<'
        @io.getbyte
        @tokens << "<<"
      else
        @tokens << "<"
      end
      tok = "".dup
      break
    when 0x3E
      # closing delimiter '>', start of new token
      @tokens << tok if tok.size > 0
      if peek_byte == 0x3E # check if token is actually '>>'
        @io.getbyte
        @tokens << ">>"
      else
        @tokens << ">"
      end
      tok = "".dup
      break
    when 0x28, 0x5B, 0x7B
      # opening delimiter, start of new token
      @tokens << tok if tok.size > 0
      @tokens << byte.chr
      tok = "".dup
      break
    when 0x29, 0x5D, 0x7D
      # closing delimiter
      @tokens << tok if tok.size > 0
      @tokens << byte.chr
      tok = "".dup
      break
    when 0x2F
      # PDF name, start of new token
      @tokens << tok if tok.size > 0
      @tokens << byte.chr
      @tokens << "" if byte == 0x2F && ([nil, 0x20, 0x0A] + TOKEN_DELIMITER).include?(peek_byte)
      tok = "".dup
      break
    else
      tok << byte
    end
  end

  @tokens << tok if tok.size > 0
end
prepare_tokens() click to toggle source

attempt to prime the buffer with the next few tokens.

# File lib/pdf/reader/buffer.rb, line 195
def prepare_tokens
  10.times do
    case state
    when :literal_string then prepare_literal_token
    when :hex_string     then prepare_hex_token
    when :regular        then prepare_regular_token
    when :inline         then prepare_inline_token
    end
  end

  save_pos
end
reset_pos() click to toggle source

Some bastard moved our IO stream cursor. Restore it.

# File lib/pdf/reader/buffer.rb, line 182
def reset_pos
  @io.seek(@pos) if @io.pos != @pos
end
save_pos() click to toggle source

save the current position of the source IO stream. If someone else (like another buffer) moves the cursor, we can then restore it.

# File lib/pdf/reader/buffer.rb, line 189
def save_pos
  @pos = @io.pos
end
state() click to toggle source

tokenising behaves slightly differently based on the current context. Determine the current context/state by examining the last token we found

# File lib/pdf/reader/buffer.rb, line 211
def state
  case @tokens.last
  when LEFT_PAREN then :literal_string
  when LESS_THAN then :hex_string
  when STREAM then :stream
  when ID
    if in_content_stream?  && @tokens[-2] != FWD_SLASH
      :inline
    else
      :regular
    end
  else
    :regular
  end
end