scanner: fix Position returned by Scan, Pos

The implementation of the position computation
was surprisingly broken. Implemented fixes and
added extra test cases.

There is a slight interface change: Calling
Pos() returns the current position; but if
called before Scan() that position may not
be the position of the next token returned
by Scan() (depending on the scan settings
and the source text) - this in contrast to
the original comment.

However, after calling Scan(), the Scanner's
Position field reports the position of the
scanned token, as before.

Fixes #1327.

R=rsc
CC=golang-dev
https://golang.org/cl/3972047

0 files changed, 0 insertions, 0 deletions