bufio: allow Scanner to accept a user-provided buffer

Add Scanner.Buffer, which lets the user give a buffer to
the scanner and set the maximum token size.

We call it Buffer not SetBuffer for consistency with Split, which
perhaps should have been called SetSplit; too late regardless.

Both Buffer and Split panic if they are called after Scan. The
panic in Split is new, but the comment on the method already
said it needed to be called first, so we might as well add the
verification while we're doing it for Buffer.

This method allows precise user control of storage.

Fixes #11702.

Change-Id: I80e3d0e3830562fdabd4f7b08f322e1378248c39
Reviewed-on: https://go-review.googlesource.com/14599
Reviewed-by: Andrew Gerrand <adg@golang.org>
Reviewed-by: roger peppe <rogpeppe@gmail.com>

1 files changed, 16 insertions, 0 deletions

diff --git a/src/bufio/scan_test.go b/src/bufio/scan_test.go
index eea87cbf7b..ac65de9c44 100644
--- a/src/bufio/scan_test.go
+++ b/src/bufio/scan_test.go
@@ -522,3 +522,19 @@ func TestEmptyLinesOK(t *testing.T) {
 		t.Fatalf("stopped with %d left to process", c)
 	}
 }
+
+// Make sure we can read a huge token if a big enough buffer is provided.
+func TestHugeBuffer(t *testing.T) {
+	text := strings.Repeat("x", 2*MaxScanTokenSize)
+	s := NewScanner(strings.NewReader(text + "\n"))
+	s.Buffer(make([]byte, 100), 3*MaxScanTokenSize)
+	for s.Scan() {
+		token := s.Text()
+		if token != text {
+			t.Errorf("scan got incorrect token of length %d", len(token))
+		}
+	}
+	if s.Err() != nil {
+		t.Fatal("after scan:", s.Err())
+	}
+}