archive/tar: add support for long binary strings in GNU format

The GNU tar format defines the following type flags:
	TypeGNULongName = 'L' // Next file has a long name
	TypeGNULongLink = 'K' // Next file symlinks to a file w/ a long name

Anytime a string exceeds the field dedicated to store it, the GNU format
permits a fake "file" to be prepended where that file entry has a Typeflag
of 'L' or 'K' and the contents of the file is a NUL-terminated string.

Contrary to previous TODO comments,
the GNU format supports arbitrary strings (without NUL) rather UTF-8 strings.
The manual says the following:
<<<
The name, linkname, magic, uname, and gname are
null-terminated character strings
>>>
<<<
All characters in header blocks are represented
by using 8-bit characters in the local variant of ASCII.
>>>

From this description, we gather the following:
* We must forbid NULs in any GNU strings
* Any 8-bit value (other than NUL) is permitted

Since the modern world has moved to UTF-8, it is really difficult to
determine what a "local variant of ASCII" means. For this reason,
we treat strings as just an arbitrary binary string (without NUL)
and leave it to the user to determine the encoding of this string.
(Practically, it seems that UTF-8 is the typical encoding used
in GNU archives seen in the wild).

The implementation of GNU tar seems to confirm this interpretation
of the manual where it permits any arbitrary binary string to exist
within these fields so long as they do not contain the NUL character.

 $ touch `echo -e "not\x80\x81\x82\x83utf8"`
 $ gnutar -H gnu --tar -cvf gnu-not-utf8.tar $(echo -e "not\x80\x81\x82\x83utf8")

The fact that we permit arbitrary binary in GNU strings goes
hand-in-hand with the fact that GNU also permits a "base-256" encoding
of numeric fields, which is effectively two-complement binary.

Change-Id: Ic037ec6bed306d07d1312f0058594bd9b64d9880
Reviewed-on: https://go-review.googlesource.com/55573
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>

1 files changed, 35 insertions, 0 deletions

diff --git a/src/archive/tar/reader_test.go b/src/archive/tar/reader_test.go
index 79d271717c..42d4ab7e14 100644
--- a/src/archive/tar/reader_test.go
+++ b/src/archive/tar/reader_test.go
@@ -351,6 +351,41 @@ func TestReader(t *testing.T) {
 			Gname:    "dsnet",
 		}},
 	}, {
+		// This archive was generated by Writer but is readable by both
+		// GNU and BSD tar utilities.
+		// The archive generated by GNU is nearly byte-for-byte identical
+		// to the Go version except the Go version sets a negative Devminor
+		// just to force the GNU format.
+		file: "testdata/gnu-utf8.tar",
+		headers: []*Header{{
+			Name: "☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹☺☻☹",
+			Mode: 0644,
+			Uid:  1000, Gid: 1000,
+			ModTime:  time.Unix(0, 0),
+			Typeflag: '0',
+			Uname:    "☺",
+			Gname:    "⚹",
+			Devminor: -1,
+		}},
+	}, {
+		// This archive was generated by Writer but is readable by both
+		// GNU and BSD tar utilities.
+		// The archive generated by GNU is nearly byte-for-byte identical
+		// to the Go version except the Go version sets a negative Devminor
+		// just to force the GNU format.
+		file: "testdata/gnu-not-utf8.tar",
+		headers: []*Header{{
+			Name:     "hi\x80\x81\x82\x83bye",
+			Mode:     0644,
+			Uid:      1000,
+			Gid:      1000,
+			ModTime:  time.Unix(0, 0),
+			Typeflag: '0',
+			Uname:    "rawr",
+			Gname:    "dsnet",
+			Devminor: -1,
+		}},
+	}, {
 		// BSD tar v3.1.2 and GNU tar v1.27.1 both rejects PAX records
 		// with NULs in the key.
 		file: "testdata/pax-nul-xattrs.tar",