os: explicitly check for invalid FD in NewFile

CL 497075 refactored NewFile to unconditionally dereference the file
returned by newFile. However, newFile can return nil if passed a
negative FD, which now causes a crash.

Resolve this by moving the invalid check earlier in NewFile, which also
lets us avoid a useless fcntl syscall on a negative FD.

Since we convert to int to check sign, adjust newFile to take an int
rather than uintptr, which cleans up a lot of conversions.

Fixes #60406

Change-Id: I382a74e22f1cc01f7a2dcf1ff4efca6a79c4dd57
Reviewed-on: https://go-review.googlesource.com/c/go/+/497877
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Auto-Submit: Michael Pratt <mpratt@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>

1 files changed, 7 insertions, 0 deletions

diff --git a/src/os/os_windows_test.go b/src/os/os_windows_test.go
index fbc8cc1b9f..a0bfd991e3 100644
--- a/src/os/os_windows_test.go
+++ b/src/os/os_windows_test.go
@@ -1446,3 +1446,10 @@ func TestUTF16Alloc(t *testing.T) {
 		syscall.UTF16FromString("abc")
 	})
 }
+
+func TestNewFileInvalid(t *testing.T) {
+	t.Parallel()
+	if f := os.NewFile(uintptr(syscall.InvalidHandle), "invalid"); f != nil {
+		t.Errorf("NewFile(InvalidHandle) got %v want nil", f)
+	}
+}