design/go-generate: copy from original go.dev/s/go1.4-generate

A step toward gathering design docs in one place.

When this is submitted I will update the go.dev/s link.

Change-Id: Ib34d955fdcfad9ecded5ed323922e8eef7b49d90
Reviewed-on: https://go-review.googlesource.com/c/proposal/+/417642
Reviewed-by: Rob Pike <r@golang.org>

1 files changed, 342 insertions, 0 deletions

diff --git a/design/go-generate.md b/design/go-generate.md
new file mode 100644
index 0000000..9129017
--- /dev/null
+++ b/design/go-generate.md
@@ -0,0 +1,342 @@
+# Go generate: A Proposal
+
+Author: Rob Pike
+
+Accepted in the Go 1.4 release.
+
+## Introduction
+
+The go build command automates the construction of Go programs but
+sometimes preliminary processing is required, processing that go build
+does not support.
+Motivating examples include:
+
+- yacc: generating .go files from yacc grammar (.y) files
+- protobufs: generating .pb.go files from protocol buffer definition (.proto) files
+- Unicode: generating tables from UnicodeData.txt
+- HTML: embedding .html files into Go source code
+- bindata: translating binary files such as JPEGs into byte arrays in Go source
+
+There are other processing steps one can imagine:
+
+- string methods: generating String() string methods for types used as enumerated constants
+- macros: generating customized implementations given generalized packages, such as sort.Ints from ints
+
+This proposal offers a design for smooth automation of such processing.
+
+## Non-goal
+
+It is not a goal of this proposal to build a generalized build system
+like the Unix make(1) utility.
+We deliberately avoid doing any dependency analysis.
+The tool does what is asked of it, nothing more.
+
+It is hoped, however, that it may replace many existing uses of
+make(1) in the Go repo at least.
+
+## Design
+
+There are two basic elements, a new subcommand for the go command,
+called go generate, and directives inside Go source files that control
+generation.
+
+When go generate runs, it scans Go source files looking for those
+directives, and for each one executes a generator that typically
+creates a new Go source file.
+The go generate tool also sets the build tag "generate" so that files
+may be examined by go generate but ignored during build.
+
+The usage is:
+
+```
+go generate [-run regexp] [file.go...|packagePath...]
+```
+
+(Plus the usual `-x`, `-n`, `-v` and `-tags` options.)
+If packages are named, each Go source file in each package is scanned
+for generator directives, and for each directive, the specified
+generator is run; if files are named, they must be Go source files and
+generation happens only for directives in those files.
+Given no arguments, generator processing is applied to the Go source
+files in the current directory.
+
+The `-run` flag takes a regular expression, analogous to that of the
+go test subcommand, that restricts generation to those directives
+whose command (see below) matches the regular expression.
+
+Generator directives may appear anywhere in the Go source file and are
+processed sequentially (no parallelism) in source order as presented
+to the tool.
+Each directive is a // comment beginning a line, with syntax
+
+```
+//go:generate command arg...
+```
+
+where command is the generator (such as `yacc`) to be run,
+corresponding to an executable file that can be run locally; it must
+either be in the shell path (`gofmt`) or fully qualified
+(`/usr/you/bin/mytool`) and is run in the package directory.
+
+The arguments are space-separated tokens (or double-quoted strings)
+passed to the generator as individual arguments when it is run.
+Shell-like variable expansion is available for any environment
+variables such as `$HOME`.
+Also, the special variable `$GOFILE` refers to the name of the file
+containing the directive.
+(We may need other special variables such as `$GOPACKAGE`.
+When the generator is run, these are also provided in the shell
+environment.)
+No other special processing, such as globbing, is provided.
+
+No further generators are run if any generator returns an error exit
+status.
+
+As an example, say we have a package `my/own/gopher` that includes a
+yacc grammar in file `gopher.y`.
+Inside `main.go` (not `gopher.y`) we place the directive
+
+```
+//go:generate yacc -o gopher.go gopher.y
+```
+
+(More about what `yacc` means in the next section.)
+Whenever we need to update the generated file, we give the shell
+command,
+
+```
+% go generate my/own/gopher
+```
+
+or, if we are already in the source directory,
+
+```
+% go generate
+```
+
+If we want to make sure that only the yacc generator is run, we
+execute
+
+```
+% go generate -run yacc
+```
+
+If we have fixed a bug in yacc and want to update all yacc-generated
+files in our tree, we can run
+
+```
+% go generate -run yacc all
+```
+
+The typical cycle for a package author developing software that uses
+`go generate` is
+
+```
+% edit …
+% go generate
+% go test
+```
+
+and once things are settled, the author commits the generated files to
+the source repository, so that they are available to clients that use
+go get:
+
+```
+% git add *.go
+% git commit
+```
+
+## Commands
+
+The yacc program is of course not the standard version, but is
+accessed from the command line by
+
+```
+go tool yacc args...
+```
+
+To make it easy to use tools like yacc that are not installed in
+$PATH, have complex access methods, or benefit from extra flags or
+other wrapping, there is a special directive that defines a shorthand
+for a command.
+It is a `go:generate` directive followed by the keyword/flag
+`-command` and which generator it defines; the rest of the line is
+substituted for the command name when the generator is run.
+Thus to define `yacc` as a generator command we access normally by
+running `go tool yacc`, we first write the directive
+
+```
+//go:generate -command yacc go tool yacc
+```
+
+and then all other generator directives using `yacc` that follow in
+that file (only) can be written as above:
+
+```
+//go:generate yacc -o gopher.go gopher.y
+```
+
+which will be translated to
+
+```
+go tool yacc -o gopher.go gopher.y
+```
+
+when run.
+
+## Discussion
+
+This design is unusual but is driven by several motivating principles.
+
+First, `go generate` is intended[^1] to be run by the author of a
+package, not the client of it.
+The author of the package generates the required Go files and includes
+them in the package; the client does a regular `go get` or `go
+build`.
+Generation through `go generate` is not part of the build, just a tool
+for package authors.
+This avoids complicating the dependency analysis done by Go build.
+
+[^1]: One can imagine scenarios where the author wishes the client to
+run the generator, but in such cases the author must guarantee that
+the client has the generator available.
+Regardless, `go get` will not automate the running of the processor,
+so further installation instructions will need to be provided by the
+author.
+
+Second, `go build` should never cause generation to happen
+automatically by the client of the package. Generators should run only
+when explicitly requested.
+
+Third, the author of the package should have great freedom in what
+generator to use (that is a key goal of the proposal), but the client
+might not have that processor available.
+As a simple example, if it is a shell script, it will not run on
+Windows.
+It is important that automated generation not break clients but be
+invisible to them, which is another reason it should be run only by
+the author of the package.
+
+Finally, it must fit well with the existing go command, which means it
+applies only to Go source files and packages.
+This is why the directives are in Go files but not, for example, in
+the .y file holding a yacc grammar.
+
+## Examples
+
+Here are some hypothetical worked examples.
+There are countless more possibilities.
+
+### String methods
+
+We wish to generate a String method for a named constant type.
+We write a tool, say `strmeth`, that reads a definition for a single
+constant type and values and prints a complete Go source file
+containing a method definition for that type.
+
+In our Go source file, `main.go`, we decorate each constant
+declaration like this (with some blank lines interposed so the
+generator directive does not appear in the doc comment):
+
+```Go
+//go:generate strmeth Day -o day_string.go $GOFILE
+
+// Day represents the day of the week
+type Day int
+const (
+	Sunday Day = iota
+	Monday
+	...
+)
+```
+
+The `strmeth` generator parses the Go source to find the definition of
+the `Day` type and its constants, and writes out a `String() string`
+method for that type.
+For the user, generation of the string method is trivial: just run `go
+generate`.
+
+### Yacc
+
+As outlined above, we define a custom command
+
+```
+//go:generate -command yacc go tool yacc
+```
+
+and then anywhere in main.go (say) we write
+
+```
+//go:generate yacc -o foo.go foo.y
+```
+
+### Protocol buffers
+
+The process is the same as with yacc.
+Inside `main.go`, we write, for each protocol buffer file we have, a
+line like
+
+```
+//go:generate protoc -go_out=. file.proto
+```
+
+Because of the way protoc works, we could generate multiple proto
+definitions into a single `.pb.go` file like this:
+
+```
+//go:generate protoc -go_out=. file1.proto file2.proto
+```
+
+Since no globbing is provided, one cannot say `*.proto`, but this is
+intentional, for simplicity and clarity of dependency.
+
+Caveat: The protoc program must be run at the root of the source tree;
+we would need to provide a `-cd` option to it or wrap it somehow.
+
+### Binary data
+
+A tool that converts binary files into byte arrays that can be
+compiled into Go binaries would work similarly.
+Again, in the Go source we write something like
+
+```
+//go:generate bindata -o jpegs.go pic1.jpg pic2.jpg pic3.jpg
+```
+
+This is also demonstrates another reason the annotations are in Go
+source: there is no easy way to inject them into binary files.
+
+### Sort
+
+One could imagine a variant sort implementation that allows one to
+specify concrete types that have custom sorters, just by automatic
+rewriting of macro-like sort definition.
+To do this, we write a `sort.go` file that contains a complete
+implementation of sort on an explicit but undefined type spelled, say,
+`TYPE`.
+In that file we provide a build tag so it is never compiled (`TYPE` is
+not defined, so it won't compile) but is processed by `go generate`:
+
+```
+// +build generate
+```
+
+Then we write an generator directive for each type for which we want a
+custom sort:
+
+```
+//go:generate rename TYPE=int
+//go:generate rename TYPE=strings
+```
+
+or perhaps
+
+```
+//go:generate rename TYPE=int TYPE=strings
+```
+
+The rename processor would be a simple wrapping of `gofmt -r`, perhaps
+written as a shell script.
+
+There are many more possibilities, and it is a goal of this proposal
+to encourage experimentation with pre-build-time code generation.