From 1f010d6bdf756129db13d1367c888aa4153f6d87 Mon Sep 17 00:00:00 2001 From: "brian m. carlson" Date: Mon, 20 Jan 2025 01:56:02 +0000 Subject: doc: use .adoc extension for AsciiDoc files We presently use the ".txt" extension for our AsciiDoc files. While not wrong, most editors do not associate this extension with AsciiDoc, meaning that contributors don't get automatic editor functionality that could be useful, such as syntax highlighting and prose linting. It is much more common to use the ".adoc" extension for AsciiDoc files, since this helps editors automatically detect files and also allows various forges to provide rich (HTML-like) rendering. Let's do that here, renaming all of the files and updating the includes where relevant. Adjust the various build scripts and makefiles to use the new extension as well. Note that this should not result in any user-visible changes to the documentation. Signed-off-by: brian m. carlson Signed-off-by: Junio C Hamano --- Documentation/technical/api-error-handling.adoc | 103 ++ Documentation/technical/api-error-handling.txt | 103 -- Documentation/technical/api-index-skel.adoc | 13 + Documentation/technical/api-index-skel.txt | 13 - Documentation/technical/api-index.sh | 8 +- Documentation/technical/api-merge.adoc | 36 + Documentation/technical/api-merge.txt | 36 - Documentation/technical/api-parse-options.adoc | 349 +++++ Documentation/technical/api-parse-options.txt | 349 ----- Documentation/technical/api-simple-ipc.adoc | 105 ++ Documentation/technical/api-simple-ipc.txt | 105 -- Documentation/technical/api-trace2.adoc | 1352 ++++++++++++++++++++ Documentation/technical/api-trace2.txt | 1352 -------------------- Documentation/technical/bitmap-format.adoc | 398 ++++++ Documentation/technical/bitmap-format.txt | 398 ------ Documentation/technical/build-systems.adoc | 224 ++++ Documentation/technical/build-systems.txt | 224 ---- Documentation/technical/bundle-uri.adoc | 572 +++++++++ Documentation/technical/bundle-uri.txt | 572 --------- Documentation/technical/commit-graph.adoc | 401 ++++++ Documentation/technical/commit-graph.txt | 401 ------ .../technical/directory-rename-detection.adoc | 118 ++ .../technical/directory-rename-detection.txt | 118 -- .../technical/hash-function-transition.adoc | 830 ++++++++++++ .../technical/hash-function-transition.txt | 830 ------------ .../technical/long-running-process-protocol.adoc | 50 + .../technical/long-running-process-protocol.txt | 50 - Documentation/technical/multi-pack-index.adoc | 203 +++ Documentation/technical/multi-pack-index.txt | 203 --- Documentation/technical/pack-heuristics.adoc | 460 +++++++ Documentation/technical/pack-heuristics.txt | 460 ------- Documentation/technical/packfile-uri.adoc | 82 ++ Documentation/technical/packfile-uri.txt | 82 -- Documentation/technical/parallel-checkout.adoc | 270 ++++ Documentation/technical/parallel-checkout.txt | 270 ---- Documentation/technical/partial-clone.adoc | 367 ++++++ Documentation/technical/partial-clone.txt | 367 ------ Documentation/technical/platform-support.adoc | 190 +++ Documentation/technical/platform-support.txt | 190 --- Documentation/technical/racy-git.adoc | 201 +++ Documentation/technical/racy-git.txt | 201 --- Documentation/technical/reftable.adoc | 1098 ++++++++++++++++ Documentation/technical/reftable.txt | 1098 ---------------- Documentation/technical/remembering-renames.adoc | 671 ++++++++++ Documentation/technical/remembering-renames.txt | 671 ---------- Documentation/technical/repository-version.adoc | 70 + Documentation/technical/repository-version.txt | 70 - Documentation/technical/rerere.adoc | 186 +++ Documentation/technical/rerere.txt | 186 --- Documentation/technical/scalar.adoc | 66 + Documentation/technical/scalar.txt | 66 - Documentation/technical/send-pack-pipeline.adoc | 63 + Documentation/technical/send-pack-pipeline.txt | 63 - Documentation/technical/shallow.adoc | 60 + Documentation/technical/shallow.txt | 60 - Documentation/technical/sparse-checkout.adoc | 1103 ++++++++++++++++ Documentation/technical/sparse-checkout.txt | 1103 ---------------- Documentation/technical/sparse-index.adoc | 208 +++ Documentation/technical/sparse-index.txt | 208 --- Documentation/technical/trivial-merge.adoc | 121 ++ Documentation/technical/trivial-merge.txt | 121 -- Documentation/technical/unit-tests.adoc | 242 ++++ Documentation/technical/unit-tests.txt | 242 ---- 63 files changed, 10216 insertions(+), 10216 deletions(-) create mode 100644 Documentation/technical/api-error-handling.adoc delete mode 100644 Documentation/technical/api-error-handling.txt create mode 100644 Documentation/technical/api-index-skel.adoc delete mode 100644 Documentation/technical/api-index-skel.txt create mode 100644 Documentation/technical/api-merge.adoc delete mode 100644 Documentation/technical/api-merge.txt create mode 100644 Documentation/technical/api-parse-options.adoc delete mode 100644 Documentation/technical/api-parse-options.txt create mode 100644 Documentation/technical/api-simple-ipc.adoc delete mode 100644 Documentation/technical/api-simple-ipc.txt create mode 100644 Documentation/technical/api-trace2.adoc delete mode 100644 Documentation/technical/api-trace2.txt create mode 100644 Documentation/technical/bitmap-format.adoc delete mode 100644 Documentation/technical/bitmap-format.txt create mode 100644 Documentation/technical/build-systems.adoc delete mode 100644 Documentation/technical/build-systems.txt create mode 100644 Documentation/technical/bundle-uri.adoc delete mode 100644 Documentation/technical/bundle-uri.txt create mode 100644 Documentation/technical/commit-graph.adoc delete mode 100644 Documentation/technical/commit-graph.txt create mode 100644 Documentation/technical/directory-rename-detection.adoc delete mode 100644 Documentation/technical/directory-rename-detection.txt create mode 100644 Documentation/technical/hash-function-transition.adoc delete mode 100644 Documentation/technical/hash-function-transition.txt create mode 100644 Documentation/technical/long-running-process-protocol.adoc delete mode 100644 Documentation/technical/long-running-process-protocol.txt create mode 100644 Documentation/technical/multi-pack-index.adoc delete mode 100644 Documentation/technical/multi-pack-index.txt create mode 100644 Documentation/technical/pack-heuristics.adoc delete mode 100644 Documentation/technical/pack-heuristics.txt create mode 100644 Documentation/technical/packfile-uri.adoc delete mode 100644 Documentation/technical/packfile-uri.txt create mode 100644 Documentation/technical/parallel-checkout.adoc delete mode 100644 Documentation/technical/parallel-checkout.txt create mode 100644 Documentation/technical/partial-clone.adoc delete mode 100644 Documentation/technical/partial-clone.txt create mode 100644 Documentation/technical/platform-support.adoc delete mode 100644 Documentation/technical/platform-support.txt create mode 100644 Documentation/technical/racy-git.adoc delete mode 100644 Documentation/technical/racy-git.txt create mode 100644 Documentation/technical/reftable.adoc delete mode 100644 Documentation/technical/reftable.txt create mode 100644 Documentation/technical/remembering-renames.adoc delete mode 100644 Documentation/technical/remembering-renames.txt create mode 100644 Documentation/technical/repository-version.adoc delete mode 100644 Documentation/technical/repository-version.txt create mode 100644 Documentation/technical/rerere.adoc delete mode 100644 Documentation/technical/rerere.txt create mode 100644 Documentation/technical/scalar.adoc delete mode 100644 Documentation/technical/scalar.txt create mode 100644 Documentation/technical/send-pack-pipeline.adoc delete mode 100644 Documentation/technical/send-pack-pipeline.txt create mode 100644 Documentation/technical/shallow.adoc delete mode 100644 Documentation/technical/shallow.txt create mode 100644 Documentation/technical/sparse-checkout.adoc delete mode 100644 Documentation/technical/sparse-checkout.txt create mode 100644 Documentation/technical/sparse-index.adoc delete mode 100644 Documentation/technical/sparse-index.txt create mode 100644 Documentation/technical/trivial-merge.adoc delete mode 100644 Documentation/technical/trivial-merge.txt create mode 100644 Documentation/technical/unit-tests.adoc delete mode 100644 Documentation/technical/unit-tests.txt (limited to 'Documentation/technical') diff --git a/Documentation/technical/api-error-handling.adoc b/Documentation/technical/api-error-handling.adoc new file mode 100644 index 0000000000..665c4960b4 --- /dev/null +++ b/Documentation/technical/api-error-handling.adoc @@ -0,0 +1,103 @@ +Error reporting in git +====================== + +`BUG`, `bug`, `die`, `usage`, `error`, and `warning` report errors of +various kinds. + +- `BUG` is for failed internal assertions that should never happen, + i.e. a bug in git itself. + +- `bug` (lower-case, not `BUG`) is supposed to be used like `BUG` but + prints a "BUG" message instead of calling `abort()`. ++ +A call to `bug()` will then result in a "real" call to the `BUG()` +function, either explicitly by invoking `BUG_if_bug()` after call(s) +to `bug()`, or implicitly at `exit()` time where we'll check if we +encountered any outstanding `bug()` invocations. ++ +If there were no prior calls to `bug()` before invoking `BUG_if_bug()` +the latter is a NOOP. The `BUG_if_bug()` function takes the same +arguments as `BUG()` itself. Calling `BUG_if_bug()` explicitly isn't +necessary, but ensures that we die as soon as possible. ++ +If you know you had prior calls to `bug()` then calling `BUG()` itself +is equivalent to calling `BUG_if_bug()`, the latter being a wrapper +calling `BUG()` if we've set a flag indicating that we've called +`bug()`. ++ +This is for the convenience of APIs who'd like to potentially report +more than one "bug", such as the optbug() validation in +parse-options.c. + +- `die` is for fatal application errors. It prints a message to + the user and exits with status 128. + +- `usage` is for errors in command line usage. After printing its + message, it exits with status 129. (See also `usage_with_options` + in the link:api-parse-options.html[parse-options API].) + +- `error` is for non-fatal library errors. It prints a message + to the user and returns -1 for convenience in signaling the error + to the caller. + +- `warning` is for reporting situations that probably should not + occur but which the user (and Git) can continue to work around + without running into too many problems. Like `error`, it + returns -1 after reporting the situation to the caller. + +These reports will be logged via the trace2 facility. See the "error" +event in link:api-trace2.html[trace2 API]. + +Customizable error handlers +--------------------------- + +The default behavior of `die` and `error` is to write a message to +stderr and then exit or return as appropriate. This behavior can be +overridden using `set_die_routine` and `set_error_routine`. For +example, "git daemon" uses set_die_routine to write the reason `die` +was called to syslog before exiting. + +Library errors +-------------- + +Functions return a negative integer on error. Details beyond that +vary from function to function: + +- Some functions return -1 for all errors. Others return a more + specific value depending on how the caller might want to react + to the error. + +- Some functions report the error to stderr with `error`, + while others leave that for the caller to do. + +- errno is not meaningful on return from most functions (except + for thin wrappers for system calls). + +Check the function's API documentation to be sure. + +Caller-handled errors +--------------------- + +An increasing number of functions take a parameter 'struct strbuf *err'. +On error, such functions append a message about what went wrong to the +'err' strbuf. The message is meant to be complete enough to be passed +to `die` or `error` as-is. For example: + + if (ref_transaction_commit(transaction, &err)) + die("%s", err.buf); + +The 'err' parameter will be untouched if no error occurred, so multiple +function calls can be chained: + + t = ref_transaction_begin(&err); + if (!t || + ref_transaction_update(t, "HEAD", ..., &err) || + ret_transaction_commit(t, &err)) + die("%s", err.buf); + +The 'err' parameter must be a pointer to a valid strbuf. To silence +a message, pass a strbuf that is explicitly ignored: + + if (thing_that_can_fail_in_an_ignorable_way(..., &err)) + /* This failure is okay. */ + strbuf_reset(&err); diff --git a/Documentation/technical/api-error-handling.txt b/Documentation/technical/api-error-handling.txt deleted file mode 100644 index 665c4960b4..0000000000 --- a/Documentation/technical/api-error-handling.txt +++ /dev/null @@ -1,103 +0,0 @@ -Error reporting in git -====================== - -`BUG`, `bug`, `die`, `usage`, `error`, and `warning` report errors of -various kinds. - -- `BUG` is for failed internal assertions that should never happen, - i.e. a bug in git itself. - -- `bug` (lower-case, not `BUG`) is supposed to be used like `BUG` but - prints a "BUG" message instead of calling `abort()`. -+ -A call to `bug()` will then result in a "real" call to the `BUG()` -function, either explicitly by invoking `BUG_if_bug()` after call(s) -to `bug()`, or implicitly at `exit()` time where we'll check if we -encountered any outstanding `bug()` invocations. -+ -If there were no prior calls to `bug()` before invoking `BUG_if_bug()` -the latter is a NOOP. The `BUG_if_bug()` function takes the same -arguments as `BUG()` itself. Calling `BUG_if_bug()` explicitly isn't -necessary, but ensures that we die as soon as possible. -+ -If you know you had prior calls to `bug()` then calling `BUG()` itself -is equivalent to calling `BUG_if_bug()`, the latter being a wrapper -calling `BUG()` if we've set a flag indicating that we've called -`bug()`. -+ -This is for the convenience of APIs who'd like to potentially report -more than one "bug", such as the optbug() validation in -parse-options.c. - -- `die` is for fatal application errors. It prints a message to - the user and exits with status 128. - -- `usage` is for errors in command line usage. After printing its - message, it exits with status 129. (See also `usage_with_options` - in the link:api-parse-options.html[parse-options API].) - -- `error` is for non-fatal library errors. It prints a message - to the user and returns -1 for convenience in signaling the error - to the caller. - -- `warning` is for reporting situations that probably should not - occur but which the user (and Git) can continue to work around - without running into too many problems. Like `error`, it - returns -1 after reporting the situation to the caller. - -These reports will be logged via the trace2 facility. See the "error" -event in link:api-trace2.html[trace2 API]. - -Customizable error handlers ---------------------------- - -The default behavior of `die` and `error` is to write a message to -stderr and then exit or return as appropriate. This behavior can be -overridden using `set_die_routine` and `set_error_routine`. For -example, "git daemon" uses set_die_routine to write the reason `die` -was called to syslog before exiting. - -Library errors --------------- - -Functions return a negative integer on error. Details beyond that -vary from function to function: - -- Some functions return -1 for all errors. Others return a more - specific value depending on how the caller might want to react - to the error. - -- Some functions report the error to stderr with `error`, - while others leave that for the caller to do. - -- errno is not meaningful on return from most functions (except - for thin wrappers for system calls). - -Check the function's API documentation to be sure. - -Caller-handled errors ---------------------- - -An increasing number of functions take a parameter 'struct strbuf *err'. -On error, such functions append a message about what went wrong to the -'err' strbuf. The message is meant to be complete enough to be passed -to `die` or `error` as-is. For example: - - if (ref_transaction_commit(transaction, &err)) - die("%s", err.buf); - -The 'err' parameter will be untouched if no error occurred, so multiple -function calls can be chained: - - t = ref_transaction_begin(&err); - if (!t || - ref_transaction_update(t, "HEAD", ..., &err) || - ret_transaction_commit(t, &err)) - die("%s", err.buf); - -The 'err' parameter must be a pointer to a valid strbuf. To silence -a message, pass a strbuf that is explicitly ignored: - - if (thing_that_can_fail_in_an_ignorable_way(..., &err)) - /* This failure is okay. */ - strbuf_reset(&err); diff --git a/Documentation/technical/api-index-skel.adoc b/Documentation/technical/api-index-skel.adoc new file mode 100644 index 0000000000..7780a76b08 --- /dev/null +++ b/Documentation/technical/api-index-skel.adoc @@ -0,0 +1,13 @@ +Git API Documents +================= + +Git has grown a set of internal APIs over time. This collection +documents them. + +//////////////////////////////////////////////////////////////// +// table of contents begin +//////////////////////////////////////////////////////////////// + +//////////////////////////////////////////////////////////////// +// table of contents end +//////////////////////////////////////////////////////////////// diff --git a/Documentation/technical/api-index-skel.txt b/Documentation/technical/api-index-skel.txt deleted file mode 100644 index 7780a76b08..0000000000 --- a/Documentation/technical/api-index-skel.txt +++ /dev/null @@ -1,13 +0,0 @@ -Git API Documents -================= - -Git has grown a set of internal APIs over time. This collection -documents them. - -//////////////////////////////////////////////////////////////// -// table of contents begin -//////////////////////////////////////////////////////////////// - -//////////////////////////////////////////////////////////////// -// table of contents end -//////////////////////////////////////////////////////////////// diff --git a/Documentation/technical/api-index.sh b/Documentation/technical/api-index.sh index 2964885574..dd206b1ca4 100755 --- a/Documentation/technical/api-index.sh +++ b/Documentation/technical/api-index.sh @@ -13,18 +13,18 @@ OUTPUT="$2" cd "$SOURCE_DIR" c=//////////////////////////////////////////////////////////////// - skel=api-index-skel.txt + skel=api-index-skel.adoc sed -e '/^\/\/ table of contents begin/q' "$skel" echo "$c" - ls api-*.txt | + ls api-*.adoc | while read filename do case "$filename" in - api-index-skel.txt | api-index.txt) continue ;; + api-index-skel.adoc | api-index.adoc) continue ;; esac title=$(sed -e 1q "$filename") - html=${filename%.txt}.html + html=${filename%.adoc}.html echo "* link:$html[$title]" done echo "$c" diff --git a/Documentation/technical/api-merge.adoc b/Documentation/technical/api-merge.adoc new file mode 100644 index 0000000000..c2ba01828c --- /dev/null +++ b/Documentation/technical/api-merge.adoc @@ -0,0 +1,36 @@ +merge API +========= + +The merge API helps a program to reconcile two competing sets of +improvements to some files (e.g., unregistered changes from the work +tree versus changes involved in switching to a new branch), reporting +conflicts if found. The library called through this API is +responsible for a few things. + + * determining which trees to merge (recursive ancestor consolidation); + + * lining up corresponding files in the trees to be merged (rename + detection, subtree shifting), reporting edge cases like add/add + and rename/rename conflicts to the user; + + * performing a three-way merge of corresponding files, taking + path-specific merge drivers (specified in `.gitattributes`) + into account. + +Data structures +--------------- + +* `mmbuffer_t`, `mmfile_t` + +These store data usable for use by the xdiff backend, for writing and +for reading, respectively. See `xdiff/xdiff.h` for the definitions +and `diff.c` for examples. + +* `struct ll_merge_options` + +Check merge-ll.h for details. + +Low-level (single file) merge +----------------------------- + +Check merge-ll.h for details. diff --git a/Documentation/technical/api-merge.txt b/Documentation/technical/api-merge.txt deleted file mode 100644 index c2ba01828c..0000000000 --- a/Documentation/technical/api-merge.txt +++ /dev/null @@ -1,36 +0,0 @@ -merge API -========= - -The merge API helps a program to reconcile two competing sets of -improvements to some files (e.g., unregistered changes from the work -tree versus changes involved in switching to a new branch), reporting -conflicts if found. The library called through this API is -responsible for a few things. - - * determining which trees to merge (recursive ancestor consolidation); - - * lining up corresponding files in the trees to be merged (rename - detection, subtree shifting), reporting edge cases like add/add - and rename/rename conflicts to the user; - - * performing a three-way merge of corresponding files, taking - path-specific merge drivers (specified in `.gitattributes`) - into account. - -Data structures ---------------- - -* `mmbuffer_t`, `mmfile_t` - -These store data usable for use by the xdiff backend, for writing and -for reading, respectively. See `xdiff/xdiff.h` for the definitions -and `diff.c` for examples. - -* `struct ll_merge_options` - -Check merge-ll.h for details. - -Low-level (single file) merge ------------------------------ - -Check merge-ll.h for details. diff --git a/Documentation/technical/api-parse-options.adoc b/Documentation/technical/api-parse-options.adoc new file mode 100644 index 0000000000..61fa6ee167 --- /dev/null +++ b/Documentation/technical/api-parse-options.adoc @@ -0,0 +1,349 @@ +parse-options API +================= + +The parse-options API is used to parse and massage options in Git +and to provide a usage help with consistent look. + +Basics +------ + +The argument vector `argv[]` may usually contain mandatory or optional +'non-option arguments', e.g. a filename or a branch, 'options', and +'subcommands'. +Options are optional arguments that start with a dash and +that allow to change the behavior of a command. + +* There are basically three types of options: + 'boolean' options, + options with (mandatory) 'arguments' and + options with 'optional arguments' + (i.e. a boolean option that can be adjusted). + +* There are basically two forms of options: + 'Short options' consist of one dash (`-`) and one alphanumeric + character. + 'Long options' begin with two dashes (`--`) and some + alphanumeric characters. + +* Options are case-sensitive. + Please define 'lower-case long options' only. + +The parse-options API allows: + +* 'stuck' and 'separate form' of options with arguments. + `-oArg` is stuck, `-o Arg` is separate form. + `--option=Arg` is stuck, `--option Arg` is separate form. + +* Long options may be 'abbreviated', as long as the abbreviation + is unambiguous. + +* Short options may be bundled, e.g. `-a -b` can be specified as `-ab`. + +* Boolean long options can be 'negated' (or 'unset') by prepending + `no-`, e.g. `--no-abbrev` instead of `--abbrev`. Conversely, + options that begin with `no-` can be 'negated' by removing it. + Other long options can be unset (e.g., set string to NULL, set + integer to 0) by prepending `no-`. + +* Options and non-option arguments can clearly be separated using the `--` + option, e.g. `-a -b --option -- --this-is-a-file` indicates that + `--this-is-a-file` must not be processed as an option. + +Subcommands are special in a couple of ways: + +* Subcommands only have long form, and they have no double dash prefix, no + negated form, and no description, and they don't take any arguments, and + can't be abbreviated. + +* There must be exactly one subcommand among the arguments, or zero if the + command has a default operation mode. + +* All arguments following the subcommand are considered to be arguments of + the subcommand, and, conversely, arguments meant for the subcommand may + not precede the subcommand. + +Therefore, if the options array contains at least one subcommand and +`parse_options()` encounters the first dashless argument, it will either: + +* stop and return, if that dashless argument is a known subcommand, setting + `value` to the function pointer associated with that subcommand, storing + the name of the subcommand in argv[0], and leaving the rest of the + arguments unprocessed, or + +* stop and return, if it was invoked with the `PARSE_OPT_SUBCOMMAND_OPTIONAL` + flag and that dashless argument doesn't match any subcommands, leaving + `value` unchanged and the rest of the arguments unprocessed, or + +* show error and usage, and abort. + +Steps to parse options +---------------------- + +. `#include "parse-options.h"` + +. define a NULL-terminated + `static const char * const builtin_foo_usage[]` array + containing alternative usage strings + +. define `builtin_foo_options` array as described below + in section 'Data Structure'. + +. in `cmd_foo(int argc, const char **argv, const char *prefix)` + call + + argc = parse_options(argc, argv, prefix, builtin_foo_options, builtin_foo_usage, flags); ++ +`parse_options()` will filter out the processed options of `argv[]` and leave the +non-option arguments in `argv[]`. +`argc` is updated appropriately because of the assignment. ++ +You can also pass NULL instead of a usage array as the fifth parameter of +parse_options(), to avoid displaying a help screen with usage info and +option list. This should only be done if necessary, e.g. to implement +a limited parser for only a subset of the options that needs to be run +before the full parser, which in turn shows the full help message. ++ +Flags are the bitwise-or of: + +`PARSE_OPT_KEEP_DASHDASH`:: + Keep the `--` that usually separates options from + non-option arguments. + +`PARSE_OPT_STOP_AT_NON_OPTION`:: + Usually the whole argument vector is massaged and reordered. + Using this flag, processing is stopped at the first non-option + argument. + +`PARSE_OPT_KEEP_ARGV0`:: + Keep the first argument, which contains the program name. It's + removed from argv[] by default. + +`PARSE_OPT_KEEP_UNKNOWN_OPT`:: + Keep unknown options instead of erroring out. This doesn't + work for all combinations of arguments as users might expect + it to do. E.g. if the first argument in `--unknown --known` + takes a value (which we can't know), the second one is + mistakenly interpreted as a known option. Similarly, if + `PARSE_OPT_STOP_AT_NON_OPTION` is set, the second argument in + `--unknown value` will be mistakenly interpreted as a + non-option, not as a value belonging to the unknown option, + the parser early. That's why parse_options() errors out if + both options are set. + Note that non-option arguments are always kept, even without + this flag. + +`PARSE_OPT_NO_INTERNAL_HELP`:: + By default, parse_options() handles `-h`, `--help` and + `--help-all` internally, by showing a help screen. This option + turns it off and allows one to add custom handlers for these + options, or to just leave them unknown. + +`PARSE_OPT_SUBCOMMAND_OPTIONAL`:: + Don't error out when no subcommand is specified. + +Note that `PARSE_OPT_STOP_AT_NON_OPTION` is incompatible with subcommands; +while `PARSE_OPT_KEEP_DASHDASH` and `PARSE_OPT_KEEP_UNKNOWN_OPT` can only be +used with subcommands when combined with `PARSE_OPT_SUBCOMMAND_OPTIONAL`. + +Data Structure +-------------- + +The main data structure is an array of the `option` struct, +say `static struct option builtin_add_options[]`. +There are some macros to easily define options: + +`OPT__ABBREV(&int_var)`:: + Add `--abbrev[=]`. + +`OPT__COLOR(&int_var, description)`:: + Add `--color[=]` and `--no-color`. + +`OPT__DRY_RUN(&int_var, description)`:: + Add `-n, --dry-run`. + +`OPT__FORCE(&int_var, description)`:: + Add `-f, --force`. + +`OPT__QUIET(&int_var, description)`:: + Add `-q, --quiet`. + +`OPT__VERBOSE(&int_var, description)`:: + Add `-v, --verbose`. + +`OPT_GROUP(description)`:: + Start an option group. `description` is a short string that + describes the group or an empty string. + Start the description with an upper-case letter. + +`OPT_BOOL(short, long, &int_var, description)`:: + Introduce a boolean option. `int_var` is set to one with + `--option` and set to zero with `--no-option`. + +`OPT_COUNTUP(short, long, &int_var, description)`:: + Introduce a count-up option. + Each use of `--option` increments `int_var`, starting from zero + (even if initially negative), and `--no-option` resets it to + zero. To determine if `--option` or `--no-option` was encountered at + all, initialize `int_var` to a negative value, and if it is still + negative after parse_options(), then neither `--option` nor + `--no-option` was seen. + +`OPT_BIT(short, long, &int_var, description, mask)`:: + Introduce a boolean option. + If used, `int_var` is bitwise-ored with `mask`. + +`OPT_NEGBIT(short, long, &int_var, description, mask)`:: + Introduce a boolean option. + If used, `int_var` is bitwise-anded with the inverted `mask`. + +`OPT_SET_INT(short, long, &int_var, description, integer)`:: + Introduce an integer option. + `int_var` is set to `integer` with `--option`, and + reset to zero with `--no-option`. + +`OPT_STRING(short, long, &str_var, arg_str, description)`:: + Introduce an option with string argument. + The string argument is put into `str_var`. + +`OPT_STRING_LIST(short, long, &struct string_list, arg_str, description)`:: + Introduce an option with string argument. + The string argument is stored as an element in `string_list`. + Use of `--no-option` will clear the list of preceding values. + +`OPT_INTEGER(short, long, &int_var, description)`:: + Introduce an option with integer argument. + The integer is put into `int_var`. + +`OPT_MAGNITUDE(short, long, &unsigned_long_var, description)`:: + Introduce an option with a size argument. The argument must be a + non-negative integer and may include a suffix of 'k', 'm' or 'g' to + scale the provided value by 1024, 1024^2 or 1024^3 respectively. + The scaled value is put into `unsigned_long_var`. + +`OPT_EXPIRY_DATE(short, long, ×tamp_t_var, description)`:: + Introduce an option with expiry date argument, see `parse_expiry_date()`. + The timestamp is put into `timestamp_t_var`. + +`OPT_CALLBACK(short, long, &var, arg_str, description, func_ptr)`:: + Introduce an option with argument. + The argument will be fed into the function given by `func_ptr` + and the result will be put into `var`. + See 'Option Callbacks' below for a more elaborate description. + +`OPT_FILENAME(short, long, &var, description)`:: + Introduce an option with a filename argument. + The filename will be prefixed by passing the filename along with + the prefix argument of `parse_options()` to `prefix_filename()`. + +`OPT_NUMBER_CALLBACK(&var, description, func_ptr)`:: + Recognize numerical options like -123 and feed the integer as + if it was an argument to the function given by `func_ptr`. + The result will be put into `var`. There can be only one such + option definition. It cannot be negated and it takes no + arguments. Short options that happen to be digits take + precedence over it. + +`OPT_COLOR_FLAG(short, long, &int_var, description)`:: + Introduce an option that takes an optional argument that can + have one of three values: "always", "never", or "auto". If the + argument is not given, it defaults to "always". The `--no-` form + works like `--long=never`; it cannot take an argument. If + "always", set `int_var` to 1; if "never", set `int_var` to 0; if + "auto", set `int_var` to 1 if stdout is a tty or a pager, + 0 otherwise. + +`OPT_NOOP_NOARG(short, long)`:: + Introduce an option that has no effect and takes no arguments. + Use it to hide deprecated options that are still to be recognized + and ignored silently. + +`OPT_PASSTHRU(short, long, &char_var, arg_str, description, flags)`:: + Introduce an option that will be reconstructed into a char* string, + which must be initialized to NULL. This is useful when you need to + pass the command-line option to another command. Any previous value + will be overwritten, so this should only be used for options where + the last one specified on the command line wins. + +`OPT_PASSTHRU_ARGV(short, long, &strvec_var, arg_str, description, flags)`:: + Introduce an option where all instances of it on the command-line will + be reconstructed into a strvec. This is useful when you need to + pass the command-line option, which can be specified multiple times, + to another command. + +`OPT_CMDMODE(short, long, &int_var, description, enum_val)`:: + Define an "operation mode" option, only one of which in the same + group of "operating mode" options that share the same `int_var` + can be given by the user. `int_var` is set to `enum_val` when the + option is used, but an error is reported if other "operating mode" + option has already set its value to the same `int_var`. + In new commands consider using subcommands instead. + +`OPT_SUBCOMMAND(long, &fn_ptr, subcommand_fn)`:: + Define a subcommand. `subcommand_fn` is put into `fn_ptr` when + this subcommand is used. + +The last element of the array must be `OPT_END()`. + +If not stated otherwise, interpret the arguments as follows: + +* `short` is a character for the short option + (e.g. `'e'` for `-e`, use `0` to omit), + +* `long` is a string for the long option + (e.g. `"example"` for `--example`, use `NULL` to omit), + +* `int_var` is an integer variable, + +* `str_var` is a string variable (`char *`), + +* `arg_str` is the string that is shown as argument + (e.g. `"branch"` will result in ``). + If set to `NULL`, three dots (`...`) will be displayed. + +* `description` is a short string to describe the effect of the option. + It shall begin with a lower-case letter and a full stop (`.`) shall be + omitted at the end. + +Option Callbacks +---------------- + +The function must be defined in this form: + + int func(const struct option *opt, const char *arg, int unset) + +The callback mechanism is as follows: + +* Inside `func`, the only interesting member of the structure + given by `opt` is the void pointer `opt->value`. + `*opt->value` will be the value that is saved into `var`, if you + use `OPT_CALLBACK()`. + For example, do `*(unsigned long *)opt->value = 42;` to get 42 + into an `unsigned long` variable. + +* Return value `0` indicates success and non-zero return + value will invoke `usage_with_options()` and, thus, die. + +* If the user negates the option, `arg` is `NULL` and `unset` is 1. + +Sophisticated option parsing +---------------------------- + +If you need, for example, option callbacks with optional arguments +or without arguments at all, or if you need other special cases, +that are not handled by the macros above, you need to specify the +members of the `option` structure manually. + +This is not covered in this document, but well documented +in `parse-options.h` itself. + +Examples +-------- + +See `test-parse-options.c` and +`builtin/add.c`, +`builtin/clone.c`, +`builtin/commit.c`, +`builtin/fetch.c`, +`builtin/fsck.c`, +`builtin/rm.c` +for real-world examples. diff --git a/Documentation/technical/api-parse-options.txt b/Documentation/technical/api-parse-options.txt deleted file mode 100644 index 61fa6ee167..0000000000 --- a/Documentation/technical/api-parse-options.txt +++ /dev/null @@ -1,349 +0,0 @@ -parse-options API -================= - -The parse-options API is used to parse and massage options in Git -and to provide a usage help with consistent look. - -Basics ------- - -The argument vector `argv[]` may usually contain mandatory or optional -'non-option arguments', e.g. a filename or a branch, 'options', and -'subcommands'. -Options are optional arguments that start with a dash and -that allow to change the behavior of a command. - -* There are basically three types of options: - 'boolean' options, - options with (mandatory) 'arguments' and - options with 'optional arguments' - (i.e. a boolean option that can be adjusted). - -* There are basically two forms of options: - 'Short options' consist of one dash (`-`) and one alphanumeric - character. - 'Long options' begin with two dashes (`--`) and some - alphanumeric characters. - -* Options are case-sensitive. - Please define 'lower-case long options' only. - -The parse-options API allows: - -* 'stuck' and 'separate form' of options with arguments. - `-oArg` is stuck, `-o Arg` is separate form. - `--option=Arg` is stuck, `--option Arg` is separate form. - -* Long options may be 'abbreviated', as long as the abbreviation - is unambiguous. - -* Short options may be bundled, e.g. `-a -b` can be specified as `-ab`. - -* Boolean long options can be 'negated' (or 'unset') by prepending - `no-`, e.g. `--no-abbrev` instead of `--abbrev`. Conversely, - options that begin with `no-` can be 'negated' by removing it. - Other long options can be unset (e.g., set string to NULL, set - integer to 0) by prepending `no-`. - -* Options and non-option arguments can clearly be separated using the `--` - option, e.g. `-a -b --option -- --this-is-a-file` indicates that - `--this-is-a-file` must not be processed as an option. - -Subcommands are special in a couple of ways: - -* Subcommands only have long form, and they have no double dash prefix, no - negated form, and no description, and they don't take any arguments, and - can't be abbreviated. - -* There must be exactly one subcommand among the arguments, or zero if the - command has a default operation mode. - -* All arguments following the subcommand are considered to be arguments of - the subcommand, and, conversely, arguments meant for the subcommand may - not precede the subcommand. - -Therefore, if the options array contains at least one subcommand and -`parse_options()` encounters the first dashless argument, it will either: - -* stop and return, if that dashless argument is a known subcommand, setting - `value` to the function pointer associated with that subcommand, storing - the name of the subcommand in argv[0], and leaving the rest of the - arguments unprocessed, or - -* stop and return, if it was invoked with the `PARSE_OPT_SUBCOMMAND_OPTIONAL` - flag and that dashless argument doesn't match any subcommands, leaving - `value` unchanged and the rest of the arguments unprocessed, or - -* show error and usage, and abort. - -Steps to parse options ----------------------- - -. `#include "parse-options.h"` - -. define a NULL-terminated - `static const char * const builtin_foo_usage[]` array - containing alternative usage strings - -. define `builtin_foo_options` array as described below - in section 'Data Structure'. - -. in `cmd_foo(int argc, const char **argv, const char *prefix)` - call - - argc = parse_options(argc, argv, prefix, builtin_foo_options, builtin_foo_usage, flags); -+ -`parse_options()` will filter out the processed options of `argv[]` and leave the -non-option arguments in `argv[]`. -`argc` is updated appropriately because of the assignment. -+ -You can also pass NULL instead of a usage array as the fifth parameter of -parse_options(), to avoid displaying a help screen with usage info and -option list. This should only be done if necessary, e.g. to implement -a limited parser for only a subset of the options that needs to be run -before the full parser, which in turn shows the full help message. -+ -Flags are the bitwise-or of: - -`PARSE_OPT_KEEP_DASHDASH`:: - Keep the `--` that usually separates options from - non-option arguments. - -`PARSE_OPT_STOP_AT_NON_OPTION`:: - Usually the whole argument vector is massaged and reordered. - Using this flag, processing is stopped at the first non-option - argument. - -`PARSE_OPT_KEEP_ARGV0`:: - Keep the first argument, which contains the program name. It's - removed from argv[] by default. - -`PARSE_OPT_KEEP_UNKNOWN_OPT`:: - Keep unknown options instead of erroring out. This doesn't - work for all combinations of arguments as users might expect - it to do. E.g. if the first argument in `--unknown --known` - takes a value (which we can't know), the second one is - mistakenly interpreted as a known option. Similarly, if - `PARSE_OPT_STOP_AT_NON_OPTION` is set, the second argument in - `--unknown value` will be mistakenly interpreted as a - non-option, not as a value belonging to the unknown option, - the parser early. That's why parse_options() errors out if - both options are set. - Note that non-option arguments are always kept, even without - this flag. - -`PARSE_OPT_NO_INTERNAL_HELP`:: - By default, parse_options() handles `-h`, `--help` and - `--help-all` internally, by showing a help screen. This option - turns it off and allows one to add custom handlers for these - options, or to just leave them unknown. - -`PARSE_OPT_SUBCOMMAND_OPTIONAL`:: - Don't error out when no subcommand is specified. - -Note that `PARSE_OPT_STOP_AT_NON_OPTION` is incompatible with subcommands; -while `PARSE_OPT_KEEP_DASHDASH` and `PARSE_OPT_KEEP_UNKNOWN_OPT` can only be -used with subcommands when combined with `PARSE_OPT_SUBCOMMAND_OPTIONAL`. - -Data Structure --------------- - -The main data structure is an array of the `option` struct, -say `static struct option builtin_add_options[]`. -There are some macros to easily define options: - -`OPT__ABBREV(&int_var)`:: - Add `--abbrev[=]`. - -`OPT__COLOR(&int_var, description)`:: - Add `--color[=]` and `--no-color`. - -`OPT__DRY_RUN(&int_var, description)`:: - Add `-n, --dry-run`. - -`OPT__FORCE(&int_var, description)`:: - Add `-f, --force`. - -`OPT__QUIET(&int_var, description)`:: - Add `-q, --quiet`. - -`OPT__VERBOSE(&int_var, description)`:: - Add `-v, --verbose`. - -`OPT_GROUP(description)`:: - Start an option group. `description` is a short string that - describes the group or an empty string. - Start the description with an upper-case letter. - -`OPT_BOOL(short, long, &int_var, description)`:: - Introduce a boolean option. `int_var` is set to one with - `--option` and set to zero with `--no-option`. - -`OPT_COUNTUP(short, long, &int_var, description)`:: - Introduce a count-up option. - Each use of `--option` increments `int_var`, starting from zero - (even if initially negative), and `--no-option` resets it to - zero. To determine if `--option` or `--no-option` was encountered at - all, initialize `int_var` to a negative value, and if it is still - negative after parse_options(), then neither `--option` nor - `--no-option` was seen. - -`OPT_BIT(short, long, &int_var, description, mask)`:: - Introduce a boolean option. - If used, `int_var` is bitwise-ored with `mask`. - -`OPT_NEGBIT(short, long, &int_var, description, mask)`:: - Introduce a boolean option. - If used, `int_var` is bitwise-anded with the inverted `mask`. - -`OPT_SET_INT(short, long, &int_var, description, integer)`:: - Introduce an integer option. - `int_var` is set to `integer` with `--option`, and - reset to zero with `--no-option`. - -`OPT_STRING(short, long, &str_var, arg_str, description)`:: - Introduce an option with string argument. - The string argument is put into `str_var`. - -`OPT_STRING_LIST(short, long, &struct string_list, arg_str, description)`:: - Introduce an option with string argument. - The string argument is stored as an element in `string_list`. - Use of `--no-option` will clear the list of preceding values. - -`OPT_INTEGER(short, long, &int_var, description)`:: - Introduce an option with integer argument. - The integer is put into `int_var`. - -`OPT_MAGNITUDE(short, long, &unsigned_long_var, description)`:: - Introduce an option with a size argument. The argument must be a - non-negative integer and may include a suffix of 'k', 'm' or 'g' to - scale the provided value by 1024, 1024^2 or 1024^3 respectively. - The scaled value is put into `unsigned_long_var`. - -`OPT_EXPIRY_DATE(short, long, ×tamp_t_var, description)`:: - Introduce an option with expiry date argument, see `parse_expiry_date()`. - The timestamp is put into `timestamp_t_var`. - -`OPT_CALLBACK(short, long, &var, arg_str, description, func_ptr)`:: - Introduce an option with argument. - The argument will be fed into the function given by `func_ptr` - and the result will be put into `var`. - See 'Option Callbacks' below for a more elaborate description. - -`OPT_FILENAME(short, long, &var, description)`:: - Introduce an option with a filename argument. - The filename will be prefixed by passing the filename along with - the prefix argument of `parse_options()` to `prefix_filename()`. - -`OPT_NUMBER_CALLBACK(&var, description, func_ptr)`:: - Recognize numerical options like -123 and feed the integer as - if it was an argument to the function given by `func_ptr`. - The result will be put into `var`. There can be only one such - option definition. It cannot be negated and it takes no - arguments. Short options that happen to be digits take - precedence over it. - -`OPT_COLOR_FLAG(short, long, &int_var, description)`:: - Introduce an option that takes an optional argument that can - have one of three values: "always", "never", or "auto". If the - argument is not given, it defaults to "always". The `--no-` form - works like `--long=never`; it cannot take an argument. If - "always", set `int_var` to 1; if "never", set `int_var` to 0; if - "auto", set `int_var` to 1 if stdout is a tty or a pager, - 0 otherwise. - -`OPT_NOOP_NOARG(short, long)`:: - Introduce an option that has no effect and takes no arguments. - Use it to hide deprecated options that are still to be recognized - and ignored silently. - -`OPT_PASSTHRU(short, long, &char_var, arg_str, description, flags)`:: - Introduce an option that will be reconstructed into a char* string, - which must be initialized to NULL. This is useful when you need to - pass the command-line option to another command. Any previous value - will be overwritten, so this should only be used for options where - the last one specified on the command line wins. - -`OPT_PASSTHRU_ARGV(short, long, &strvec_var, arg_str, description, flags)`:: - Introduce an option where all instances of it on the command-line will - be reconstructed into a strvec. This is useful when you need to - pass the command-line option, which can be specified multiple times, - to another command. - -`OPT_CMDMODE(short, long, &int_var, description, enum_val)`:: - Define an "operation mode" option, only one of which in the same - group of "operating mode" options that share the same `int_var` - can be given by the user. `int_var` is set to `enum_val` when the - option is used, but an error is reported if other "operating mode" - option has already set its value to the same `int_var`. - In new commands consider using subcommands instead. - -`OPT_SUBCOMMAND(long, &fn_ptr, subcommand_fn)`:: - Define a subcommand. `subcommand_fn` is put into `fn_ptr` when - this subcommand is used. - -The last element of the array must be `OPT_END()`. - -If not stated otherwise, interpret the arguments as follows: - -* `short` is a character for the short option - (e.g. `'e'` for `-e`, use `0` to omit), - -* `long` is a string for the long option - (e.g. `"example"` for `--example`, use `NULL` to omit), - -* `int_var` is an integer variable, - -* `str_var` is a string variable (`char *`), - -* `arg_str` is the string that is shown as argument - (e.g. `"branch"` will result in ``). - If set to `NULL`, three dots (`...`) will be displayed. - -* `description` is a short string to describe the effect of the option. - It shall begin with a lower-case letter and a full stop (`.`) shall be - omitted at the end. - -Option Callbacks ----------------- - -The function must be defined in this form: - - int func(const struct option *opt, const char *arg, int unset) - -The callback mechanism is as follows: - -* Inside `func`, the only interesting member of the structure - given by `opt` is the void pointer `opt->value`. - `*opt->value` will be the value that is saved into `var`, if you - use `OPT_CALLBACK()`. - For example, do `*(unsigned long *)opt->value = 42;` to get 42 - into an `unsigned long` variable. - -* Return value `0` indicates success and non-zero return - value will invoke `usage_with_options()` and, thus, die. - -* If the user negates the option, `arg` is `NULL` and `unset` is 1. - -Sophisticated option parsing ----------------------------- - -If you need, for example, option callbacks with optional arguments -or without arguments at all, or if you need other special cases, -that are not handled by the macros above, you need to specify the -members of the `option` structure manually. - -This is not covered in this document, but well documented -in `parse-options.h` itself. - -Examples --------- - -See `test-parse-options.c` and -`builtin/add.c`, -`builtin/clone.c`, -`builtin/commit.c`, -`builtin/fetch.c`, -`builtin/fsck.c`, -`builtin/rm.c` -for real-world examples. diff --git a/Documentation/technical/api-simple-ipc.adoc b/Documentation/technical/api-simple-ipc.adoc new file mode 100644 index 0000000000..c4fb152b23 --- /dev/null +++ b/Documentation/technical/api-simple-ipc.adoc @@ -0,0 +1,105 @@ +Simple-IPC API +============== + +The Simple-IPC API is a collection of `ipc_` prefixed library routines +and a basic communication protocol that allows an IPC-client process to +send an application-specific IPC-request message to an IPC-server +process and receive an application-specific IPC-response message. + +Communication occurs over a named pipe on Windows and a Unix domain +socket on other platforms. IPC-clients and IPC-servers rendezvous at +a previously agreed-to application-specific pathname (which is outside +the scope of this design) that is local to the computer system. + +The IPC-server routines within the server application process create a +thread pool to listen for connections and receive request messages +from multiple concurrent IPC-clients. When received, these messages +are dispatched up to the server application callbacks for handling. +IPC-server routines then incrementally relay responses back to the +IPC-client. + +The IPC-client routines within a client application process connect +to the IPC-server and send a request message and wait for a response. +When received, the response is returned back to the caller. + +For example, the `fsmonitor--daemon` feature will be built as a server +application on top of the IPC-server library routines. It will have +threads watching for file system events and a thread pool waiting for +client connections. Clients, such as `git status`, will request a list +of file system events since a point in time and the server will +respond with a list of changed files and directories. The formats of +the request and response are application-specific; the IPC-client and +IPC-server routines treat them as opaque byte streams. + + +Comparison with sub-process model +--------------------------------- + +The Simple-IPC mechanism differs from the existing `sub-process.c` +model (Documentation/technical/long-running-process-protocol.txt) and +used by applications like Git-LFS. In the LFS-style sub-process model, +the helper is started by the foreground process, communication happens +via a pair of file descriptors bound to the stdin/stdout of the +sub-process, the sub-process only serves the current foreground +process, and the sub-process exits when the foreground process +terminates. + +In the Simple-IPC model the server is a very long-running service. It +can service many clients at the same time and has a private socket or +named pipe connection to each active client. It might be started +(on-demand) by the current client process or it might have been +started by a previous client or by the OS at boot time. The server +process is not associated with a terminal and it persists after +clients terminate. Clients do not have access to the stdin/stdout of +the server process and therefore must communicate over sockets or +named pipes. + + +Server startup and shutdown +--------------------------- + +How an application server based upon IPC-server is started is also +outside the scope of the Simple-IPC design and is a property of the +application using it. For example, the server might be started or +restarted during routine maintenance operations, or it might be +started as a system service during the system boot-up sequence, or it +might be started on-demand by a foreground Git command when needed. + +Similarly, server shutdown is a property of the application using +the simple-ipc routines. For example, the server might decide to +shutdown when idle or only upon explicit request. + + +Simple-IPC protocol +------------------- + +The Simple-IPC protocol consists of a single request message from the +client and an optional response message from the server. Both the +client and server messages are unlimited in length and are terminated +with a flush packet. + +The pkt-line routines (linkgit:gitprotocol-common[5]) +are used to simplify buffer management during message generation, +transmission, and reception. A flush packet is used to mark the end +of the message. This allows the sender to incrementally generate and +transmit the message. It allows the receiver to incrementally receive +the message in chunks and to know when they have received the entire +message. + +The actual byte format of the client request and server response +messages are application specific. The IPC layer transmits and +receives them as opaque byte buffers without any concern for the +content within. It is the job of the calling application layer to +understand the contents of the request and response messages. + + +Summary +------- + +Conceptually, the Simple-IPC protocol is similar to an HTTP REST +request. Clients connect, make an application-specific and +stateless request, receive an application-specific +response, and disconnect. It is a one round trip facility for +querying the server. The Simple-IPC routines hide the socket, +named pipe, and thread pool details and allow the application +layer to focus on the task at hand. diff --git a/Documentation/technical/api-simple-ipc.txt b/Documentation/technical/api-simple-ipc.txt deleted file mode 100644 index c4fb152b23..0000000000 --- a/Documentation/technical/api-simple-ipc.txt +++ /dev/null @@ -1,105 +0,0 @@ -Simple-IPC API -============== - -The Simple-IPC API is a collection of `ipc_` prefixed library routines -and a basic communication protocol that allows an IPC-client process to -send an application-specific IPC-request message to an IPC-server -process and receive an application-specific IPC-response message. - -Communication occurs over a named pipe on Windows and a Unix domain -socket on other platforms. IPC-clients and IPC-servers rendezvous at -a previously agreed-to application-specific pathname (which is outside -the scope of this design) that is local to the computer system. - -The IPC-server routines within the server application process create a -thread pool to listen for connections and receive request messages -from multiple concurrent IPC-clients. When received, these messages -are dispatched up to the server application callbacks for handling. -IPC-server routines then incrementally relay responses back to the -IPC-client. - -The IPC-client routines within a client application process connect -to the IPC-server and send a request message and wait for a response. -When received, the response is returned back to the caller. - -For example, the `fsmonitor--daemon` feature will be built as a server -application on top of the IPC-server library routines. It will have -threads watching for file system events and a thread pool waiting for -client connections. Clients, such as `git status`, will request a list -of file system events since a point in time and the server will -respond with a list of changed files and directories. The formats of -the request and response are application-specific; the IPC-client and -IPC-server routines treat them as opaque byte streams. - - -Comparison with sub-process model ---------------------------------- - -The Simple-IPC mechanism differs from the existing `sub-process.c` -model (Documentation/technical/long-running-process-protocol.txt) and -used by applications like Git-LFS. In the LFS-style sub-process model, -the helper is started by the foreground process, communication happens -via a pair of file descriptors bound to the stdin/stdout of the -sub-process, the sub-process only serves the current foreground -process, and the sub-process exits when the foreground process -terminates. - -In the Simple-IPC model the server is a very long-running service. It -can service many clients at the same time and has a private socket or -named pipe connection to each active client. It might be started -(on-demand) by the current client process or it might have been -started by a previous client or by the OS at boot time. The server -process is not associated with a terminal and it persists after -clients terminate. Clients do not have access to the stdin/stdout of -the server process and therefore must communicate over sockets or -named pipes. - - -Server startup and shutdown ---------------------------- - -How an application server based upon IPC-server is started is also -outside the scope of the Simple-IPC design and is a property of the -application using it. For example, the server might be started or -restarted during routine maintenance operations, or it might be -started as a system service during the system boot-up sequence, or it -might be started on-demand by a foreground Git command when needed. - -Similarly, server shutdown is a property of the application using -the simple-ipc routines. For example, the server might decide to -shutdown when idle or only upon explicit request. - - -Simple-IPC protocol -------------------- - -The Simple-IPC protocol consists of a single request message from the -client and an optional response message from the server. Both the -client and server messages are unlimited in length and are terminated -with a flush packet. - -The pkt-line routines (linkgit:gitprotocol-common[5]) -are used to simplify buffer management during message generation, -transmission, and reception. A flush packet is used to mark the end -of the message. This allows the sender to incrementally generate and -transmit the message. It allows the receiver to incrementally receive -the message in chunks and to know when they have received the entire -message. - -The actual byte format of the client request and server response -messages are application specific. The IPC layer transmits and -receives them as opaque byte buffers without any concern for the -content within. It is the job of the calling application layer to -understand the contents of the request and response messages. - - -Summary -------- - -Conceptually, the Simple-IPC protocol is similar to an HTTP REST -request. Clients connect, make an application-specific and -stateless request, receive an application-specific -response, and disconnect. It is a one round trip facility for -querying the server. The Simple-IPC routines hide the socket, -named pipe, and thread pool details and allow the application -layer to focus on the task at hand. diff --git a/Documentation/technical/api-trace2.adoc b/Documentation/technical/api-trace2.adoc new file mode 100644 index 0000000000..cf493dae03 --- /dev/null +++ b/Documentation/technical/api-trace2.adoc @@ -0,0 +1,1352 @@ += Trace2 API + +The Trace2 API can be used to print debug, performance, and telemetry +information to stderr or a file. The Trace2 feature is inactive unless +explicitly enabled by enabling one or more Trace2 Targets. + +The Trace2 API is intended to replace the existing (Trace1) +`printf()`-style tracing provided by the existing `GIT_TRACE` and +`GIT_TRACE_PERFORMANCE` facilities. During initial implementation, +Trace2 and Trace1 may operate in parallel. + +The Trace2 API defines a set of high-level messages with known fields, +such as (`start`: `argv`) and (`exit`: {`exit-code`, `elapsed-time`}). + +Trace2 instrumentation throughout the Git code base sends Trace2 +messages to the enabled Trace2 Targets. Targets transform these +messages content into purpose-specific formats and write events to +their data streams. In this manner, the Trace2 API can drive +many different types of analysis. + +Targets are defined using a VTable allowing easy extension to other +formats in the future. This might be used to define a binary format, +for example. + +Trace2 is controlled using `trace2.*` config values in the system and +global config files and `GIT_TRACE2*` environment variables. Trace2 does +not read from repo local or worktree config files, nor does it respect +`-c` command line config settings. + +== Trace2 Targets + +Trace2 defines the following set of Trace2 Targets. +Format details are given in a later section. + +=== The Normal Format Target + +The normal format target is a traditional `printf()` format and similar +to the `GIT_TRACE` format. This format is enabled with the `GIT_TRACE2` +environment variable or the `trace2.normalTarget` system or global +config setting. + +For example + +------------ +$ export GIT_TRACE2=~/log.normal +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +or + +------------ +$ git config --global trace2.normalTarget ~/log.normal +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +yields + +------------ +$ cat ~/log.normal +12:28:42.620009 common-main.c:38 version 2.20.1.155.g426c96fcdb +12:28:42.620989 common-main.c:39 start git version +12:28:42.621101 git.c:432 cmd_name version (version) +12:28:42.621215 git.c:662 exit elapsed:0.001227 code:0 +12:28:42.621250 trace2/tr2_tgt_normal.c:124 atexit elapsed:0.001265 code:0 +------------ + +=== The Performance Format Target + +The performance format target (PERF) is a column-based format to +replace `GIT_TRACE_PERFORMANCE` and is suitable for development and +testing, possibly to complement tools like `gprof`. This format is +enabled with the `GIT_TRACE2_PERF` environment variable or the +`trace2.perfTarget` system or global config setting. + +For example + +------------ +$ export GIT_TRACE2_PERF=~/log.perf +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +or + +------------ +$ git config --global trace2.perfTarget ~/log.perf +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +yields + +------------ +$ cat ~/log.perf +12:28:42.620675 common-main.c:38 | d0 | main | version | | | | | 2.20.1.155.g426c96fcdb +12:28:42.621001 common-main.c:39 | d0 | main | start | | 0.001173 | | | git version +12:28:42.621111 git.c:432 | d0 | main | cmd_name | | | | | version (version) +12:28:42.621225 git.c:662 | d0 | main | exit | | 0.001227 | | | code:0 +12:28:42.621259 trace2/tr2_tgt_perf.c:211 | d0 | main | atexit | | 0.001265 | | | code:0 +------------ + +=== The Event Format Target + +The event format target is a JSON-based format of event data suitable +for telemetry analysis. This format is enabled with the `GIT_TRACE2_EVENT` +environment variable or the `trace2.eventTarget` system or global config +setting. + +For example + +------------ +$ export GIT_TRACE2_EVENT=~/log.event +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +or + +------------ +$ git config --global trace2.eventTarget ~/log.event +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +yields + +------------ +$ cat ~/log.event +{"event":"version","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.620713Z","file":"common-main.c","line":38,"evt":"4","exe":"2.20.1.155.g426c96fcdb"} +{"event":"start","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621027Z","file":"common-main.c","line":39,"t_abs":0.001173,"argv":["git","version"]} +{"event":"cmd_name","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621122Z","file":"git.c","line":432,"name":"version","hierarchy":"version"} +{"event":"exit","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621236Z","file":"git.c","line":662,"t_abs":0.001227,"code":0} +{"event":"atexit","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621268Z","file":"trace2/tr2_tgt_event.c","line":163,"t_abs":0.001265,"code":0} +------------ + +=== Enabling a Target + +To enable a target, set the corresponding environment variable or +system or global config value to one of the following: + +include::../trace2-target-values.adoc[] + +When trace files are written to a target directory, they will be named according +to the last component of the SID (optionally followed by a counter to avoid +filename collisions). + +== Trace2 API + +The Trace2 public API is defined and documented in `trace2.h`; refer to it for +more information. All public functions and macros are prefixed +with `trace2_` and are implemented in `trace2.c`. + +There are no public Trace2 data structures. + +The Trace2 code also defines a set of private functions and data types +in the `trace2/` directory. These symbols are prefixed with `tr2_` +and should only be used by functions in `trace2.c` (or other private +source files in `trace2/`). + +=== Conventions for Public Functions and Macros + +Some functions have a `_fl()` suffix to indicate that they take `file` +and `line-number` arguments. + +Some functions have a `_va_fl()` suffix to indicate that they also +take a `va_list` argument. + +Some functions have a `_printf_fl()` suffix to indicate that they also +take a `printf()` style format with a variable number of arguments. + +CPP wrapper macros are defined to hide most of these details. + +== Trace2 Target Formats + +=== NORMAL Format + +Events are written as lines of the form: + +------------ +[