hook: add jobs option

Allow the API callers to specify the number of jobs across which
hook execution can be parallelized. It defaults to 1 and no hook
currently changes it, so all hooks run sequentially as before.

This allows us to both pave the way for parallel hook execution
(that will be a follow-up patch series building upon this) and to
finish the API conversion of builtin/receive-pack.c, keeping the
output async sideband thread ("muxer") design as Peff suggested.

When .jobs==1 nothing changes, the "copy_to_sideband" async thread
still outputs directly via sideband channel 2, keeping the current
(mostly) real-time output characteristics, avoids unnecessary poll
delays or deadlock risks.

When .jobs > 1, a more complex muxer is needed to buffer the hook
output and avoid interleaving. After working on this mux I quickly
realized I was re-implementing run-command with ungroup=0 so that
idea was dropped in favor of run-command which outputs to stderr.

In other words, run-command itself already can buffer/deinterleave
pp child outputs (ungroup=0), so we can just connect its stderr to
the sideband async task when jobs > 1.

Maybe it helps to illustrate how it works with ascii graphics:

 [ Sequential (jobs = 1) ]             [ Parallel (jobs > 1) ]

 +--------------+                      +--------+   +--------+
 | Hook Process |                      | Hook 1 |   | Hook 2 |
 +--------------+                      +--------+   +--------+
        |                                  |             |
        | stderr (inherited)               | stderr pipe |
        |                                  | (captured)  |
        v                                  v             v
 +-------------------------------------------------------------+
 |                      Parent Process                         |
 |                                                             |
 |      (direct write)              [run-command (buffered)]   |
 |             |                                 |             |
 |             |                                 | writes      |
 |             v                                 v             |
 |      +-------------------------------------------+          |
 |      |             stderr (FD 2)                 |          |
 |      +-------------------------------------------+          |
 |                           |                                 |
 |                           | (dup2'd to pipe)                |
 |                           v                                 |
 |               +-----------------------+                     |
 |               | sideband async thread |                     |
 |               +-----------------------+                     |
 +-------------------------------------------------------------+

When use_sideband == 0, the sideband async thread is missing, so
this same architecture just outputs via the parent stderr stream.

See the following commits for the hook API conversions doing this,
using pre-existing sideband thread logic from `copy_to_sideband`.

Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

2 files changed, 14 insertions, 2 deletions

diff --git a/hook.c b/hook.c
index fde1f88ce8..cde7198412 100644
--- a/hook.c
+++ b/hook.c
@@ -152,8 +152,8 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 		.tr2_category = "hook",
 		.tr2_label = hook_name,
 
-		.processes = 1,
-		.ungroup = 1,
+		.processes = options->jobs,
+		.ungroup = options->jobs == 1,
 
 		.get_next_task = pick_next_hook,
 		.start_failure = notify_start_failure,
@@ -169,6 +169,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
 	if (options->path_to_stdin && options->feed_pipe)
 		BUG("options path_to_stdin and feed_pipe are mutually exclusive");
 
+	if (!options->jobs)
+		BUG("run_hooks_opt must be called with options.jobs >= 1");
+
 	if (options->invoked_hook)
 		*options->invoked_hook = 0;
 
diff --git a/hook.h b/hook.h
index 2c8a23a569..20eb56fd63 100644
--- a/hook.h
+++ b/hook.h
@@ -17,6 +17,14 @@ struct run_hooks_opt
 	unsigned int error_if_missing:1;
 
 	/**
+	 *  Number of processes to parallelize across.
+	 *
+	 * If > 1, output will be buffered and de-interleaved (ungroup=0).
+	 * If == 1, output will be real-time (ungroup=1).
+	 */
+	unsigned int jobs;
+
+	/**
 	 * An optional initial working directory for the hook,
 	 * translates to "struct child_process"'s "dir" member.
 	 */
@@ -90,6 +98,7 @@ struct run_hooks_opt
 	.env = STRVEC_INIT, \
 	.args = STRVEC_INIT, \
 	.stdout_to_stderr = 1, \
+	.jobs = 1, \
 }
 
 struct hook_cb_data {