From 8872941fd21e2afe37032e7d9beec87b69aca9c9 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 19 Mar 2026 06:33:20 +0100 Subject: Introduce new "tools/" directory According to its readme, the "contrib/" directory's main intent is to collect stuff that is not an official part of Git, either because it is too specialized or because it is still considered experimental. The reality tells a bit of a different story though: while it _does_ contain such things, it also contains other things: - Our credential helpers, which are being distributed by many packagers nowadays and which can be considered "stable". - A bunch of tooling that relates to our build and test infrastructure. Especially the second category is somewhat of a sore spot. You really wouldn't expect build-related tooling to be considered an optional part of Git. Quite the opposite. Create a new top-level "tools/" directory to fix this discrepancy. This directory will contain all kind of tools that are related to our build infrastructure and that Git developers are likely to use day to day. For now, this directory doesn't contain anything yet except for a readme and a Meson skeleton. This will change in subsequent commits. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- Makefile | 2 ++ meson.build | 1 + tools/README.md | 7 +++++++ tools/meson.build | 0 4 files changed, 10 insertions(+) create mode 100644 tools/README.md create mode 100644 tools/meson.build diff --git a/Makefile b/Makefile index f3264d0a37..c7cedbcd7c 100644 --- a/Makefile +++ b/Makefile @@ -1066,11 +1066,13 @@ SOURCES_CMD = ( \ '*.sh' \ ':!*[tp][0-9][0-9][0-9][0-9]*' \ ':!contrib' \ + ':!tools' \ 2>/dev/null || \ $(FIND) . \ \( -name .git -type d -prune \) \ -o \( -name '[tp][0-9][0-9][0-9][0-9]*' -prune \) \ -o \( -name contrib -type d -prune \) \ + -o \( -name tools -type d -prune \) \ -o \( -name build -type d -prune \) \ -o \( -name .build -type d -prune \) \ -o \( -name 'trash*' -type d -prune \) \ diff --git a/meson.build b/meson.build index 4b536e0124..1d66b5181e 100644 --- a/meson.build +++ b/meson.build @@ -2149,6 +2149,7 @@ else endif subdir('contrib') +subdir('tools') # Note that the target is intentionally configured after including the # 'contrib' directory, as some tool there also have their own manpages. diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 0000000000..d732997136 --- /dev/null +++ b/tools/README.md @@ -0,0 +1,7 @@ +Developer Tooling +----------------- + +This directory is expected to contain all sorts of tooling that +relates to our build infrastructure. This includes scripts and +inputs required by our build systems, but also scripts that +developers are expected to run manually. diff --git a/tools/meson.build b/tools/meson.build new file mode 100644 index 0000000000..e69de29bb2 -- cgit v1.3 From 8ca1b4472ce97ae1d120608f9f02a86fa33d4187 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 19 Mar 2026 06:33:21 +0100 Subject: contrib: move "coccinelle/" directory into "tools/" The Coccinelle tool is an ingrained part of our build infrastructure. It is executed by our CI to detect antipatterns and is used to detect misuses of certain interfaces. It's presence in "contrib/" is thus rather misleading. Promote the configuration into the new "tools/" directory. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- Makefile | 40 ++-- ci/run-static-analysis.sh | 2 +- contrib/coccinelle/.gitignore | 1 - contrib/coccinelle/README | 124 ---------- contrib/coccinelle/array.cocci | 147 ------------ contrib/coccinelle/commit.cocci | 53 ----- contrib/coccinelle/config_fn_ctx.pending.cocci | 144 ------------ contrib/coccinelle/equals-null.cocci | 30 --- contrib/coccinelle/flex_alloc.cocci | 13 -- contrib/coccinelle/free.cocci | 45 ---- contrib/coccinelle/git_config_number.cocci | 27 --- contrib/coccinelle/hashmap.cocci | 16 -- contrib/coccinelle/index-compatibility.cocci | 157 ------------- contrib/coccinelle/meson.build | 85 ------- contrib/coccinelle/object_id.cocci | 75 ------ contrib/coccinelle/preincr.cocci | 5 - contrib/coccinelle/qsort.cocci | 37 --- contrib/coccinelle/refs.cocci | 103 --------- contrib/coccinelle/spatchcache | 304 ------------------------- contrib/coccinelle/strbuf.cocci | 62 ----- contrib/coccinelle/swap.cocci | 28 --- contrib/coccinelle/tests/free.c | 11 - contrib/coccinelle/tests/free.res | 9 - contrib/coccinelle/the_repository.cocci | 18 -- contrib/coccinelle/xcalloc.cocci | 10 - contrib/coccinelle/xopen.cocci | 19 -- contrib/coccinelle/xstrdup_or_null.cocci | 5 - contrib/coccinelle/xstrncmpz.cocci | 28 --- contrib/meson.build | 1 - tools/coccinelle/.gitignore | 1 + tools/coccinelle/README | 124 ++++++++++ tools/coccinelle/array.cocci | 147 ++++++++++++ tools/coccinelle/commit.cocci | 53 +++++ tools/coccinelle/config_fn_ctx.pending.cocci | 144 ++++++++++++ tools/coccinelle/equals-null.cocci | 30 +++ tools/coccinelle/flex_alloc.cocci | 13 ++ tools/coccinelle/free.cocci | 45 ++++ tools/coccinelle/git_config_number.cocci | 27 +++ tools/coccinelle/hashmap.cocci | 16 ++ tools/coccinelle/index-compatibility.cocci | 157 +++++++++++++ tools/coccinelle/meson.build | 85 +++++++ tools/coccinelle/object_id.cocci | 75 ++++++ tools/coccinelle/preincr.cocci | 5 + tools/coccinelle/qsort.cocci | 37 +++ tools/coccinelle/refs.cocci | 103 +++++++++ tools/coccinelle/spatchcache | 304 +++++++++++++++++++++++++ tools/coccinelle/strbuf.cocci | 62 +++++ tools/coccinelle/swap.cocci | 28 +++ tools/coccinelle/tests/free.c | 11 + tools/coccinelle/tests/free.res | 9 + tools/coccinelle/the_repository.cocci | 18 ++ tools/coccinelle/xcalloc.cocci | 10 + tools/coccinelle/xopen.cocci | 19 ++ tools/coccinelle/xstrdup_or_null.cocci | 5 + tools/coccinelle/xstrncmpz.cocci | 28 +++ tools/meson.build | 1 + 56 files changed, 1578 insertions(+), 1578 deletions(-) delete mode 100644 contrib/coccinelle/.gitignore delete mode 100644 contrib/coccinelle/README delete mode 100644 contrib/coccinelle/array.cocci delete mode 100644 contrib/coccinelle/commit.cocci delete mode 100644 contrib/coccinelle/config_fn_ctx.pending.cocci delete mode 100644 contrib/coccinelle/equals-null.cocci delete mode 100644 contrib/coccinelle/flex_alloc.cocci delete mode 100644 contrib/coccinelle/free.cocci delete mode 100644 contrib/coccinelle/git_config_number.cocci delete mode 100644 contrib/coccinelle/hashmap.cocci delete mode 100644 contrib/coccinelle/index-compatibility.cocci delete mode 100644 contrib/coccinelle/meson.build delete mode 100644 contrib/coccinelle/object_id.cocci delete mode 100644 contrib/coccinelle/preincr.cocci delete mode 100644 contrib/coccinelle/qsort.cocci delete mode 100644 contrib/coccinelle/refs.cocci delete mode 100755 contrib/coccinelle/spatchcache delete mode 100644 contrib/coccinelle/strbuf.cocci delete mode 100644 contrib/coccinelle/swap.cocci delete mode 100644 contrib/coccinelle/tests/free.c delete mode 100644 contrib/coccinelle/tests/free.res delete mode 100644 contrib/coccinelle/the_repository.cocci delete mode 100644 contrib/coccinelle/xcalloc.cocci delete mode 100644 contrib/coccinelle/xopen.cocci delete mode 100644 contrib/coccinelle/xstrdup_or_null.cocci delete mode 100644 contrib/coccinelle/xstrncmpz.cocci create mode 100644 tools/coccinelle/.gitignore create mode 100644 tools/coccinelle/README create mode 100644 tools/coccinelle/array.cocci create mode 100644 tools/coccinelle/commit.cocci create mode 100644 tools/coccinelle/config_fn_ctx.pending.cocci create mode 100644 tools/coccinelle/equals-null.cocci create mode 100644 tools/coccinelle/flex_alloc.cocci create mode 100644 tools/coccinelle/free.cocci create mode 100644 tools/coccinelle/git_config_number.cocci create mode 100644 tools/coccinelle/hashmap.cocci create mode 100644 tools/coccinelle/index-compatibility.cocci create mode 100644 tools/coccinelle/meson.build create mode 100644 tools/coccinelle/object_id.cocci create mode 100644 tools/coccinelle/preincr.cocci create mode 100644 tools/coccinelle/qsort.cocci create mode 100644 tools/coccinelle/refs.cocci create mode 100755 tools/coccinelle/spatchcache create mode 100644 tools/coccinelle/strbuf.cocci create mode 100644 tools/coccinelle/swap.cocci create mode 100644 tools/coccinelle/tests/free.c create mode 100644 tools/coccinelle/tests/free.res create mode 100644 tools/coccinelle/the_repository.cocci create mode 100644 tools/coccinelle/xcalloc.cocci create mode 100644 tools/coccinelle/xopen.cocci create mode 100644 tools/coccinelle/xstrdup_or_null.cocci create mode 100644 tools/coccinelle/xstrncmpz.cocci diff --git a/Makefile b/Makefile index c7cedbcd7c..8564b1be36 100644 --- a/Makefile +++ b/Makefile @@ -1005,8 +1005,8 @@ SPATCH_TEST_FLAGS = # COMPUTE_HEADER_DEPENDENCIES=no this will be unset too. SPATCH_USE_O_DEPENDENCIES = YesPlease -# Set SPATCH_CONCAT_COCCI to concatenate the contrib/cocci/*.cocci -# files into a single contrib/cocci/ALL.cocci before running +# Set SPATCH_CONCAT_COCCI to concatenate the tools/coccinelle/*.cocci +# files into a single tools/coccinelle/ALL.cocci before running # "coccicheck". # # Pros: @@ -1025,7 +1025,7 @@ SPATCH_USE_O_DEPENDENCIES = YesPlease # generate a specific patch, e.g. this will always use strbuf.cocci, # not ALL.cocci: # -# make contrib/coccinelle/strbuf.cocci.patch +# make tools/coccinelle/strbuf.cocci.patch SPATCH_CONCAT_COCCI = YesPlease # Rebuild 'coccicheck' if $(SPATCH), its flags etc. change @@ -3457,15 +3457,15 @@ check: exit 1; \ fi -COCCI_GEN_ALL = .build/contrib/coccinelle/ALL.cocci -COCCI_GLOB = $(wildcard contrib/coccinelle/*.cocci) +COCCI_GEN_ALL = .build/tools/coccinelle/ALL.cocci +COCCI_GLOB = $(wildcard tools/coccinelle/*.cocci) COCCI_RULES_TRACKED = $(COCCI_GLOB:%=.build/%) COCCI_RULES_TRACKED_NO_PENDING = $(filter-out %.pending.cocci,$(COCCI_RULES_TRACKED)) COCCI_RULES = COCCI_RULES += $(COCCI_GEN_ALL) COCCI_RULES += $(COCCI_RULES_TRACKED) COCCI_NAMES = -COCCI_NAMES += $(COCCI_RULES:.build/contrib/coccinelle/%.cocci=%) +COCCI_NAMES += $(COCCI_RULES:.build/tools/coccinelle/%.cocci=%) COCCICHECK_PENDING = $(filter %.pending.cocci,$(COCCI_RULES)) COCCICHECK = $(filter-out $(COCCICHECK_PENDING),$(COCCI_RULES)) @@ -3480,20 +3480,20 @@ COCCICHECK_PATCHES_PENDING_INTREE = $(COCCICHECK_PATCHES_PENDING:.build/%=%) # on $(MAKECMDGOALS) that match these $(COCCI_RULES) COCCI_RULES_GLOB = COCCI_RULES_GLOB += cocci% -COCCI_RULES_GLOB += .build/contrib/coccinelle/% +COCCI_RULES_GLOB += .build/tools/coccinelle/% COCCI_RULES_GLOB += $(COCCICHECK_PATCHES) COCCI_RULES_GLOB += $(COCCICHEC_PATCHES_PENDING) COCCI_RULES_GLOB += $(COCCICHECK_PATCHES_INTREE) COCCI_RULES_GLOB += $(COCCICHECK_PATCHES_PENDING_INTREE) COCCI_GOALS = $(filter $(COCCI_RULES_GLOB),$(MAKECMDGOALS)) -COCCI_TEST_RES = $(wildcard contrib/coccinelle/tests/*.res) +COCCI_TEST_RES = $(wildcard tools/coccinelle/tests/*.res) $(COCCI_RULES_TRACKED): .build/% : % $(call mkdir_p_parent_template) $(QUIET_CP)cp $< $@ -.build/contrib/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) +.build/tools/coccinelle/FOUND_H_SOURCES: $(FOUND_H_SOURCES) $(call mkdir_p_parent_template) $(QUIET_GEN) >$@ @@ -3507,12 +3507,12 @@ endif define cocci-rule ## Rule for .build/$(1).patch/$(2); Params: -# $(1) = e.g. ".build/contrib/coccinelle/free.cocci" +# $(1) = e.g. ".build/tools/coccinelle/free.cocci" # $(2) = e.g. "grep.c" # $(3) = e.g. "grep.o" -COCCI_$(1:.build/contrib/coccinelle/%.cocci=%) += $(1).d/$(2).patch +COCCI_$(1:.build/tools/coccinelle/%.cocci=%) += $(1).d/$(2).patch $(1).d/$(2).patch: GIT-SPATCH-DEFINES -$(1).d/$(2).patch: $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/contrib/coccinelle/FOUND_H_SOURCES) +$(1).d/$(2).patch: $(if $(and $(SPATCH_USE_O_DEPENDENCIES),$(wildcard $(3))),$(3),.build/tools/coccinelle/FOUND_H_SOURCES) $(1).d/$(2).patch: $(1) $(1).d/$(2).patch: $(1).d/%.patch : % $$(call mkdir_p_parent_template) @@ -3538,13 +3538,13 @@ endif define spatch-rule -.build/contrib/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) +.build/tools/coccinelle/$(1).cocci.patch: $$(COCCI_$(1)) $$(QUIET_SPATCH_CAT)cat $$^ >$$@ && \ if test -s $$@; \ then \ echo ' ' SPATCH result: $$@; \ fi -contrib/coccinelle/$(1).cocci.patch: .build/contrib/coccinelle/$(1).cocci.patch +tools/coccinelle/$(1).cocci.patch: .build/tools/coccinelle/$(1).cocci.patch $$(QUIET_CP)cp $$< $$@ endef @@ -3558,9 +3558,9 @@ $(COCCI_TEST_RES_GEN): GIT-SPATCH-DEFINES $(COCCI_TEST_RES_GEN): .build/%.res : %.c $(COCCI_TEST_RES_GEN): .build/%.res : %.res ifdef SPATCH_CONCAT_COCCI -$(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : $(COCCI_GEN_ALL) +$(COCCI_TEST_RES_GEN): .build/tools/coccinelle/tests/%.res : $(COCCI_GEN_ALL) else -$(COCCI_TEST_RES_GEN): .build/contrib/coccinelle/tests/%.res : contrib/coccinelle/%.cocci +$(COCCI_TEST_RES_GEN): .build/tools/coccinelle/tests/%.res : tools/coccinelle/%.cocci endif $(call mkdir_p_parent_template) $(QUIET_SPATCH_TEST)$(SPATCH) $(SPATCH_TEST_FLAGS) \ @@ -3576,14 +3576,14 @@ coccicheck-test: $(COCCI_TEST_RES_GEN) coccicheck: coccicheck-test ifdef SPATCH_CONCAT_COCCI -COCCICHECK_PATCH_MUST_BE_EMPTY_FILES = contrib/coccinelle/ALL.cocci.patch +COCCICHECK_PATCH_MUST_BE_EMPTY_FILES = tools/coccinelle/ALL.cocci.patch else COCCICHECK_PATCH_MUST_BE_EMPTY_FILES = $(COCCICHECK_PATCHES_INTREE) endif coccicheck: $(COCCICHECK_PATCH_MUST_BE_EMPTY_FILES) ! grep ^ $(COCCICHECK_PATCH_MUST_BE_EMPTY_FILES) /dev/null -# See contrib/coccinelle/README +# See tools/coccinelle/README coccicheck-pending: coccicheck-test coccicheck-pending: $(COCCICHECK_PATCHES_PENDING_INTREE) @@ -3857,8 +3857,8 @@ profile-clean: cocciclean: $(RM) GIT-SPATCH-DEFINES - $(RM) -r .build/contrib/coccinelle - $(RM) contrib/coccinelle/*.cocci.patch + $(RM) -r .build/tools/coccinelle + $(RM) tools/coccinelle/*.cocci.patch clean: profile-clean coverage-clean cocciclean $(RM) -r .build $(UNIT_TEST_BIN) diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh index 9e9c72681d..ba67e80b4d 100755 --- a/ci/run-static-analysis.sh +++ b/ci/run-static-analysis.sh @@ -10,7 +10,7 @@ make coccicheck set +x fail= -for cocci_patch in contrib/coccinelle/*.patch +for cocci_patch in tools/coccinelle/*.patch do if test -s "$cocci_patch" then diff --git a/contrib/coccinelle/.gitignore b/contrib/coccinelle/.gitignore deleted file mode 100644 index 1d45c0a40c..0000000000 --- a/contrib/coccinelle/.gitignore +++ /dev/null @@ -1 +0,0 @@ -*.patch diff --git a/contrib/coccinelle/README b/contrib/coccinelle/README deleted file mode 100644 index 055ad0e06a..0000000000 --- a/contrib/coccinelle/README +++ /dev/null @@ -1,124 +0,0 @@ -= coccinelle - -This directory provides Coccinelle (http://coccinelle.lip6.fr/) semantic patches -that might be useful to developers. - -== Types of semantic patches - - * Using the semantic transformation to check for bad patterns in the code; - The target 'make coccicheck' is designed to check for these patterns and - it is expected that any resulting patch indicates a regression. - The patches resulting from 'make coccicheck' are small and infrequent, - so once they are found, they can be sent to the mailing list as per usual. - - Example for introducing new patterns: - 67947c34ae (convert "hashcmp() != 0" to "!hasheq()", 2018-08-28) - b84c783882 (fsck: s/++i > 1/i++/, 2018-10-24) - - Example of fixes using this approach: - 248f66ed8e (run-command: use strbuf_addstr() for adding a string to - a strbuf, 2018-03-25) - f919ffebed (Use MOVE_ARRAY, 2018-01-22) - - These types of semantic patches are usually part of testing, c.f. - 0860a7641b (travis-ci: fail if Coccinelle static analysis found something - to transform, 2018-07-23) - - * Using semantic transformations in large scale refactorings throughout - the code base. - - When applying the semantic patch into a real patch, sending it to the - mailing list in the usual way, such a patch would be expected to have a - lot of textual and semantic conflicts as such large scale refactorings - change function signatures that are used widely in the code base. - A textual conflict would arise if surrounding code near any call of such - function changes. A semantic conflict arises when other patch series in - flight introduce calls to such functions. - - So to aid these large scale refactorings, semantic patches can be used. - However we do not want to store them in the same place as the checks for - bad patterns, as then automated builds would fail. - That is why semantic patches 'contrib/coccinelle/*.pending.cocci' - are ignored for checks, and can be applied using 'make coccicheck-pending'. - - This allows to expose plans of pending large scale refactorings without - impacting the bad pattern checks. - -== Git-specific tips & things to know about how we run "spatch": - - * The "make coccicheck" will piggy-back on - "COMPUTE_HEADER_DEPENDENCIES". If you've built a given object file - the "coccicheck" target will consider its depednency to decide if - it needs to re-run on the corresponding source file. - - This means that a "make coccicheck" will re-compile object files - before running. This might be unexpected, but speeds up the run in - the common case, as e.g. a change to "column.h" won't require all - coccinelle rules to be re-run against "grep.c" (or another file - that happens not to use "column.h"). - - To disable this behavior use the "SPATCH_USE_O_DEPENDENCIES=NoThanks" - flag. - - * To speed up our rules the "make coccicheck" target will by default - concatenate all of the *.cocci files here into an "ALL.cocci", and - apply it to each source file. - - This makes the run faster, as we don't need to run each rule - against each source file. See the Makefile for further discussion, - this behavior can be disabled with "SPATCH_CONCAT_COCCI=". - - But since they're concatenated any in the (e.g. "@ - my_name", v.s. anonymous "@@") needs to be unique across all our - *.cocci files. You should only need to name rules if other rules - depend on them (currently only one rule is named). - - * To speed up incremental runs even more use the "spatchcache" tool - in this directory as your "SPATCH". It aimns to be a "ccache" for - coccinelle, and piggy-backs on "COMPUTE_HEADER_DEPENDENCIES". - - It caches in Redis by default, see it source for a how-to. - - In one setup with a primed cache "make coccicheck" followed by a - "make clean && make" takes around 10s to run, but 2m30s with the - default of "SPATCH_CONCAT_COCCI=Y". - - With "SPATCH_CONCAT_COCCI=" the total runtime is around ~6m, sped - up to ~1m with "spatchcache". - - Most of the 10s (or ~1m) being spent on re-running "spatch" on - files we couldn't cache, as we didn't compile them (in contrib/* - and compat/* mostly). - - The absolute times will differ for you, but the relative speedup - from caching should be on that order. - -== Authoring and reviewing coccinelle changes - -* When a .cocci is made, both the Git changes and .cocci file should be - reviewed. When reviewing such a change, do your best to understand the .cocci - changes (e.g. by asking the author to explain the change) and be explicit - about your understanding of the changes. This helps us decide whether input - from coccinelle experts is needed or not. If you aren't sure of the cocci - changes, indicate what changes you actively endorse and leave an Acked-by - (instead of Reviewed-by). - -* Authors should consider that reviewers may not be coccinelle experts, thus the - the .cocci changes may not be self-evident. A plain text description of the - changes is strongly encouraged, especially when using more esoteric features - of the language. - -* .cocci rules should target only the problem it is trying to solve; "collateral - damage" is not allowed. Reviewers should look out and flag overly-broad rules. - -* Consider the cost-benefit ratio of .cocci changes. In particular, consider the - effect on the runtime of "make coccicheck", and how often your .cocci check - will catch something valuable. As a rule of thumb, rules that can bail early - if a file doesn't have a particular token will have a small impact on runtime, - and vice-versa. - -* .cocci files used for refactoring should be temporarily kept in-tree to aid - the refactoring of out-of-tree code (e.g. in-flight topics). Periodically - evaluate the cost-benefit ratio to determine when the file should be removed. - For example, consider how many out-of-tree users are left and how much this - slows down "make coccicheck". diff --git a/contrib/coccinelle/array.cocci b/contrib/coccinelle/array.cocci deleted file mode 100644 index e71baea00b..0000000000 --- a/contrib/coccinelle/array.cocci +++ /dev/null @@ -1,147 +0,0 @@ -@@ -type T; -T *dst_ptr; -T *src_ptr; -expression n; -@@ -- memcpy(dst_ptr, src_ptr, (n) * \( sizeof(T) -- \| sizeof(*(dst_ptr)) -- \| sizeof(*(src_ptr)) -- \| sizeof(dst_ptr[...]) -- \| sizeof(src_ptr[...]) -- \) ) -+ COPY_ARRAY(dst_ptr, src_ptr, n) - -@@ -type T; -T *dst_ptr; -T[] src_arr; -expression n; -@@ -- memcpy(dst_ptr, src_arr, (n) * \( sizeof(T) -- \| sizeof(*(dst_ptr)) -- \| sizeof(*(src_arr)) -- \| sizeof(dst_ptr[...]) -- \| sizeof(src_arr[...]) -- \) ) -+ COPY_ARRAY(dst_ptr, src_arr, n) - -@@ -type T; -T[] dst_arr; -T *src_ptr; -expression n; -@@ -- memcpy(dst_arr, src_ptr, (n) * \( sizeof(T) -- \| sizeof(*(dst_arr)) -- \| sizeof(*(src_ptr)) -- \| sizeof(dst_arr[...]) -- \| sizeof(src_ptr[...]) -- \) ) -+ COPY_ARRAY(dst_arr, src_ptr, n) - -@@ -type T; -T[] dst_arr; -T[] src_arr; -expression n; -@@ -- memcpy(dst_arr, src_arr, (n) * \( sizeof(T) -- \| sizeof(*(dst_arr)) -- \| sizeof(*(src_arr)) -- \| sizeof(dst_arr[...]) -- \| sizeof(src_arr[...]) -- \) ) -+ COPY_ARRAY(dst_arr, src_arr, n) - -@@ -type T; -T *dst; -T *src; -expression n; -@@ -( -- memmove(dst, src, (n) * sizeof(*dst)); -+ MOVE_ARRAY(dst, src, n); -| -- memmove(dst, src, (n) * sizeof(*src)); -+ MOVE_ARRAY(dst, src, n); -| -- memmove(dst, src, (n) * sizeof(T)); -+ MOVE_ARRAY(dst, src, n); -) - -@@ -type T; -T *ptr; -expression n; -@@ -- ptr = xmalloc((n) * sizeof(*ptr)); -+ ALLOC_ARRAY(ptr, n); - -@@ -type T; -T *ptr; -expression n; -@@ -- ptr = xmalloc((n) * sizeof(T)); -+ ALLOC_ARRAY(ptr, n); - -@@ -type T; -T *ptr; -expression n != 1; -@@ -- ptr = xcalloc(n, \( sizeof(*ptr) \| sizeof(T) \) ) -+ CALLOC_ARRAY(ptr, n) - -@@ -expression dst, src, n; -@@ --ALLOC_ARRAY(dst, n); --COPY_ARRAY(dst, src, n); -+DUP_ARRAY(dst, src, n); - -@@ -type T; -T *ptr; -expression n; -@@ -- memset(ptr, \( 0 \| '\0' \), \( (n) \| n \) * \( sizeof(T) -- \| sizeof(ptr[...]) -- \| sizeof(*ptr) -- \) ) -+ MEMZERO_ARRAY(ptr, n) - -@@ -type T; -T *ptr; -expression n; -@@ -- memset(ptr, \( 0 \| '\0' \), \( sizeof(T) -- \| sizeof(ptr[...]) -- \| sizeof(*ptr) -- \) * \( (n) \| n \) ) -+ MEMZERO_ARRAY(ptr, n) - -@@ -type T; -T[] ptr; -expression n; -@@ -- memset(ptr, \( 0 \| '\0' \), \( (n) \| n \) * \( sizeof(T) -- \| sizeof(ptr[...]) -- \| sizeof(*ptr) -- \) ) -+ MEMZERO_ARRAY(ptr, n) - -@@ -type T; -T[] ptr; -expression n; -@@ -- memset(ptr, \( 0 \| '\0' \), \( sizeof(T) -- \| sizeof(ptr[...]) -- \| sizeof(*ptr) -- \) * \( (n) \| n \) ) -+ MEMZERO_ARRAY(ptr, n) diff --git a/contrib/coccinelle/commit.cocci b/contrib/coccinelle/commit.cocci deleted file mode 100644 index 42725161e9..0000000000 --- a/contrib/coccinelle/commit.cocci +++ /dev/null @@ -1,53 +0,0 @@ -@@ -expression c; -@@ -- &c->maybe_tree->object.oid -+ get_commit_tree_oid(c) - -@@ -expression c; -@@ -- c->maybe_tree->object.oid.hash -+ get_commit_tree_oid(c)->hash - -@@ -identifier f !~ "^set_commit_tree$"; -expression c; -expression s; -@@ - f(...) {<... -- c->maybe_tree = s -+ set_commit_tree(c, s) - ...>} - -// These excluded functions must access c->maybe_tree directly. -// Note that if c->maybe_tree is written somewhere outside of these -// functions, then the recommended transformation will be bogus with -// repo_get_commit_tree() on the LHS. -@@ -identifier f != { repo_get_commit_tree, get_commit_tree_in_graph_one, - load_tree_for_commit, set_commit_tree, repo_parse_commit_no_graph }; -expression c; -@@ - f(...) {<... -- c->maybe_tree -+ repo_get_commit_tree(specify_the_right_repo_here, c) - ...>} - -@@ -struct commit *c; -expression E; -@@ -( -- c->generation = E; -+ commit_graph_data_at(c)->generation = E; -| -- c->graph_pos = E; -+ commit_graph_data_at(c)->graph_pos = E; -| -- c->generation -+ commit_graph_generation(c) -| -- c->graph_pos -+ commit_graph_position(c) -) diff --git a/contrib/coccinelle/config_fn_ctx.pending.cocci b/contrib/coccinelle/config_fn_ctx.pending.cocci deleted file mode 100644 index 54f09fcbcd..0000000000 --- a/contrib/coccinelle/config_fn_ctx.pending.cocci +++ /dev/null @@ -1,144 +0,0 @@ -@ get_fn @ -identifier fn, R; -@@ -( -( -git_config_from_file -| -git_config_from_file_with_options -| -git_config_from_mem -| -git_config_from_blob_oid -| -read_early_config -| -read_very_early_config -| -config_with_options -| -git_config -| -git_protected_config -| -config_from_gitmodules -) - (fn, ...) -| -repo_config(R, fn, ...) -) - -@ extends get_fn @ -identifier C1, C2, D; -@@ -int fn(const char *C1, const char *C2, -+ const struct config_context *ctx, - void *D); - -@ extends get_fn @ -@@ -int fn(const char *, const char *, -+ const struct config_context *, - void *); - -@ extends get_fn @ -// Don't change fns that look like callback fns but aren't -identifier fn2 != tar_filter_config && != git_diff_heuristic_config && - != git_default_submodule_config && != git_color_config && - != bundle_list_update && != parse_object_filter_config; -identifier C1, C2, D1, D2, S; -attribute name UNUSED; -@@ -int fn(const char *C1, const char *C2, -+ const struct config_context *ctx, - void *D1) { -<+... -( -fn2(C1, C2 -+ , ctx -, D2); -| -if(fn2(C1, C2 -+ , ctx -, D2) < 0) { ... } -| -return fn2(C1, C2 -+ , ctx -, D2); -| -S = fn2(C1, C2 -+ , ctx -, D2); -) -...+> - } - -@ extends get_fn@ -identifier C1, C2, D; -attribute name UNUSED; -@@ -int fn(const char *C1, const char *C2, -+ const struct config_context *ctx UNUSED, - void *D) {...} - - -// The previous rules don't catch all callbacks, especially if they're defined -// in a separate file from the repo_config() call. Fix these manually. -@@ -identifier C1, C2, D; -attribute name UNUSED; -@@ -int -( -git_ident_config -| -urlmatch_collect_fn -| -write_one_config -| -forbid_remote_url -| -credential_config_callback -) - (const char *C1, const char *C2, -+ const struct config_context *ctx UNUSED, - void *D) {...} - -@@ -identifier C1, C2, D, D2, S, fn2; -@@ -int -( -http_options -| -git_status_config -| -git_commit_config -| -git_default_core_config -| -grep_config -) - (const char *C1, const char *C2, -+ const struct config_context *ctx, - void *D) { -<+... -( -fn2(C1, C2 -+ , ctx -, D2); -| -if(fn2(C1, C2 -+ , ctx -, D2) < 0) { ... } -| -return fn2(C1, C2 -+ , ctx -, D2); -| -S = fn2(C1, C2 -+ , ctx -, D2); -) -...+> - } diff --git a/contrib/coccinelle/equals-null.cocci b/contrib/coccinelle/equals-null.cocci deleted file mode 100644 index 92c7054013..0000000000 --- a/contrib/coccinelle/equals-null.cocci +++ /dev/null @@ -1,30 +0,0 @@ -/* SPDX-License-Identifier: LGPL-2.1-or-later */ -@@ -expression e; -statement s; -@@ -if ( -( -!e -| -- e == NULL -+ !e -) - ) - {...} -else s - -@@ -expression e; -statement s; -@@ -if ( -( -e -| -- e != NULL -+ e -) - ) - {...} -else s diff --git a/contrib/coccinelle/flex_alloc.cocci b/contrib/coccinelle/flex_alloc.cocci deleted file mode 100644 index e9f7f6d861..0000000000 --- a/contrib/coccinelle/flex_alloc.cocci +++ /dev/null @@ -1,13 +0,0 @@ -@@ -expression str; -identifier x, flexname; -@@ -- FLEX_ALLOC_MEM(x, flexname, str, strlen(str)); -+ FLEX_ALLOC_STR(x, flexname, str); - -@@ -expression str; -identifier x, ptrname; -@@ -- FLEXPTR_ALLOC_MEM(x, ptrname, str, strlen(str)); -+ FLEXPTR_ALLOC_STR(x, ptrname, str); diff --git a/contrib/coccinelle/free.cocci b/contrib/coccinelle/free.cocci deleted file mode 100644 index 03799e1908..0000000000 --- a/contrib/coccinelle/free.cocci +++ /dev/null @@ -1,45 +0,0 @@ -@@ -expression E; -@@ -- if (E) -( - free(E); -| - commit_list_free(E); -) - -@@ -expression E; -@@ -- if (!E) -( - free(E); -| - commit_list_free(E); -) - -@@ -expression E; -@@ -- free(E); -+ FREE_AND_NULL(E); -- E = NULL; - -@@ -expression E; -@@ -- if (E) -- { - commit_list_free(E); - E = NULL; -- } - -@@ -expression E; -statement S; -@@ -- if (E) { -+ if (E) - S - commit_list_free(E); -- } diff --git a/contrib/coccinelle/git_config_number.cocci b/contrib/coccinelle/git_config_number.cocci deleted file mode 100644 index 7b57dceefe..0000000000 --- a/contrib/coccinelle/git_config_number.cocci +++ /dev/null @@ -1,27 +0,0 @@ -@@ -identifier C1, C2, C3; -@@ -( -( -git_config_int -| -git_config_int64 -| -git_config_ulong -| -git_config_ssize_t -) - (C1, C2 -+ , ctx->kvi - ) -| -( -git_configset_get_value -| -git_config_bool_or_int -) - (C1, C2 -+ , ctx->kvi - , C3 - ) -) diff --git a/contrib/coccinelle/hashmap.cocci b/contrib/coccinelle/hashmap.cocci deleted file mode 100644 index c5dbb4557b..0000000000 --- a/contrib/coccinelle/hashmap.cocci +++ /dev/null @@ -1,16 +0,0 @@ -@@ -expression E; -struct hashmap_entry HME; -@@ -- HME.hash = E; -+ hashmap_entry_init(&HME, E); - -@@ -identifier f !~ "^hashmap_entry_init$"; -expression E; -struct hashmap_entry *HMEP; -@@ - f(...) {<... -- HMEP->hash = E; -+ hashmap_entry_init(HMEP, E); - ...>} diff --git a/contrib/coccinelle/index-compatibility.cocci b/contrib/coccinelle/index-compatibility.cocci deleted file mode 100644 index 31e36cf3c4..0000000000 --- a/contrib/coccinelle/index-compatibility.cocci +++ /dev/null @@ -1,157 +0,0 @@ -// the_index.* variables -@@ -identifier AC = active_cache; -identifier AN = active_nr; -identifier ACC = active_cache_changed; -identifier ACT = active_cache_tree; -@@ -( -- AC -+ the_index.cache -| -- AN -+ the_index.cache_nr -| -- ACC -+ the_index.cache_changed -| -- ACT -+ the_index.cache_tree -) - -// "the_repository" simple cases -@@ -@@ -( -- read_cache -+ repo_read_index -| -- read_cache_unmerged -+ repo_read_index_unmerged -| -- hold_locked_index -+ repo_hold_locked_index -) - ( -+ the_repository, - ...) - -// "the_repository" special-cases -@@ -@@ -( -- read_cache_preload -+ repo_read_index_preload -) - ( -+ the_repository, - ... -+ , 0 - ) - -// "the_index" simple cases -@@ -@@ -( -- is_cache_unborn -+ is_index_unborn -| -- unmerged_cache -+ unmerged_index -| -- rename_cache_entry_at -+ rename_index_entry_at -| -- chmod_cache_entry -+ chmod_index_entry -| -- cache_file_exists -+ index_file_exists -| -- cache_name_is_other -+ index_name_is_other -| -- unmerge_cache_entry_at -+ unmerge_index_entry_at -| -- add_to_cache -+ add_to_index -| -- add_file_to_cache -+ add_file_to_index -| -- add_cache_entry -+ add_index_entry -| -- remove_file_from_cache -+ remove_file_from_index -| -- ce_match_stat -+ ie_match_stat -| -- ce_modified -+ ie_modified -| -- resolve_undo_clear -+ resolve_undo_clear_index -| -- cache_name_pos -+ index_name_pos -| -- update_main_cache_tree -+ cache_tree_update -| -- discard_cache -+ discard_index -) - ( -+ &the_index, - ...) - -@@ -@@ -( -- refresh_and_write_cache -+ repo_refresh_and_write_index -) - ( -+ the_repository, - ... -+ , NULL, NULL, NULL - ) - -// "the_index" special-cases -@@ -@@ -( -- read_cache_from -+ read_index_from -) - ( -+ &the_index, - ... -+ , get_git_dir() - ) - -@@ -@@ -( -- refresh_cache -+ refresh_index -) - ( -+ &the_index, - ... -+ , NULL, NULL, NULL - ) - -@@ -expression O; -@@ -- write_cache_as_tree -+ write_index_as_tree - ( -- O, -+ O, &the_index, get_index_file(), - ... - ) diff --git a/contrib/coccinelle/meson.build b/contrib/coccinelle/meson.build deleted file mode 100644 index ae7f5b5460..0000000000 --- a/contrib/coccinelle/meson.build +++ /dev/null @@ -1,85 +0,0 @@ -coccinelle_opt = get_option('coccinelle').require( - fs.exists(meson.project_source_root() / '.git'), - error_message: 'coccinelle can only be run from a git checkout', -) - -spatch = find_program('spatch', required: coccinelle_opt) -if not spatch.found() - subdir_done() -endif - -rules = [ - 'array.cocci', - 'commit.cocci', - 'config_fn_ctx.pending.cocci', - 'equals-null.cocci', - 'flex_alloc.cocci', - 'free.cocci', - 'git_config_number.cocci', - 'hashmap.cocci', - 'index-compatibility.cocci', - 'object_id.cocci', - 'preincr.cocci', - 'qsort.cocci', - 'refs.cocci', - 'strbuf.cocci', - 'swap.cocci', - 'the_repository.cocci', - 'xcalloc.cocci', - 'xopen.cocci', - 'xstrdup_or_null.cocci', - 'xstrncmpz.cocci', -] - -concatenated_rules = custom_target( - command: [ - 'cat', '@INPUT@', - ], - input: rules, - output: 'rules.cocci', - capture: true, -) - -coccinelle_sources = [] -foreach source : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.c', third_party_excludes, check: true).stdout().split() - coccinelle_sources += source -endforeach - -coccinelle_headers = [] -foreach header : headers_to_check - coccinelle_headers += meson.project_source_root() / header -endforeach - -coccinelle_includes = [] -foreach path : ['compat', 'ewah', 'refs', 'sha256', 'trace2', 'win32', 'xdiff'] - coccinelle_includes += ['-I', meson.project_source_root() / path] -endforeach - -patches = [ ] -foreach source : coccinelle_sources - patches += custom_target( - command: [ - spatch, - '--all-includes', - '--sp-file', concatenated_rules, - '--patch', meson.project_source_root(), - coccinelle_includes, - '@INPUT@', - ], - input: meson.project_source_root() / source, - output: source.underscorify() + '.patch', - capture: true, - depend_files: coccinelle_headers, - ) -endforeach - -concatenated_patch = custom_target( - command: [ - 'cat', '@INPUT@', - ], - input: patches, - output: 'cocci.patch', - capture: true, -) - -alias_target('coccicheck', concatenated_patch) diff --git a/contrib/coccinelle/object_id.cocci b/contrib/coccinelle/object_id.cocci deleted file mode 100644 index 01f8d6935b..0000000000 --- a/contrib/coccinelle/object_id.cocci +++ /dev/null @@ -1,75 +0,0 @@ -@@ -struct object_id OID; -@@ -- hashclr(OID.hash) -+ oidclr(&OID) - -@@ -identifier f != oidclr; -struct object_id *OIDPTR; -@@ - f(...) {<... -- hashclr(OIDPTR->hash) -+ oidclr(OIDPTR) - ...>} - -@@ -struct object_id OID1, OID2; -@@ -- hashcmp(OID1.hash, OID2.hash) -+ oidcmp(&OID1, &OID2) - -@@ -identifier f != oidcmp; -struct object_id *OIDPTR1, OIDPTR2; -@@ - f(...) {<... -- hashcmp(OIDPTR1->hash, OIDPTR2->hash) -+ oidcmp(OIDPTR1, OIDPTR2) - ...>} - -@@ -struct object_id *OIDPTR; -struct object_id OID; -@@ -- hashcmp(OIDPTR->hash, OID.hash) -+ oidcmp(OIDPTR, &OID) - -@@ -struct object_id *OIDPTR; -struct object_id OID; -@@ -- hashcmp(OID.hash, OIDPTR->hash) -+ oidcmp(&OID, OIDPTR) - -@@ -struct object_id *OIDPTR1; -struct object_id *OIDPTR2; -@@ -- oidcmp(OIDPTR1, OIDPTR2) == 0 -+ oideq(OIDPTR1, OIDPTR2) - -@@ -identifier f != hasheq; -expression E1, E2; -@@ - f(...) {<... -- hashcmp(E1, E2) == 0 -+ hasheq(E1, E2) - ...>} - -@@ -struct object_id *OIDPTR1; -struct object_id *OIDPTR2; -@@ -- oidcmp(OIDPTR1, OIDPTR2) != 0 -+ !oideq(OIDPTR1, OIDPTR2) - -@@ -identifier f != hasheq; -expression E1, E2; -@@ - f(...) {<... -- hashcmp(E1, E2) != 0 -+ !hasheq(E1, E2) - ...>} diff --git a/contrib/coccinelle/preincr.cocci b/contrib/coccinelle/preincr.cocci deleted file mode 100644 index ae42cb0730..0000000000 --- a/contrib/coccinelle/preincr.cocci +++ /dev/null @@ -1,5 +0,0 @@ -@@ -identifier i; -@@ -- ++i > 1 -+ i++ diff --git a/contrib/coccinelle/qsort.cocci b/contrib/coccinelle/qsort.cocci deleted file mode 100644 index 22b93a9966..0000000000 --- a/contrib/coccinelle/qsort.cocci +++ /dev/null @@ -1,37 +0,0 @@ -@@ -expression base, nmemb, compar; -@@ -- qsort(base, nmemb, sizeof(*base), compar); -+ QSORT(base, nmemb, compar); - -@@ -expression base, nmemb, compar; -@@ -- qsort(base, nmemb, sizeof(base[0]), compar); -+ QSORT(base, nmemb, compar); - -@@ -type T; -T *base; -expression nmemb, compar; -@@ -- qsort(base, nmemb, sizeof(T), compar); -+ QSORT(base, nmemb, compar); - -@@ -expression base, nmemb, compar; -@@ -- if (nmemb) - QSORT(base, nmemb, compar); - -@@ -expression base, nmemb, compar; -@@ -- if (nmemb > 0) - QSORT(base, nmemb, compar); - -@@ -expression base, nmemb, compar; -@@ -- if (nmemb > 1) - QSORT(base, nmemb, compar); diff --git a/contrib/coccinelle/refs.cocci b/contrib/coccinelle/refs.cocci deleted file mode 100644 index 31d9cad8f3..0000000000 --- a/contrib/coccinelle/refs.cocci +++ /dev/null @@ -1,103 +0,0 @@ -// Migrate "refs.h" to not rely on `the_repository` implicitly anymore. -@@ -@@ -( -- resolve_ref_unsafe -+ refs_resolve_ref_unsafe -| -- resolve_refdup -+ refs_resolve_refdup -| -- read_ref_full -+ refs_read_ref_full -| -- read_ref -+ refs_read_ref -| -- ref_exists -+ refs_ref_exists -| -- head_ref -+ refs_head_ref -| -- for_each_ref -+ refs_for_each_ref -| -- for_each_ref_in -+ refs_for_each_ref_in -| -- for_each_fullref_in -+ refs_for_each_fullref_in -| -- for_each_tag_ref -+ refs_for_each_tag_ref -| -- for_each_branch_ref -+ refs_for_each_branch_ref -| -- for_each_remote_ref -+ refs_for_each_remote_ref -| -- for_each_glob_ref -+ refs_for_each_glob_ref -| -- for_each_glob_ref_in -+ refs_for_each_glob_ref_in -| -- head_ref_namespaced -+ refs_head_ref_namespaced -| -- for_each_namespaced_ref -+ refs_for_each_namespaced_ref -| -- for_each_rawref -+ refs_for_each_rawref -| -- safe_create_reflog -+ refs_create_reflog -| -- reflog_exists -+ refs_reflog_exists -| -- delete_ref -+ refs_delete_ref -| -- delete_refs -+ refs_delete_refs -| -- delete_reflog -+ refs_delete_reflog -| -- for_each_reflog_ent -+ refs_for_each_reflog_ent -| -- for_each_reflog_ent_reverse -+ refs_for_each_reflog_ent_reverse -| -- for_each_reflog -+ refs_for_each_reflog -| -- shorten_unambiguous_ref -+ refs_shorten_unambiguous_ref -| -- rename_ref -+ refs_rename_ref -| -- copy_existing_ref -+ refs_copy_existing_ref -| -- create_symref -+ refs_create_symref -| -- ref_transaction_begin -+ ref_store_transaction_begin -| -- update_ref -+ refs_update_ref -| -- reflog_expire -+ refs_reflog_expire -) - ( -+ get_main_ref_store(the_repository), - ...) diff --git a/contrib/coccinelle/spatchcache b/contrib/coccinelle/spatchcache deleted file mode 100755 index 29e9352d8a..0000000000 --- a/contrib/coccinelle/spatchcache +++ /dev/null @@ -1,304 +0,0 @@ -#!/bin/sh -# -# spatchcache: a poor-man's "ccache"-alike for "spatch" in git.git -# -# This caching command relies on the peculiarities of the Makefile -# driving "spatch" in git.git, in particular if we invoke: -# -# make -# # See "spatchCache.cacheWhenStderr" for why "--very-quiet" is -# # used -# make coccicheck SPATCH_FLAGS=--very-quiet -# -# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with -# "gcc" and "clang") write e.g. a .depend/grep.o.d for grep.c, when we -# compile grep.o. -# -# The .depend/grep.o.d will have the full header dependency tree of -# grep.c, and we can thus cache the output of "spatch" by: -# -# 1. Hashing all of those files -# 2. Hashing our source file, and the *.cocci rule we're -# applying -# 3. Running spatch, if suggests no changes (by far the common -# case) we invoke "spatchCache.getCmd" and -# "spatchCache.setCmd" with a hash SHA-256 to ask "does this -# ID have no changes" or "say that ID had no changes> -# 4. If no "spatchCache.{set,get}Cmd" is specified we'll use -# "redis-cli" and maintain a SET called "spatch-cache". Set -# appropriate redis memory policies to keep it from growing -# out of control. -# -# This along with the general incremental "make" support for -# "contrib/coccinelle" makes it viable to (re-)run coccicheck -# e.g. when merging integration branches. -# -# Note that the "--very-quiet" flag is currently critical. The cache -# will refuse to cache anything that has output on STDERR (which might -# be errors from spatch), but see spatchCache.cacheWhenStderr below. -# -# The STDERR (and exit code) could in principle be cached (as with -# ccache), but then the simple structure in the Redis cache would need -# to change, so just supply "--very-quiet" for now. -# -# To use this, simply set SPATCH to -# contrib/coccinelle/spatchcache. Then optionally set: -# -# [spatchCache] -# # Optional: path to a custom spatch -# spatch = ~/g/coccicheck/spatch.opt -# -# As well as this trace config (debug implies trace): -# -# cacheWhenStderr = true -# trace = false -# debug = false -# -# The ".depend/grep.o.d" can also be customized, as a string that will -# be eval'd, it has access to a "$dirname" and "$basename": -# -# [spatchCache] -# dependFormat = "$dirname/.depend/${basename%.c}.o.d" -# -# Setting "trace" to "true" allows for seeing when we have a cache HIT -# or MISS. To debug whether the cache is working do that, and run e.g.: -# -# redis-cli FLUSHALL -# -# grep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/contrib/coccinelle | sort | uniq -c -# 600 CANTCACHE -# 7365 MISS -# 7365 SET -# -# A subsequent "make cocciclean && make coccicheck" should then have -# all "HIT"'s and "CANTCACHE"'s. -# -# The "spatchCache.cacheWhenStderr" option is critical when using -# spatchCache.{trace,debug} to debug whether something is set in the -# cache, as we'll write to the spatch logs in .build/* we'd otherwise -# always emit a NOCACHE. -# -# Reading the config can make the command much slower, to work around -# this the config can be set in the environment, with environment -# variable name corresponding to the config key. "default" can be used -# to use whatever's the script default, e.g. setting -# spatchCache.cacheWhenStderr=true and deferring to the defaults for -# the rest is: -# -# export GIT_CONTRIB_SPATCHCACHE_DEBUG=default -# export GIT_CONTRIB_SPATCHCACHE_TRACE=default -# export GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR=true -# export GIT_CONTRIB_SPATCHCACHE_SPATCH=default -# export GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT=default -# export GIT_CONTRIB_SPATCHCACHE_SETCMD=default -# export GIT_CONTRIB_SPATCHCACHE_GETCMD=default - -set -e - -env_or_config () { - env="$1" - shift - if test "$env" = "default" - then - # Avoid expensive "git config" invocation - return - elif test -n "$env" - then - echo "$env" - else - git config $@ || : - fi -} - -## Our own configuration & options -debug=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_DEBUG" --bool "spatchCache.debug") -if test "$debug" != "true" -then - debug= -fi -if test -n "$debug" -then - set -x -fi - -trace=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_TRACE" --bool "spatchCache.trace") -if test "$trace" != "true" -then - trace= -fi -if test -n "$debug" -then - # debug implies trace - trace=true -fi - -cacheWhenStderr=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR" --bool "spatchCache.cacheWhenStderr") -if test "$cacheWhenStderr" != "true" -then - cacheWhenStderr= -fi - -trace_it () { - if test -z "$trace" - then - return - fi - echo "$@" >&2 -} - -spatch=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_SPATCH" --path "spatchCache.spatch") -if test -n "$spatch" -then - if test -n "$debug" - then - trace_it "custom spatchCache.spatch='$spatch'" - fi -else - spatch=spatch -fi - -dependFormat='$dirname/.depend/${basename%.c}.o.d' -dependFormatCfg=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT" "spatchCache.dependFormat") -if test -n "$dependFormatCfg" -then - dependFormat="$dependFormatCfg" -fi - -set=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_SETCMD" "spatchCache.setCmd") -get=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_GETCMD" "spatchCache.getCmd") - -## Parse spatch()-like command-line for caching info -arg_sp= -arg_file= -args="$@" -spatch_opts() { - while test $# != 0 - do - arg_file="$1" - case "$1" in - --sp-file) - arg_sp="$2" - ;; - esac - shift - done -} -spatch_opts "$@" -if ! test -f "$arg_file" -then - arg_file= -fi - -hash_for_cache() { - # Parameters that should affect the cache - echo "args=$args" - echo "config spatchCache.spatch=$spatch" - echo "config spatchCache.debug=$debug" - echo "config spatchCache.trace=$trace" - echo "config spatchCache.cacheWhenStderr=$cacheWhenStderr" - echo - - # Our target file and its dependencies - git hash-object "$1" "$2" $(grep -E -o '^[^:]+:$' "$3" | tr -d ':') -} - -# Sanity checks -if ! test -f "$arg_sp" && ! test -f "$arg_file" -then - echo $0: no idea how to cache "$@" >&2 - exit 128 -fi - -# Main logic -dirname=$(dirname "$arg_file") -basename=$(basename "$arg_file") -eval "dep=$dependFormat" - -if ! test -f "$dep" -then - trace_it "$0: CANTCACHE have no '$dep' for '$arg_file'!" - exec "$spatch" "$@" -fi - -if test -n "$debug" -then - trace_it "$0: The full cache input for '$arg_sp' '$arg_file' '$dep'" - hash_for_cache "$arg_sp" "$arg_file" "$dep" >&2 -fi -sum=$(hash_for_cache "$arg_sp" "$arg_file" "$dep" | git hash-object --stdin) - -trace_it "$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'" - -getret= -if test -z "$get" -then - if test $(redis-cli SISMEMBER spatch-cache "$sum") = 1 - then - getret=0 - else - getret=1 - fi -else - $set "$sum" - getret=$? -fi - -if test "$getret" = 0 -then - trace_it "$0: HIT for '$arg_file' with '$arg_sp'" - exit 0 -else - trace_it "$0: MISS: for '$arg_file' with '$arg_sp'" -fi - -out="$(mktemp)" -err="$(mktemp)" - -set +e -"$spatch" "$@" >"$out" 2>>"$err" -ret=$? -cat "$out" -cat "$err" >&2 -set -e - -nocache= -if test $ret != 0 -then - nocache="exited non-zero: $ret" -elif test -s "$out" -then - nocache="had patch output" -elif test -z "$cacheWhenStderr" && test -s "$err" -then - nocache="had stderr (use --very-quiet or spatchCache.cacheWhenStderr=true?)" -fi - -if test -n "$nocache" -then - trace_it "$0: NOCACHE ($nocache): for '$arg_file' with '$arg_sp'" - exit "$ret" -fi - -trace_it "$0: SET: for '$arg_file' with '$arg_sp'" - -setret= -if test -z "$set" -then - if test $(redis-cli SADD spatch-cache "$sum") = 1 - then - setret=0 - else - setret=1 - fi -else - "$set" "$sum" - setret=$? -fi - -if test "$setret" != 0 -then - echo "FAILED to set '$sum' in cache!" >&2 - exit 128 -fi - -exit "$ret" diff --git a/contrib/coccinelle/strbuf.cocci b/contrib/coccinelle/strbuf.cocci deleted file mode 100644 index 5f06105df6..0000000000 --- a/contrib/coccinelle/strbuf.cocci +++ /dev/null @@ -1,62 +0,0 @@ -@@ -expression E; -constant fmt !~ "%"; -@@ -- strbuf_addf -+ strbuf_addstr - (E, -( - fmt -| - _(fmt) -) - ); - -@@ -expression E; -struct strbuf SB; -format F =~ "^s$"; -@@ -- strbuf_addf(E, "%@F@", SB.buf); -+ strbuf_addbuf(E, &SB); - -@@ -expression E; -struct strbuf *SBP; -format F =~ "^s$"; -@@ -- strbuf_addf(E, "%@F@", SBP->buf); -+ strbuf_addbuf(E, SBP); - -@@ -expression E; -struct strbuf SB; -@@ -- strbuf_addstr(E, SB.buf); -+ strbuf_addbuf(E, &SB); - -@@ -expression E; -struct strbuf *SBP; -@@ -- strbuf_addstr(E, SBP->buf); -+ strbuf_addbuf(E, SBP); - -@@ -expression E1, E2; -format F =~ "^s$"; -@@ -- strbuf_addf(E1, "%@F@", E2); -+ strbuf_addstr(E1, E2); - -@@ -expression E1, E2, E3; -@@ -- strbuf_addstr(E1, find_unique_abbrev(E2, E3)); -+ strbuf_add_unique_abbrev(E1, E2, E3); - -@@ -expression E1, E2; -@@ -- strbuf_addstr(E1, real_path(E2)); -+ strbuf_add_real_path(E1, E2); diff --git a/contrib/coccinelle/swap.cocci b/contrib/coccinelle/swap.cocci deleted file mode 100644 index 522177afb6..0000000000 --- a/contrib/coccinelle/swap.cocci +++ /dev/null @@ -1,28 +0,0 @@ -@@ -type T; -identifier tmp; -T a, b; -@@ -- T tmp = a; -+ T tmp; -+ tmp = a; - a = b; - b = tmp; - -@ swap @ -type T; -T tmp, a, b; -@@ -- tmp = a; -- a = b; -- b = tmp; -+ SWAP(a, b); - -@ extends swap @ -identifier unused; -@@ - { - ... -- T unused; - ... when != unused - } diff --git a/contrib/coccinelle/tests/free.c b/contrib/coccinelle/tests/free.c deleted file mode 100644 index 96d4abc0c7..0000000000 --- a/contrib/coccinelle/tests/free.c +++ /dev/null @@ -1,11 +0,0 @@ -int use_FREE_AND_NULL(int *v) -{ - free(*v); - *v = NULL; -} - -int need_no_if(int *v) -{ - if (v) - free(v); -} diff --git a/contrib/coccinelle/tests/free.res b/contrib/coccinelle/tests/free.res deleted file mode 100644 index f90fd9f48e..0000000000 --- a/contrib/coccinelle/tests/free.res +++ /dev/null @@ -1,9 +0,0 @@ -int use_FREE_AND_NULL(int *v) -{ - FREE_AND_NULL(*v); -} - -int need_no_if(int *v) -{ - free(v); -} diff --git a/contrib/coccinelle/the_repository.cocci b/contrib/coccinelle/the_repository.cocci deleted file mode 100644 index f1129f7985..0000000000 --- a/contrib/coccinelle/the_repository.cocci +++ /dev/null @@ -1,18 +0,0 @@ -// Fully migrated "the_repository" additions -@@ -@@ -( -// TODO: remove the rules below and the macros from tree.h after the -// next Git release. -- parse_tree -+ repo_parse_tree -| -- parse_tree_gently -+ repo_parse_tree_gently -| -- parse_tree_indirect -+ repo_parse_tree_indirect -) - ( -+ the_repository, - ...) diff --git a/contrib/coccinelle/xcalloc.cocci b/contrib/coccinelle/xcalloc.cocci deleted file mode 100644 index c291011607..0000000000 --- a/contrib/coccinelle/xcalloc.cocci +++ /dev/null @@ -1,10 +0,0 @@ -@@ -type T; -T *ptr; -expression n; -@@ - xcalloc( -+ n, - \( sizeof(T) \| sizeof(*ptr) \) -- , n - ) diff --git a/contrib/coccinelle/xopen.cocci b/contrib/coccinelle/xopen.cocci deleted file mode 100644 index b71db67019..0000000000 --- a/contrib/coccinelle/xopen.cocci +++ /dev/null @@ -1,19 +0,0 @@ -@@ -identifier fd; -identifier die_fn =~ "^(die|die_errno)$"; -@@ - int fd = -- open -+ xopen - (...); -- if ( \( fd < 0 \| fd == -1 \) ) { die_fn(...); } - -@@ -expression fd; -identifier die_fn =~ "^(die|die_errno)$"; -@@ - fd = -- open -+ xopen - (...); -- if ( \( fd < 0 \| fd == -1 \) ) { die_fn(...); } diff --git a/contrib/coccinelle/xstrdup_or_null.cocci b/contrib/coccinelle/xstrdup_or_null.cocci deleted file mode 100644 index 9c1d2939b6..0000000000 --- a/contrib/coccinelle/xstrdup_or_null.cocci +++ /dev/null @@ -1,5 +0,0 @@ -@@ -expression E; -@@ -- xstrdup(absolute_path(E)) -+ absolute_pathdup(E) diff --git a/contrib/coccinelle/xstrncmpz.cocci b/contrib/coccinelle/xstrncmpz.cocci deleted file mode 100644 index ccb39e2bc0..0000000000 --- a/contrib/coccinelle/xstrncmpz.cocci +++ /dev/null @@ -1,28 +0,0 @@ -@@ -expression S, T, L; -@@ -( -- strncmp(S, T, L) || S[L] -+ !!xstrncmpz(S, T, L) -| -- strncmp(S, T, L) || S[L] != '\0' -+ !!xstrncmpz(S, T, L) -| -- strncmp(S, T, L) || T[L] -+ !!xstrncmpz(T, S, L) -| -- strncmp(S, T, L) || T[L] != '\0' -+ !!xstrncmpz(T, S, L) -| -- !strncmp(S, T, L) && !S[L] -+ !xstrncmpz(S, T, L) -| -- !strncmp(S, T, L) && S[L] == '\0' -+ !xstrncmpz(S, T, L) -| -- !strncmp(S, T, L) && !T[L] -+ !xstrncmpz(T, S, L) -| -- !strncmp(S, T, L) && T[L] == '\0' -+ !xstrncmpz(T, S, L) -) diff --git a/contrib/meson.build b/contrib/meson.build index a88c5dfe09..569c23ee76 100644 --- a/contrib/meson.build +++ b/contrib/meson.build @@ -2,5 +2,4 @@ foreach feature : get_option('contrib') subdir(feature) endforeach -subdir('coccinelle') subdir('credential') diff --git a/tools/coccinelle/.gitignore b/tools/coccinelle/.gitignore new file mode 100644 index 0000000000..1d45c0a40c --- /dev/null +++ b/tools/coccinelle/.gitignore @@ -0,0 +1 @@ +*.patch diff --git a/tools/coccinelle/README b/tools/coccinelle/README new file mode 100644 index 0000000000..fd0a543cc2 --- /dev/null +++ b/tools/coccinelle/README @@ -0,0 +1,124 @@ += coccinelle + +This directory provides Coccinelle (http://coccinelle.lip6.fr/) semantic patches +that might be useful to developers. + +== Types of semantic patches + + * Using the semantic transformation to check for bad patterns in the code; + The target 'make coccicheck' is designed to check for these patterns and + it is expected that any resulting patch indicates a regression. + The patches resulting from 'make coccicheck' are small and infrequent, + so once they are found, they can be sent to the mailing list as per usual. + + Example for introducing new patterns: + 67947c34ae (convert "hashcmp() != 0" to "!hasheq()", 2018-08-28) + b84c783882 (fsck: s/++i > 1/i++/, 2018-10-24) + + Example of fixes using this approach: + 248f66ed8e (run-command: use strbuf_addstr() for adding a string to + a strbuf, 2018-03-25) + f919ffebed (Use MOVE_ARRAY, 2018-01-22) + + These types of semantic patches are usually part of testing, c.f. + 0860a7641b (travis-ci: fail if Coccinelle static analysis found something + to transform, 2018-07-23) + + * Using semantic transformations in large scale refactorings throughout + the code base. + + When applying the semantic patch into a real patch, sending it to the + mailing list in the usual way, such a patch would be expected to have a + lot of textual and semantic conflicts as such large scale refactorings + change function signatures that are used widely in the code base. + A textual conflict would arise if surrounding code near any call of such + function changes. A semantic conflict arises when other patch series in + flight introduce calls to such functions. + + So to aid these large scale refactorings, semantic patches can be used. + However we do not want to store them in the same place as the checks for + bad patterns, as then automated builds would fail. + That is why semantic patches 'tools/coccinelle/*.pending.cocci' + are ignored for checks, and can be applied using 'make coccicheck-pending'. + + This allows to expose plans of pending large scale refactorings without + impacting the bad pattern checks. + +== Git-specific tips & things to know about how we run "spatch": + + * The "make coccicheck" will piggy-back on + "COMPUTE_HEADER_DEPENDENCIES". If you've built a given object file + the "coccicheck" target will consider its depednency to decide if + it needs to re-run on the corresponding source file. + + This means that a "make coccicheck" will re-compile object files + before running. This might be unexpected, but speeds up the run in + the common case, as e.g. a change to "column.h" won't require all + coccinelle rules to be re-run against "grep.c" (or another file + that happens not to use "column.h"). + + To disable this behavior use the "SPATCH_USE_O_DEPENDENCIES=NoThanks" + flag. + + * To speed up our rules the "make coccicheck" target will by default + concatenate all of the *.cocci files here into an "ALL.cocci", and + apply it to each source file. + + This makes the run faster, as we don't need to run each rule + against each source file. See the Makefile for further discussion, + this behavior can be disabled with "SPATCH_CONCAT_COCCI=". + + But since they're concatenated any in the (e.g. "@ + my_name", v.s. anonymous "@@") needs to be unique across all our + *.cocci files. You should only need to name rules if other rules + depend on them (currently only one rule is named). + + * To speed up incremental runs even more use the "spatchcache" tool + in this directory as your "SPATCH". It aimns to be a "ccache" for + coccinelle, and piggy-backs on "COMPUTE_HEADER_DEPENDENCIES". + + It caches in Redis by default, see it source for a how-to. + + In one setup with a primed cache "make coccicheck" followed by a + "make clean && make" takes around 10s to run, but 2m30s with the + default of "SPATCH_CONCAT_COCCI=Y". + + With "SPATCH_CONCAT_COCCI=" the total runtime is around ~6m, sped + up to ~1m with "spatchcache". + + Most of the 10s (or ~1m) being spent on re-running "spatch" on + files we couldn't cache, as we didn't compile them (in contrib/* + and compat/* mostly). + + The absolute times will differ for you, but the relative speedup + from caching should be on that order. + +== Authoring and reviewing coccinelle changes + +* When a .cocci is made, both the Git changes and .cocci file should be + reviewed. When reviewing such a change, do your best to understand the .cocci + changes (e.g. by asking the author to explain the change) and be explicit + about your understanding of the changes. This helps us decide whether input + from coccinelle experts is needed or not. If you aren't sure of the cocci + changes, indicate what changes you actively endorse and leave an Acked-by + (instead of Reviewed-by). + +* Authors should consider that reviewers may not be coccinelle experts, thus the + the .cocci changes may not be self-evident. A plain text description of the + changes is strongly encouraged, especially when using more esoteric features + of the language. + +* .cocci rules should target only the problem it is trying to solve; "collateral + damage" is not allowed. Reviewers should look out and flag overly-broad rules. + +* Consider the cost-benefit ratio of .cocci changes. In particular, consider the + effect on the runtime of "make coccicheck", and how often your .cocci check + will catch something valuable. As a rule of thumb, rules that can bail early + if a file doesn't have a particular token will have a small impact on runtime, + and vice-versa. + +* .cocci files used for refactoring should be temporarily kept in-tree to aid + the refactoring of out-of-tree code (e.g. in-flight topics). Periodically + evaluate the cost-benefit ratio to determine when the file should be removed. + For example, consider how many out-of-tree users are left and how much this + slows down "make coccicheck". diff --git a/tools/coccinelle/array.cocci b/tools/coccinelle/array.cocci new file mode 100644 index 0000000000..e71baea00b --- /dev/null +++ b/tools/coccinelle/array.cocci @@ -0,0 +1,147 @@ +@@ +type T; +T *dst_ptr; +T *src_ptr; +expression n; +@@ +- memcpy(dst_ptr, src_ptr, (n) * \( sizeof(T) +- \| sizeof(*(dst_ptr)) +- \| sizeof(*(src_ptr)) +- \| sizeof(dst_ptr[...]) +- \| sizeof(src_ptr[...]) +- \) ) ++ COPY_ARRAY(dst_ptr, src_ptr, n) + +@@ +type T; +T *dst_ptr; +T[] src_arr; +expression n; +@@ +- memcpy(dst_ptr, src_arr, (n) * \( sizeof(T) +- \| sizeof(*(dst_ptr)) +- \| sizeof(*(src_arr)) +- \| sizeof(dst_ptr[...]) +- \| sizeof(src_arr[...]) +- \) ) ++ COPY_ARRAY(dst_ptr, src_arr, n) + +@@ +type T; +T[] dst_arr; +T *src_ptr; +expression n; +@@ +- memcpy(dst_arr, src_ptr, (n) * \( sizeof(T) +- \| sizeof(*(dst_arr)) +- \| sizeof(*(src_ptr)) +- \| sizeof(dst_arr[...]) +- \| sizeof(src_ptr[...]) +- \) ) ++ COPY_ARRAY(dst_arr, src_ptr, n) + +@@ +type T; +T[] dst_arr; +T[] src_arr; +expression n; +@@ +- memcpy(dst_arr, src_arr, (n) * \( sizeof(T) +- \| sizeof(*(dst_arr)) +- \| sizeof(*(src_arr)) +- \| sizeof(dst_arr[...]) +- \| sizeof(src_arr[...]) +- \) ) ++ COPY_ARRAY(dst_arr, src_arr, n) + +@@ +type T; +T *dst; +T *src; +expression n; +@@ +( +- memmove(dst, src, (n) * sizeof(*dst)); ++ MOVE_ARRAY(dst, src, n); +| +- memmove(dst, src, (n) * sizeof(*src)); ++ MOVE_ARRAY(dst, src, n); +| +- memmove(dst, src, (n) * sizeof(T)); ++ MOVE_ARRAY(dst, src, n); +) + +@@ +type T; +T *ptr; +expression n; +@@ +- ptr = xmalloc((n) * sizeof(*ptr)); ++ ALLOC_ARRAY(ptr, n); + +@@ +type T; +T *ptr; +expression n; +@@ +- ptr = xmalloc((n) * sizeof(T)); ++ ALLOC_ARRAY(ptr, n); + +@@ +type T; +T *ptr; +expression n != 1; +@@ +- ptr = xcalloc(n, \( sizeof(*ptr) \| sizeof(T) \) ) ++ CALLOC_ARRAY(ptr, n) + +@@ +expression dst, src, n; +@@ +-ALLOC_ARRAY(dst, n); +-COPY_ARRAY(dst, src, n); ++DUP_ARRAY(dst, src, n); + +@@ +type T; +T *ptr; +expression n; +@@ +- memset(ptr, \( 0 \| '\0' \), \( (n) \| n \) * \( sizeof(T) +- \| sizeof(ptr[...]) +- \| sizeof(*ptr) +- \) ) ++ MEMZERO_ARRAY(ptr, n) + +@@ +type T; +T *ptr; +expression n; +@@ +- memset(ptr, \( 0 \| '\0' \), \( sizeof(T) +- \| sizeof(ptr[...]) +- \| sizeof(*ptr) +- \) * \( (n) \| n \) ) ++ MEMZERO_ARRAY(ptr, n) + +@@ +type T; +T[] ptr; +expression n; +@@ +- memset(ptr, \( 0 \| '\0' \), \( (n) \| n \) * \( sizeof(T) +- \| sizeof(ptr[...]) +- \| sizeof(*ptr) +- \) ) ++ MEMZERO_ARRAY(ptr, n) + +@@ +type T; +T[] ptr; +expression n; +@@ +- memset(ptr, \( 0 \| '\0' \), \( sizeof(T) +- \| sizeof(ptr[...]) +- \| sizeof(*ptr) +- \) * \( (n) \| n \) ) ++ MEMZERO_ARRAY(ptr, n) diff --git a/tools/coccinelle/commit.cocci b/tools/coccinelle/commit.cocci new file mode 100644 index 0000000000..42725161e9 --- /dev/null +++ b/tools/coccinelle/commit.cocci @@ -0,0 +1,53 @@ +@@ +expression c; +@@ +- &c->maybe_tree->object.oid ++ get_commit_tree_oid(c) + +@@ +expression c; +@@ +- c->maybe_tree->object.oid.hash ++ get_commit_tree_oid(c)->hash + +@@ +identifier f !~ "^set_commit_tree$"; +expression c; +expression s; +@@ + f(...) {<... +- c->maybe_tree = s ++ set_commit_tree(c, s) + ...>} + +// These excluded functions must access c->maybe_tree directly. +// Note that if c->maybe_tree is written somewhere outside of these +// functions, then the recommended transformation will be bogus with +// repo_get_commit_tree() on the LHS. +@@ +identifier f != { repo_get_commit_tree, get_commit_tree_in_graph_one, + load_tree_for_commit, set_commit_tree, repo_parse_commit_no_graph }; +expression c; +@@ + f(...) {<... +- c->maybe_tree ++ repo_get_commit_tree(specify_the_right_repo_here, c) + ...>} + +@@ +struct commit *c; +expression E; +@@ +( +- c->generation = E; ++ commit_graph_data_at(c)->generation = E; +| +- c->graph_pos = E; ++ commit_graph_data_at(c)->graph_pos = E; +| +- c->generation ++ commit_graph_generation(c) +| +- c->graph_pos ++ commit_graph_position(c) +) diff --git a/tools/coccinelle/config_fn_ctx.pending.cocci b/tools/coccinelle/config_fn_ctx.pending.cocci new file mode 100644 index 0000000000..54f09fcbcd --- /dev/null +++ b/tools/coccinelle/config_fn_ctx.pending.cocci @@ -0,0 +1,144 @@ +@ get_fn @ +identifier fn, R; +@@ +( +( +git_config_from_file +| +git_config_from_file_with_options +| +git_config_from_mem +| +git_config_from_blob_oid +| +read_early_config +| +read_very_early_config +| +config_with_options +| +git_config +| +git_protected_config +| +config_from_gitmodules +) + (fn, ...) +| +repo_config(R, fn, ...) +) + +@ extends get_fn @ +identifier C1, C2, D; +@@ +int fn(const char *C1, const char *C2, ++ const struct config_context *ctx, + void *D); + +@ extends get_fn @ +@@ +int fn(const char *, const char *, ++ const struct config_context *, + void *); + +@ extends get_fn @ +// Don't change fns that look like callback fns but aren't +identifier fn2 != tar_filter_config && != git_diff_heuristic_config && + != git_default_submodule_config && != git_color_config && + != bundle_list_update && != parse_object_filter_config; +identifier C1, C2, D1, D2, S; +attribute name UNUSED; +@@ +int fn(const char *C1, const char *C2, ++ const struct config_context *ctx, + void *D1) { +<+... +( +fn2(C1, C2 ++ , ctx +, D2); +| +if(fn2(C1, C2 ++ , ctx +, D2) < 0) { ... } +| +return fn2(C1, C2 ++ , ctx +, D2); +| +S = fn2(C1, C2 ++ , ctx +, D2); +) +...+> + } + +@ extends get_fn@ +identifier C1, C2, D; +attribute name UNUSED; +@@ +int fn(const char *C1, const char *C2, ++ const struct config_context *ctx UNUSED, + void *D) {...} + + +// The previous rules don't catch all callbacks, especially if they're defined +// in a separate file from the repo_config() call. Fix these manually. +@@ +identifier C1, C2, D; +attribute name UNUSED; +@@ +int +( +git_ident_config +| +urlmatch_collect_fn +| +write_one_config +| +forbid_remote_url +| +credential_config_callback +) + (const char *C1, const char *C2, ++ const struct config_context *ctx UNUSED, + void *D) {...} + +@@ +identifier C1, C2, D, D2, S, fn2; +@@ +int +( +http_options +| +git_status_config +| +git_commit_config +| +git_default_core_config +| +grep_config +) + (const char *C1, const char *C2, ++ const struct config_context *ctx, + void *D) { +<+... +( +fn2(C1, C2 ++ , ctx +, D2); +| +if(fn2(C1, C2 ++ , ctx +, D2) < 0) { ... } +| +return fn2(C1, C2 ++ , ctx +, D2); +| +S = fn2(C1, C2 ++ , ctx +, D2); +) +...+> + } diff --git a/tools/coccinelle/equals-null.cocci b/tools/coccinelle/equals-null.cocci new file mode 100644 index 0000000000..92c7054013 --- /dev/null +++ b/tools/coccinelle/equals-null.cocci @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: LGPL-2.1-or-later */ +@@ +expression e; +statement s; +@@ +if ( +( +!e +| +- e == NULL ++ !e +) + ) + {...} +else s + +@@ +expression e; +statement s; +@@ +if ( +( +e +| +- e != NULL ++ e +) + ) + {...} +else s diff --git a/tools/coccinelle/flex_alloc.cocci b/tools/coccinelle/flex_alloc.cocci new file mode 100644 index 0000000000..e9f7f6d861 --- /dev/null +++ b/tools/coccinelle/flex_alloc.cocci @@ -0,0 +1,13 @@ +@@ +expression str; +identifier x, flexname; +@@ +- FLEX_ALLOC_MEM(x, flexname, str, strlen(str)); ++ FLEX_ALLOC_STR(x, flexname, str); + +@@ +expression str; +identifier x, ptrname; +@@ +- FLEXPTR_ALLOC_MEM(x, ptrname, str, strlen(str)); ++ FLEXPTR_ALLOC_STR(x, ptrname, str); diff --git a/tools/coccinelle/free.cocci b/tools/coccinelle/free.cocci new file mode 100644 index 0000000000..03799e1908 --- /dev/null +++ b/tools/coccinelle/free.cocci @@ -0,0 +1,45 @@ +@@ +expression E; +@@ +- if (E) +( + free(E); +| + commit_list_free(E); +) + +@@ +expression E; +@@ +- if (!E) +( + free(E); +| + commit_list_free(E); +) + +@@ +expression E; +@@ +- free(E); ++ FREE_AND_NULL(E); +- E = NULL; + +@@ +expression E; +@@ +- if (E) +- { + commit_list_free(E); + E = NULL; +- } + +@@ +expression E; +statement S; +@@ +- if (E) { ++ if (E) + S + commit_list_free(E); +- } diff --git a/tools/coccinelle/git_config_number.cocci b/tools/coccinelle/git_config_number.cocci new file mode 100644 index 0000000000..7b57dceefe --- /dev/null +++ b/tools/coccinelle/git_config_number.cocci @@ -0,0 +1,27 @@ +@@ +identifier C1, C2, C3; +@@ +( +( +git_config_int +| +git_config_int64 +| +git_config_ulong +| +git_config_ssize_t +) + (C1, C2 ++ , ctx->kvi + ) +| +( +git_configset_get_value +| +git_config_bool_or_int +) + (C1, C2 ++ , ctx->kvi + , C3 + ) +) diff --git a/tools/coccinelle/hashmap.cocci b/tools/coccinelle/hashmap.cocci new file mode 100644 index 0000000000..c5dbb4557b --- /dev/null +++ b/tools/coccinelle/hashmap.cocci @@ -0,0 +1,16 @@ +@@ +expression E; +struct hashmap_entry HME; +@@ +- HME.hash = E; ++ hashmap_entry_init(&HME, E); + +@@ +identifier f !~ "^hashmap_entry_init$"; +expression E; +struct hashmap_entry *HMEP; +@@ + f(...) {<... +- HMEP->hash = E; ++ hashmap_entry_init(HMEP, E); + ...>} diff --git a/tools/coccinelle/index-compatibility.cocci b/tools/coccinelle/index-compatibility.cocci new file mode 100644 index 0000000000..31e36cf3c4 --- /dev/null +++ b/tools/coccinelle/index-compatibility.cocci @@ -0,0 +1,157 @@ +// the_index.* variables +@@ +identifier AC = active_cache; +identifier AN = active_nr; +identifier ACC = active_cache_changed; +identifier ACT = active_cache_tree; +@@ +( +- AC ++ the_index.cache +| +- AN ++ the_index.cache_nr +| +- ACC ++ the_index.cache_changed +| +- ACT ++ the_index.cache_tree +) + +// "the_repository" simple cases +@@ +@@ +( +- read_cache ++ repo_read_index +| +- read_cache_unmerged ++ repo_read_index_unmerged +| +- hold_locked_index ++ repo_hold_locked_index +) + ( ++ the_repository, + ...) + +// "the_repository" special-cases +@@ +@@ +( +- read_cache_preload ++ repo_read_index_preload +) + ( ++ the_repository, + ... ++ , 0 + ) + +// "the_index" simple cases +@@ +@@ +( +- is_cache_unborn ++ is_index_unborn +| +- unmerged_cache ++ unmerged_index +| +- rename_cache_entry_at ++ rename_index_entry_at +| +- chmod_cache_entry ++ chmod_index_entry +| +- cache_file_exists ++ index_file_exists +| +- cache_name_is_other ++ index_name_is_other +| +- unmerge_cache_entry_at ++ unmerge_index_entry_at +| +- add_to_cache ++ add_to_index +| +- add_file_to_cache ++ add_file_to_index +| +- add_cache_entry ++ add_index_entry +| +- remove_file_from_cache ++ remove_file_from_index +| +- ce_match_stat ++ ie_match_stat +| +- ce_modified ++ ie_modified +| +- resolve_undo_clear ++ resolve_undo_clear_index +| +- cache_name_pos ++ index_name_pos +| +- update_main_cache_tree ++ cache_tree_update +| +- discard_cache ++ discard_index +) + ( ++ &the_index, + ...) + +@@ +@@ +( +- refresh_and_write_cache ++ repo_refresh_and_write_index +) + ( ++ the_repository, + ... ++ , NULL, NULL, NULL + ) + +// "the_index" special-cases +@@ +@@ +( +- read_cache_from ++ read_index_from +) + ( ++ &the_index, + ... ++ , get_git_dir() + ) + +@@ +@@ +( +- refresh_cache ++ refresh_index +) + ( ++ &the_index, + ... ++ , NULL, NULL, NULL + ) + +@@ +expression O; +@@ +- write_cache_as_tree ++ write_index_as_tree + ( +- O, ++ O, &the_index, get_index_file(), + ... + ) diff --git a/tools/coccinelle/meson.build b/tools/coccinelle/meson.build new file mode 100644 index 0000000000..ae7f5b5460 --- /dev/null +++ b/tools/coccinelle/meson.build @@ -0,0 +1,85 @@ +coccinelle_opt = get_option('coccinelle').require( + fs.exists(meson.project_source_root() / '.git'), + error_message: 'coccinelle can only be run from a git checkout', +) + +spatch = find_program('spatch', required: coccinelle_opt) +if not spatch.found() + subdir_done() +endif + +rules = [ + 'array.cocci', + 'commit.cocci', + 'config_fn_ctx.pending.cocci', + 'equals-null.cocci', + 'flex_alloc.cocci', + 'free.cocci', + 'git_config_number.cocci', + 'hashmap.cocci', + 'index-compatibility.cocci', + 'object_id.cocci', + 'preincr.cocci', + 'qsort.cocci', + 'refs.cocci', + 'strbuf.cocci', + 'swap.cocci', + 'the_repository.cocci', + 'xcalloc.cocci', + 'xopen.cocci', + 'xstrdup_or_null.cocci', + 'xstrncmpz.cocci', +] + +concatenated_rules = custom_target( + command: [ + 'cat', '@INPUT@', + ], + input: rules, + output: 'rules.cocci', + capture: true, +) + +coccinelle_sources = [] +foreach source : run_command(git, '-C', meson.project_source_root(), 'ls-files', '--deduplicate', '*.c', third_party_excludes, check: true).stdout().split() + coccinelle_sources += source +endforeach + +coccinelle_headers = [] +foreach header : headers_to_check + coccinelle_headers += meson.project_source_root() / header +endforeach + +coccinelle_includes = [] +foreach path : ['compat', 'ewah', 'refs', 'sha256', 'trace2', 'win32', 'xdiff'] + coccinelle_includes += ['-I', meson.project_source_root() / path] +endforeach + +patches = [ ] +foreach source : coccinelle_sources + patches += custom_target( + command: [ + spatch, + '--all-includes', + '--sp-file', concatenated_rules, + '--patch', meson.project_source_root(), + coccinelle_includes, + '@INPUT@', + ], + input: meson.project_source_root() / source, + output: source.underscorify() + '.patch', + capture: true, + depend_files: coccinelle_headers, + ) +endforeach + +concatenated_patch = custom_target( + command: [ + 'cat', '@INPUT@', + ], + input: patches, + output: 'cocci.patch', + capture: true, +) + +alias_target('coccicheck', concatenated_patch) diff --git a/tools/coccinelle/object_id.cocci b/tools/coccinelle/object_id.cocci new file mode 100644 index 0000000000..01f8d6935b --- /dev/null +++ b/tools/coccinelle/object_id.cocci @@ -0,0 +1,75 @@ +@@ +struct object_id OID; +@@ +- hashclr(OID.hash) ++ oidclr(&OID) + +@@ +identifier f != oidclr; +struct object_id *OIDPTR; +@@ + f(...) {<... +- hashclr(OIDPTR->hash) ++ oidclr(OIDPTR) + ...>} + +@@ +struct object_id OID1, OID2; +@@ +- hashcmp(OID1.hash, OID2.hash) ++ oidcmp(&OID1, &OID2) + +@@ +identifier f != oidcmp; +struct object_id *OIDPTR1, OIDPTR2; +@@ + f(...) {<... +- hashcmp(OIDPTR1->hash, OIDPTR2->hash) ++ oidcmp(OIDPTR1, OIDPTR2) + ...>} + +@@ +struct object_id *OIDPTR; +struct object_id OID; +@@ +- hashcmp(OIDPTR->hash, OID.hash) ++ oidcmp(OIDPTR, &OID) + +@@ +struct object_id *OIDPTR; +struct object_id OID; +@@ +- hashcmp(OID.hash, OIDPTR->hash) ++ oidcmp(&OID, OIDPTR) + +@@ +struct object_id *OIDPTR1; +struct object_id *OIDPTR2; +@@ +- oidcmp(OIDPTR1, OIDPTR2) == 0 ++ oideq(OIDPTR1, OIDPTR2) + +@@ +identifier f != hasheq; +expression E1, E2; +@@ + f(...) {<... +- hashcmp(E1, E2) == 0 ++ hasheq(E1, E2) + ...>} + +@@ +struct object_id *OIDPTR1; +struct object_id *OIDPTR2; +@@ +- oidcmp(OIDPTR1, OIDPTR2) != 0 ++ !oideq(OIDPTR1, OIDPTR2) + +@@ +identifier f != hasheq; +expression E1, E2; +@@ + f(...) {<... +- hashcmp(E1, E2) != 0 ++ !hasheq(E1, E2) + ...>} diff --git a/tools/coccinelle/preincr.cocci b/tools/coccinelle/preincr.cocci new file mode 100644 index 0000000000..ae42cb0730 --- /dev/null +++ b/tools/coccinelle/preincr.cocci @@ -0,0 +1,5 @@ +@@ +identifier i; +@@ +- ++i > 1 ++ i++ diff --git a/tools/coccinelle/qsort.cocci b/tools/coccinelle/qsort.cocci new file mode 100644 index 0000000000..22b93a9966 --- /dev/null +++ b/tools/coccinelle/qsort.cocci @@ -0,0 +1,37 @@ +@@ +expression base, nmemb, compar; +@@ +- qsort(base, nmemb, sizeof(*base), compar); ++ QSORT(base, nmemb, compar); + +@@ +expression base, nmemb, compar; +@@ +- qsort(base, nmemb, sizeof(base[0]), compar); ++ QSORT(base, nmemb, compar); + +@@ +type T; +T *base; +expression nmemb, compar; +@@ +- qsort(base, nmemb, sizeof(T), compar); ++ QSORT(base, nmemb, compar); + +@@ +expression base, nmemb, compar; +@@ +- if (nmemb) + QSORT(base, nmemb, compar); + +@@ +expression base, nmemb, compar; +@@ +- if (nmemb > 0) + QSORT(base, nmemb, compar); + +@@ +expression base, nmemb, compar; +@@ +- if (nmemb > 1) + QSORT(base, nmemb, compar); diff --git a/tools/coccinelle/refs.cocci b/tools/coccinelle/refs.cocci new file mode 100644 index 0000000000..31d9cad8f3 --- /dev/null +++ b/tools/coccinelle/refs.cocci @@ -0,0 +1,103 @@ +// Migrate "refs.h" to not rely on `the_repository` implicitly anymore. +@@ +@@ +( +- resolve_ref_unsafe ++ refs_resolve_ref_unsafe +| +- resolve_refdup ++ refs_resolve_refdup +| +- read_ref_full ++ refs_read_ref_full +| +- read_ref ++ refs_read_ref +| +- ref_exists ++ refs_ref_exists +| +- head_ref ++ refs_head_ref +| +- for_each_ref ++ refs_for_each_ref +| +- for_each_ref_in ++ refs_for_each_ref_in +| +- for_each_fullref_in ++ refs_for_each_fullref_in +| +- for_each_tag_ref ++ refs_for_each_tag_ref +| +- for_each_branch_ref ++ refs_for_each_branch_ref +| +- for_each_remote_ref ++ refs_for_each_remote_ref +| +- for_each_glob_ref ++ refs_for_each_glob_ref +| +- for_each_glob_ref_in ++ refs_for_each_glob_ref_in +| +- head_ref_namespaced ++ refs_head_ref_namespaced +| +- for_each_namespaced_ref ++ refs_for_each_namespaced_ref +| +- for_each_rawref ++ refs_for_each_rawref +| +- safe_create_reflog ++ refs_create_reflog +| +- reflog_exists ++ refs_reflog_exists +| +- delete_ref ++ refs_delete_ref +| +- delete_refs ++ refs_delete_refs +| +- delete_reflog ++ refs_delete_reflog +| +- for_each_reflog_ent ++ refs_for_each_reflog_ent +| +- for_each_reflog_ent_reverse ++ refs_for_each_reflog_ent_reverse +| +- for_each_reflog ++ refs_for_each_reflog +| +- shorten_unambiguous_ref ++ refs_shorten_unambiguous_ref +| +- rename_ref ++ refs_rename_ref +| +- copy_existing_ref ++ refs_copy_existing_ref +| +- create_symref ++ refs_create_symref +| +- ref_transaction_begin ++ ref_store_transaction_begin +| +- update_ref ++ refs_update_ref +| +- reflog_expire ++ refs_reflog_expire +) + ( ++ get_main_ref_store(the_repository), + ...) diff --git a/tools/coccinelle/spatchcache b/tools/coccinelle/spatchcache new file mode 100755 index 0000000000..efbcbc3827 --- /dev/null +++ b/tools/coccinelle/spatchcache @@ -0,0 +1,304 @@ +#!/bin/sh +# +# spatchcache: a poor-man's "ccache"-alike for "spatch" in git.git +# +# This caching command relies on the peculiarities of the Makefile +# driving "spatch" in git.git, in particular if we invoke: +# +# make +# # See "spatchCache.cacheWhenStderr" for why "--very-quiet" is +# # used +# make coccicheck SPATCH_FLAGS=--very-quiet +# +# We can with COMPUTE_HEADER_DEPENDENCIES (auto-detected as true with +# "gcc" and "clang") write e.g. a .depend/grep.o.d for grep.c, when we +# compile grep.o. +# +# The .depend/grep.o.d will have the full header dependency tree of +# grep.c, and we can thus cache the output of "spatch" by: +# +# 1. Hashing all of those files +# 2. Hashing our source file, and the *.cocci rule we're +# applying +# 3. Running spatch, if suggests no changes (by far the common +# case) we invoke "spatchCache.getCmd" and +# "spatchCache.setCmd" with a hash SHA-256 to ask "does this +# ID have no changes" or "say that ID had no changes> +# 4. If no "spatchCache.{set,get}Cmd" is specified we'll use +# "redis-cli" and maintain a SET called "spatch-cache". Set +# appropriate redis memory policies to keep it from growing +# out of control. +# +# This along with the general incremental "make" support for +# "tools/coccinelle" makes it viable to (re-)run coccicheck +# e.g. when merging integration branches. +# +# Note that the "--very-quiet" flag is currently critical. The cache +# will refuse to cache anything that has output on STDERR (which might +# be errors from spatch), but see spatchCache.cacheWhenStderr below. +# +# The STDERR (and exit code) could in principle be cached (as with +# ccache), but then the simple structure in the Redis cache would need +# to change, so just supply "--very-quiet" for now. +# +# To use this, simply set SPATCH to +# tools/coccinelle/spatchcache. Then optionally set: +# +# [spatchCache] +# # Optional: path to a custom spatch +# spatch = ~/g/coccicheck/spatch.opt +# +# As well as this trace config (debug implies trace): +# +# cacheWhenStderr = true +# trace = false +# debug = false +# +# The ".depend/grep.o.d" can also be customized, as a string that will +# be eval'd, it has access to a "$dirname" and "$basename": +# +# [spatchCache] +# dependFormat = "$dirname/.depend/${basename%.c}.o.d" +# +# Setting "trace" to "true" allows for seeing when we have a cache HIT +# or MISS. To debug whether the cache is working do that, and run e.g.: +# +# redis-cli FLUSHALL +# +# grep -hore HIT -e MISS -e SET -e NOCACHE -e CANTCACHE .build/tools/coccinelle | sort | uniq -c +# 600 CANTCACHE +# 7365 MISS +# 7365 SET +# +# A subsequent "make cocciclean && make coccicheck" should then have +# all "HIT"'s and "CANTCACHE"'s. +# +# The "spatchCache.cacheWhenStderr" option is critical when using +# spatchCache.{trace,debug} to debug whether something is set in the +# cache, as we'll write to the spatch logs in .build/* we'd otherwise +# always emit a NOCACHE. +# +# Reading the config can make the command much slower, to work around +# this the config can be set in the environment, with environment +# variable name corresponding to the config key. "default" can be used +# to use whatever's the script default, e.g. setting +# spatchCache.cacheWhenStderr=true and deferring to the defaults for +# the rest is: +# +# export GIT_CONTRIB_SPATCHCACHE_DEBUG=default +# export GIT_CONTRIB_SPATCHCACHE_TRACE=default +# export GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR=true +# export GIT_CONTRIB_SPATCHCACHE_SPATCH=default +# export GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT=default +# export GIT_CONTRIB_SPATCHCACHE_SETCMD=default +# export GIT_CONTRIB_SPATCHCACHE_GETCMD=default + +set -e + +env_or_config () { + env="$1" + shift + if test "$env" = "default" + then + # Avoid expensive "git config" invocation + return + elif test -n "$env" + then + echo "$env" + else + git config $@ || : + fi +} + +## Our own configuration & options +debug=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_DEBUG" --bool "spatchCache.debug") +if test "$debug" != "true" +then + debug= +fi +if test -n "$debug" +then + set -x +fi + +trace=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_TRACE" --bool "spatchCache.trace") +if test "$trace" != "true" +then + trace= +fi +if test -n "$debug" +then + # debug implies trace + trace=true +fi + +cacheWhenStderr=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_CACHEWHENSTDERR" --bool "spatchCache.cacheWhenStderr") +if test "$cacheWhenStderr" != "true" +then + cacheWhenStderr= +fi + +trace_it () { + if test -z "$trace" + then + return + fi + echo "$@" >&2 +} + +spatch=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_SPATCH" --path "spatchCache.spatch") +if test -n "$spatch" +then + if test -n "$debug" + then + trace_it "custom spatchCache.spatch='$spatch'" + fi +else + spatch=spatch +fi + +dependFormat='$dirname/.depend/${basename%.c}.o.d' +dependFormatCfg=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_DEPENDFORMAT" "spatchCache.dependFormat") +if test -n "$dependFormatCfg" +then + dependFormat="$dependFormatCfg" +fi + +set=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_SETCMD" "spatchCache.setCmd") +get=$(env_or_config "$GIT_CONTRIB_SPATCHCACHE_GETCMD" "spatchCache.getCmd") + +## Parse spatch()-like command-line for caching info +arg_sp= +arg_file= +args="$@" +spatch_opts() { + while test $# != 0 + do + arg_file="$1" + case "$1" in + --sp-file) + arg_sp="$2" + ;; + esac + shift + done +} +spatch_opts "$@" +if ! test -f "$arg_file" +then + arg_file= +fi + +hash_for_cache() { + # Parameters that should affect the cache + echo "args=$args" + echo "config spatchCache.spatch=$spatch" + echo "config spatchCache.debug=$debug" + echo "config spatchCache.trace=$trace" + echo "config spatchCache.cacheWhenStderr=$cacheWhenStderr" + echo + + # Our target file and its dependencies + git hash-object "$1" "$2" $(grep -E -o '^[^:]+:$' "$3" | tr -d ':') +} + +# Sanity checks +if ! test -f "$arg_sp" && ! test -f "$arg_file" +then + echo $0: no idea how to cache "$@" >&2 + exit 128 +fi + +# Main logic +dirname=$(dirname "$arg_file") +basename=$(basename "$arg_file") +eval "dep=$dependFormat" + +if ! test -f "$dep" +then + trace_it "$0: CANTCACHE have no '$dep' for '$arg_file'!" + exec "$spatch" "$@" +fi + +if test -n "$debug" +then + trace_it "$0: The full cache input for '$arg_sp' '$arg_file' '$dep'" + hash_for_cache "$arg_sp" "$arg_file" "$dep" >&2 +fi +sum=$(hash_for_cache "$arg_sp" "$arg_file" "$dep" | git hash-object --stdin) + +trace_it "$0: processing '$arg_file' with '$arg_sp' rule, and got hash '$sum' for it + '$dep'" + +getret= +if test -z "$get" +then + if test $(redis-cli SISMEMBER spatch-cache "$sum") = 1 + then + getret=0 + else + getret=1 + fi +else + $set "$sum" + getret=$? +fi + +if test "$getret" = 0 +then + trace_it "$0: HIT for '$arg_file' with '$arg_sp'" + exit 0 +else + trace_it "$0: MISS: for '$arg_file' with '$arg_sp'" +fi + +out="$(mktemp)" +err="$(mktemp)" + +set +e +"$spatch" "$@" >"$out" 2>>"$err" +ret=$? +cat "$out" +cat "$err" >&2 +set -e + +nocache= +if test $ret != 0 +then + nocache="exited non-zero: $ret" +elif test -s "$out" +then + nocache="had patch output" +elif test -z "$cacheWhenStderr" && test -s "$err" +then + nocache="had stderr (use --very-quiet or spatchCache.cacheWhenStderr=true?)" +fi + +if test -n "$nocache" +then + trace_it "$0: NOCACHE ($nocache): for '$arg_file' with '$arg_sp'" + exit "$ret" +fi + +trace_it "$0: SET: for '$arg_file' with '$arg_sp'" + +setret= +if test -z "$set" +then + if test $(redis-cli SADD spatch-cache "$sum") = 1 + then + setret=0 + else + setret=1 + fi +else + "$set" "$sum" + setret=$? +fi + +if test "$setret" != 0 +then + echo "FAILED to set '$sum' in cache!" >&2 + exit 128 +fi + +exit "$ret" diff --git a/tools/coccinelle/strbuf.cocci b/tools/coccinelle/strbuf.cocci new file mode 100644 index 0000000000..5f06105df6 --- /dev/null +++ b/tools/coccinelle/strbuf.cocci @@ -0,0 +1,62 @@ +@@ +expression E; +constant fmt !~ "%"; +@@ +- strbuf_addf ++ strbuf_addstr + (E, +( + fmt +| + _(fmt) +) + ); + +@@ +expression E; +struct strbuf SB; +format F =~ "^s$"; +@@ +- strbuf_addf(E, "%@F@", SB.buf); ++ strbuf_addbuf(E, &SB); + +@@ +expression E; +struct strbuf *SBP; +format F =~ "^s$"; +@@ +- strbuf_addf(E, "%@F@", SBP->buf); ++ strbuf_addbuf(E, SBP); + +@@ +expression E; +struct strbuf SB; +@@ +- strbuf_addstr(E, SB.buf); ++ strbuf_addbuf(E, &SB); + +@@ +expression E; +struct strbuf *SBP; +@@ +- strbuf_addstr(E, SBP->buf); ++ strbuf_addbuf(E, SBP); + +@@ +expression E1, E2; +format F =~ "^s$"; +@@ +- strbuf_addf(E1, "%@F@", E2); ++ strbuf_addstr(E1, E2); + +@@ +expression E1, E2, E3; +@@ +- strbuf_addstr(E1, find_unique_abbrev(E2, E3)); ++ strbuf_add_unique_abbrev(E1, E2, E3); + +@@ +expression E1, E2; +@@ +- strbuf_addstr(E1, real_path(E2)); ++ strbuf_add_real_path(E1, E2); diff --git a/tools/coccinelle/swap.cocci b/tools/coccinelle/swap.cocci new file mode 100644 index 0000000000..522177afb6 --- /dev/null +++ b/tools/coccinelle/swap.cocci @@ -0,0 +1,28 @@ +@@ +type T; +identifier tmp; +T a, b; +@@ +- T tmp = a; ++ T tmp; ++ tmp = a; + a = b; + b = tmp; + +@ swap @ +type T; +T tmp, a, b; +@@ +- tmp = a; +- a = b; +- b = tmp; ++ SWAP(a, b); + +@ extends swap @ +identifier unused; +@@ + { + ... +- T unused; + ... when != unused + } diff --git a/tools/coccinelle/tests/free.c b/tools/coccinelle/tests/free.c new file mode 100644 index 0000000000..96d4abc0c7 --- /dev/null +++ b/tools/coccinelle/tests/free.c @@ -0,0 +1,11 @@ +int use_FREE_AND_NULL(int *v) +{ + free(*v); + *v = NULL; +} + +int need_no_if(int *v) +{ + if (v) + free(v); +} diff --git a/tools/coccinelle/tests/free.res b/tools/coccinelle/tests/free.res new file mode 100644 index 0000000000..f90fd9f48e --- /dev/null +++ b/tools/coccinelle/tests/free.res @@ -0,0 +1,9 @@ +int use_FREE_AND_NULL(int *v) +{ + FREE_AND_NULL(*v); +} + +int need_no_if(int *v) +{ + free(v); +} diff --git a/tools/coccinelle/the_repository.cocci b/tools/coccinelle/the_repository.cocci new file mode 100644 index 0000000000..f1129f7985 --- /dev/null +++ b/tools/coccinelle/the_repository.cocci @@ -0,0 +1,18 @@ +// Fully migrated "the_repository" additions +@@ +@@ +( +// TODO: remove the rules below and the macros from tree.h after the +// next Git release. +- parse_tree ++ repo_parse_tree +| +- parse_tree_gently ++ repo_parse_tree_gently +| +- parse_tree_indirect ++ repo_parse_tree_indirect +) + ( ++ the_repository, + ...) diff --git a/tools/coccinelle/xcalloc.cocci b/tools/coccinelle/xcalloc.cocci new file mode 100644 index 0000000000..c291011607 --- /dev/null +++ b/tools/coccinelle/xcalloc.cocci @@ -0,0 +1,10 @@ +@@ +type T; +T *ptr; +expression n; +@@ + xcalloc( ++ n, + \( sizeof(T) \| sizeof(*ptr) \) +- , n + ) diff --git a/tools/coccinelle/xopen.cocci b/tools/coccinelle/xopen.cocci new file mode 100644 index 0000000000..b71db67019 --- /dev/null +++ b/tools/coccinelle/xopen.cocci @@ -0,0 +1,19 @@ +@@ +identifier fd; +identifier die_fn =~ "^(die|die_errno)$"; +@@ + int fd = +- open ++ xopen + (...); +- if ( \( fd < 0 \| fd == -1 \) ) { die_fn(...); } + +@@ +expression fd; +identifier die_fn =~ "^(die|die_errno)$"; +@@ + fd = +- open ++ xopen + (...); +- if ( \( fd < 0 \| fd == -1 \) ) { die_fn(...); } diff --git a/tools/coccinelle/xstrdup_or_null.cocci b/tools/coccinelle/xstrdup_or_null.cocci new file mode 100644 index 0000000000..9c1d2939b6 --- /dev/null +++ b/tools/coccinelle/xstrdup_or_null.cocci @@ -0,0 +1,5 @@ +@@ +expression E; +@@ +- xstrdup(absolute_path(E)) ++ absolute_pathdup(E) diff --git a/tools/coccinelle/xstrncmpz.cocci b/tools/coccinelle/xstrncmpz.cocci new file mode 100644 index 0000000000..ccb39e2bc0 --- /dev/null +++ b/tools/coccinelle/xstrncmpz.cocci @@ -0,0 +1,28 @@ +@@ +expression S, T, L; +@@ +( +- strncmp(S, T, L) || S[L] ++ !!xstrncmpz(S, T, L) +| +- strncmp(S, T, L) || S[L] != '\0' ++ !!xstrncmpz(S, T, L) +| +- strncmp(S, T, L) || T[L] ++ !!xstrncmpz(T, S, L) +| +- strncmp(S, T, L) || T[L] != '\0' ++ !!xstrncmpz(T, S, L) +| +- !strncmp(S, T, L) && !S[L] ++ !xstrncmpz(S, T, L) +| +- !strncmp(S, T, L) && S[L] == '\0' ++ !xstrncmpz(S, T, L) +| +- !strncmp(S, T, L) && !T[L] ++ !xstrncmpz(T, S, L) +| +- !strncmp(S, T, L) && T[L] == '\0' ++ !xstrncmpz(T, S, L) +) diff --git a/tools/meson.build b/tools/meson.build index e69de29bb2..f731f74312 100644 --- a/tools/meson.build +++ b/tools/meson.build @@ -0,0 +1 @@ +subdir('coccinelle') -- cgit v1.3 From fe309664ea804d17812bab22927756dc35e5e955 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 19 Mar 2026 06:33:22 +0100 Subject: contrib: move "coverage-diff.sh" script into "tools/" The "coverage-diff.sh" script can be used to get information about test coverage fro the Git codebase. It is thus rather specific to our build and test infrastructure and part of the developer-facing tooling. The fact that this script is part of "contrib/" is thus rather misleading and a historic wart. Promote the tool into the new "tools/" directory. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- contrib/coverage-diff.sh | 103 ----------------------------------------------- tools/coverage-diff.sh | 103 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+), 103 deletions(-) delete mode 100755 contrib/coverage-diff.sh create mode 100755 tools/coverage-diff.sh diff --git a/contrib/coverage-diff.sh b/contrib/coverage-diff.sh deleted file mode 100755 index 6ce9603568..0000000000 --- a/contrib/coverage-diff.sh +++ /dev/null @@ -1,103 +0,0 @@ -#!/bin/sh - -# Usage: Run 'contrib/coverage-diff.sh ' from source-root -# after running -# -# make coverage-test -# make coverage-report -# -# while checked out at . This script combines the *.gcov files -# generated by the 'make' commands above with 'git diff ' -# to report new lines that are not covered by the test suite. - -V1=$1 -V2=$2 - -diff_lines () { - perl -e ' - my $line_num; - while (<>) { - # Hunk header? Grab the beginning in postimage. - if (/^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@/) { - $line_num = $1; - next; - } - - # Have we seen a hunk? Ignore "diff --git" etc. - next unless defined $line_num; - - # Deleted line? Ignore. - if (/^-/) { - next; - } - - # Show only the line number of added lines. - if (/^\+/) { - print "$line_num\n"; - } - # Either common context or added line appear in - # the postimage. Count it. - $line_num++; - } - ' -} - -files=$(git diff --name-only "$V1" "$V2" -- \*.c) - -# create empty file ->coverage-data.txt - -for file in $files -do - git diff "$V1" "$V2" -- "$file" | - diff_lines | - sort >new_lines.txt - - if ! test -s new_lines.txt - then - continue - fi - - hash_file=$(echo $file | sed "s/\//\#/") - - if ! test -s "$hash_file.gcov" - then - continue - fi - - sed -ne '/#####:/{ - s/ #####:// - s/:.*// - s/ //g - p - }' "$hash_file.gcov" | - sort >uncovered_lines.txt - - comm -12 uncovered_lines.txt new_lines.txt | - sed -e 's/$/\)/' -e 's/^/ /' >uncovered_new_lines.txt - - grep -q '[^[:space:]]' >coverage-data.txt && - git blame -s "$V2" -- "$file" | - sed 's/\t//g' | - grep -f uncovered_new_lines.txt >>coverage-data.txt && - echo >>coverage-data.txt - - rm -f new_lines.txt uncovered_lines.txt uncovered_new_lines.txt -done - -cat coverage-data.txt - -echo "Commits introducing uncovered code:" - -commit_list=$(awk '/^[0-9a-f]{7,}/ { print $1 }' coverage-data.txt | sort -u) - -( - for commit in $commit_list - do - git log --no-decorate --pretty=format:'%an %h: %s' -1 $commit - echo - done -) | sort - -rm coverage-data.txt diff --git a/tools/coverage-diff.sh b/tools/coverage-diff.sh new file mode 100755 index 0000000000..6ce9603568 --- /dev/null +++ b/tools/coverage-diff.sh @@ -0,0 +1,103 @@ +#!/bin/sh + +# Usage: Run 'contrib/coverage-diff.sh ' from source-root +# after running +# +# make coverage-test +# make coverage-report +# +# while checked out at . This script combines the *.gcov files +# generated by the 'make' commands above with 'git diff ' +# to report new lines that are not covered by the test suite. + +V1=$1 +V2=$2 + +diff_lines () { + perl -e ' + my $line_num; + while (<>) { + # Hunk header? Grab the beginning in postimage. + if (/^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@/) { + $line_num = $1; + next; + } + + # Have we seen a hunk? Ignore "diff --git" etc. + next unless defined $line_num; + + # Deleted line? Ignore. + if (/^-/) { + next; + } + + # Show only the line number of added lines. + if (/^\+/) { + print "$line_num\n"; + } + # Either common context or added line appear in + # the postimage. Count it. + $line_num++; + } + ' +} + +files=$(git diff --name-only "$V1" "$V2" -- \*.c) + +# create empty file +>coverage-data.txt + +for file in $files +do + git diff "$V1" "$V2" -- "$file" | + diff_lines | + sort >new_lines.txt + + if ! test -s new_lines.txt + then + continue + fi + + hash_file=$(echo $file | sed "s/\//\#/") + + if ! test -s "$hash_file.gcov" + then + continue + fi + + sed -ne '/#####:/{ + s/ #####:// + s/:.*// + s/ //g + p + }' "$hash_file.gcov" | + sort >uncovered_lines.txt + + comm -12 uncovered_lines.txt new_lines.txt | + sed -e 's/$/\)/' -e 's/^/ /' >uncovered_new_lines.txt + + grep -q '[^[:space:]]' >coverage-data.txt && + git blame -s "$V2" -- "$file" | + sed 's/\t//g' | + grep -f uncovered_new_lines.txt >>coverage-data.txt && + echo >>coverage-data.txt + + rm -f new_lines.txt uncovered_lines.txt uncovered_new_lines.txt +done + +cat coverage-data.txt + +echo "Commits introducing uncovered code:" + +commit_list=$(awk '/^[0-9a-f]{7,}/ { print $1 }' coverage-data.txt | sort -u) + +( + for commit in $commit_list + do + git log --no-decorate --pretty=format:'%an %h: %s' -1 $commit + echo + done +) | sort + +rm coverage-data.txt -- cgit v1.3 From 405c98a6a0e017f41f5de9c649a8f6f1b3fc4314 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 19 Mar 2026 06:33:23 +0100 Subject: contrib: move "update-unicode.sh" script into "tools/" The "update-unicode.sh" script is used to update the unicode data compiled into Git whenever a new version of the Unicode standard has been released. As such, it is a natural part of our developer-facing tooling, and its presence in "contrib/" is misleading. Promote the script into the new "tools/" directory. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- contrib/update-unicode/.gitignore | 3 --- contrib/update-unicode/README | 20 ------------------- contrib/update-unicode/update_unicode.sh | 33 -------------------------------- tools/update-unicode/.gitignore | 3 +++ tools/update-unicode/README | 20 +++++++++++++++++++ tools/update-unicode/update_unicode.sh | 33 ++++++++++++++++++++++++++++++++ 6 files changed, 56 insertions(+), 56 deletions(-) delete mode 100644 contrib/update-unicode/.gitignore delete mode 100644 contrib/update-unicode/README delete mode 100755 contrib/update-unicode/update_unicode.sh create mode 100644 tools/update-unicode/.gitignore create mode 100644 tools/update-unicode/README create mode 100755 tools/update-unicode/update_unicode.sh diff --git a/contrib/update-unicode/.gitignore b/contrib/update-unicode/.gitignore deleted file mode 100644 index b0ebc6aad2..0000000000 --- a/contrib/update-unicode/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -uniset/ -UnicodeData.txt -EastAsianWidth.txt diff --git a/contrib/update-unicode/README b/contrib/update-unicode/README deleted file mode 100644 index 151a197041..0000000000 --- a/contrib/update-unicode/README +++ /dev/null @@ -1,20 +0,0 @@ -TL;DR: Run update_unicode.sh after the publication of a new Unicode -standard and commit the resulting unicode-widths.h file. - -The long version -================ - -The Git source code ships the file unicode-widths.h which contains -tables of zero and double width Unicode code points, respectively. -These tables are generated using update_unicode.sh in this directory. -update_unicode.sh itself uses a third-party tool, uniset, to query two -Unicode data files for the interesting code points. - -On first run, update_unicode.sh clones uniset from Github and builds it. -This requires a current-ish version of autoconf (2.69 works per December -2016). - -On each run, update_unicode.sh checks whether more recent Unicode data -files are available from the Unicode consortium, and rebuilds the header -unicode-widths.h with the new data. The new header can then be -committed. diff --git a/contrib/update-unicode/update_unicode.sh b/contrib/update-unicode/update_unicode.sh deleted file mode 100755 index aa90865bef..0000000000 --- a/contrib/update-unicode/update_unicode.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/sh -#See http://www.unicode.org/reports/tr44/ -# -#Me Enclosing_Mark an enclosing combining mark -#Mn Nonspacing_Mark a nonspacing combining mark (zero advance width) -#Cf Format a format control character -# -cd "$(dirname "$0")" -UNICODEWIDTH_H=$(git rev-parse --show-toplevel)/unicode-width.h - -wget -N http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt \ - http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt && -if ! test -d uniset; then - git clone https://github.com/depp/uniset.git && - ( cd uniset && git checkout 4b186196dd ) -fi && -( - cd uniset && - if ! test -x uniset; then - autoreconf -i && - ./configure --enable-warnings=-Werror CFLAGS='-O0 -ggdb' - fi && - make -) && -UNICODE_DIR=. && export UNICODE_DIR && -cat >$UNICODEWIDTH_H <<-EOF -static const struct interval zero_width[] = { - $(uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD) -}; -static const struct interval double_width[] = { - $(uniset/uniset --32 eaw:F,W) -}; -EOF diff --git a/tools/update-unicode/.gitignore b/tools/update-unicode/.gitignore new file mode 100644 index 0000000000..b0ebc6aad2 --- /dev/null +++ b/tools/update-unicode/.gitignore @@ -0,0 +1,3 @@ +uniset/ +UnicodeData.txt +EastAsianWidth.txt diff --git a/tools/update-unicode/README b/tools/update-unicode/README new file mode 100644 index 0000000000..151a197041 --- /dev/null +++ b/tools/update-unicode/README @@ -0,0 +1,20 @@ +TL;DR: Run update_unicode.sh after the publication of a new Unicode +standard and commit the resulting unicode-widths.h file. + +The long version +================ + +The Git source code ships the file unicode-widths.h which contains +tables of zero and double width Unicode code points, respectively. +These tables are generated using update_unicode.sh in this directory. +update_unicode.sh itself uses a third-party tool, uniset, to query two +Unicode data files for the interesting code points. + +On first run, update_unicode.sh clones uniset from Github and builds it. +This requires a current-ish version of autoconf (2.69 works per December +2016). + +On each run, update_unicode.sh checks whether more recent Unicode data +files are available from the Unicode consortium, and rebuilds the header +unicode-widths.h with the new data. The new header can then be +committed. diff --git a/tools/update-unicode/update_unicode.sh b/tools/update-unicode/update_unicode.sh new file mode 100755 index 0000000000..aa90865bef --- /dev/null +++ b/tools/update-unicode/update_unicode.sh @@ -0,0 +1,33 @@ +#!/bin/sh +#See http://www.unicode.org/reports/tr44/ +# +#Me Enclosing_Mark an enclosing combining mark +#Mn Nonspacing_Mark a nonspacing combining mark (zero advance width) +#Cf Format a format control character +# +cd "$(dirname "$0")" +UNICODEWIDTH_H=$(git rev-parse --show-toplevel)/unicode-width.h + +wget -N http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt \ + http://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt && +if ! test -d uniset; then + git clone https://github.com/depp/uniset.git && + ( cd uniset && git checkout 4b186196dd ) +fi && +( + cd uniset && + if ! test -x uniset; then + autoreconf -i && + ./configure --enable-warnings=-Werror CFLAGS='-O0 -ggdb' + fi && + make +) && +UNICODE_DIR=. && export UNICODE_DIR && +cat >$UNICODEWIDTH_H <<-EOF +static const struct interval zero_width[] = { + $(uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD) +}; +static const struct interval double_width[] = { + $(uniset/uniset --32 eaw:F,W) +}; +EOF -- cgit v1.3 From a767f2fd6c5a6104ff32a35a27f0c15aec546957 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 19 Mar 2026 06:33:24 +0100 Subject: builds: move build scripts into "tools/" We have a bunch of scripts used by our different build systems that are all located in the top-level directory. Now that we have introduced the new "tools/" directory though we have a better home for them. Move the scripts into the "tools/" directory. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- Makefile | 34 +++++----- check-builtins.sh | 34 ---------- config.mak.dev | 2 +- contrib/buildsystems/CMakeLists.txt | 18 +++--- contrib/subtree/meson.build | 2 +- detect-compiler | 58 ----------------- generate-cmdlist.sh | 120 ------------------------------------ generate-configlist.sh | 52 ---------------- generate-hooklist.sh | 33 ---------- generate-perl.sh | 37 ----------- generate-python.sh | 20 ------ generate-script.sh | 34 ---------- meson.build | 14 ++--- tools/check-builtins.sh | 34 ++++++++++ tools/detect-compiler | 58 +++++++++++++++++ tools/generate-cmdlist.sh | 120 ++++++++++++++++++++++++++++++++++++ tools/generate-configlist.sh | 52 ++++++++++++++++ tools/generate-hooklist.sh | 33 ++++++++++ tools/generate-perl.sh | 37 +++++++++++ tools/generate-python.sh | 20 ++++++ tools/generate-script.sh | 34 ++++++++++ 21 files changed, 423 insertions(+), 423 deletions(-) delete mode 100755 check-builtins.sh delete mode 100755 detect-compiler delete mode 100755 generate-cmdlist.sh delete mode 100755 generate-configlist.sh delete mode 100755 generate-hooklist.sh delete mode 100755 generate-perl.sh delete mode 100755 generate-python.sh delete mode 100755 generate-script.sh create mode 100755 tools/check-builtins.sh create mode 100755 tools/detect-compiler create mode 100755 tools/generate-cmdlist.sh create mode 100755 tools/generate-configlist.sh create mode 100755 tools/generate-hooklist.sh create mode 100755 tools/generate-perl.sh create mode 100755 tools/generate-python.sh create mode 100755 tools/generate-script.sh diff --git a/Makefile b/Makefile index 8564b1be36..322f5940e3 100644 --- a/Makefile +++ b/Makefile @@ -2689,21 +2689,21 @@ $(BUILT_INS): git$X ln -s $< $@ 2>/dev/null || \ cp $< $@ -config-list.h: generate-configlist.sh +config-list.h: tools/generate-configlist.sh @mkdir -p .depend - $(QUIET_GEN)$(SHELL_PATH) ./generate-configlist.sh . $@ .depend/config-list.h.d + $(QUIET_GEN)$(SHELL_PATH) ./tools/generate-configlist.sh . $@ .depend/config-list.h.d -include .depend/config-list.h.d -command-list.h: generate-cmdlist.sh command-list.txt +command-list.h: tools/generate-cmdlist.sh command-list.txt command-list.h: $(wildcard Documentation/git*.adoc) - $(QUIET_GEN)$(SHELL_PATH) ./generate-cmdlist.sh \ + $(QUIET_GEN)$(SHELL_PATH) ./tools/generate-cmdlist.sh \ $(patsubst %,--exclude-program %,$(EXCLUDED_PROGRAMS)) \ . $@ -hook-list.h: generate-hooklist.sh Documentation/githooks.adoc - $(QUIET_GEN)$(SHELL_PATH) ./generate-hooklist.sh . $@ +hook-list.h: tools/generate-hooklist.sh Documentation/githooks.adoc + $(QUIET_GEN)$(SHELL_PATH) ./tools/generate-hooklist.sh . $@ SCRIPT_DEFINES = $(SHELL_PATH_SQ):$(DIFF_SQ):\ $(localedir_SQ):$(USE_GETTEXT_SCHEME):$(SANE_TOOL_PATH_SQ):\ @@ -2716,8 +2716,8 @@ GIT-SCRIPT-DEFINES: FORCE echo "$$FLAGS" >$@; \ fi -$(SCRIPT_SH_GEN) $(SCRIPT_LIB) : % : %.sh generate-script.sh GIT-BUILD-OPTIONS GIT-SCRIPT-DEFINES - $(QUIET_GEN)./generate-script.sh "$<" "$@+" ./GIT-BUILD-OPTIONS && \ +$(SCRIPT_SH_GEN) $(SCRIPT_LIB) : % : %.sh tools/generate-script.sh GIT-BUILD-OPTIONS GIT-SCRIPT-DEFINES + $(QUIET_GEN)./tools/generate-script.sh "$<" "$@+" ./GIT-BUILD-OPTIONS && \ mv $@+ $@ git.rc: git.rc.in GIT-VERSION-GEN GIT-VERSION-FILE @@ -2757,8 +2757,8 @@ endif PERL_DEFINES += $(gitexecdir) $(perllibdir) $(localedir) -$(SCRIPT_PERL_GEN): % : %.perl generate-perl.sh GIT-PERL-DEFINES GIT-PERL-HEADER GIT-VERSION-FILE - $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS ./GIT-VERSION-FILE GIT-PERL-HEADER "$<" "$@+" && \ +$(SCRIPT_PERL_GEN): % : %.perl tools/generate-perl.sh GIT-PERL-DEFINES GIT-PERL-HEADER GIT-VERSION-FILE + $(QUIET_GEN)$(SHELL_PATH) tools/generate-perl.sh ./GIT-BUILD-OPTIONS ./GIT-VERSION-FILE GIT-PERL-HEADER "$<" "$@+" && \ mv $@+ $@ PERL_DEFINES := $(subst $(space),:,$(PERL_DEFINES)) @@ -2786,8 +2786,8 @@ GIT-PERL-HEADER: $(PERL_HEADER_TEMPLATE) GIT-PERL-DEFINES Makefile perllibdir: @echo '$(perllibdir_SQ)' -git-instaweb: git-instaweb.sh generate-script.sh GIT-BUILD-OPTIONS GIT-SCRIPT-DEFINES - $(QUIET_GEN)./generate-script.sh "$<" "$@+" ./GIT-BUILD-OPTIONS && \ +git-instaweb: git-instaweb.sh tools/generate-script.sh GIT-BUILD-OPTIONS GIT-SCRIPT-DEFINES + $(QUIET_GEN)./tools/generate-script.sh "$<" "$@+" ./GIT-BUILD-OPTIONS && \ chmod +x $@+ && \ mv $@+ $@ else # NO_PERL @@ -2804,9 +2804,9 @@ endif # NO_PERL $(SCRIPT_PYTHON_GEN): GIT-BUILD-OPTIONS ifndef NO_PYTHON -$(SCRIPT_PYTHON_GEN): generate-python.sh +$(SCRIPT_PYTHON_GEN): tools/generate-python.sh $(SCRIPT_PYTHON_GEN): % : %.py - $(QUIET_GEN)$(SHELL_PATH) generate-python.sh ./GIT-BUILD-OPTIONS "$<" "$@" + $(QUIET_GEN)$(SHELL_PATH) tools/generate-python.sh ./GIT-BUILD-OPTIONS "$<" "$@" else # NO_PYTHON $(SCRIPT_PYTHON_GEN): % : unimplemented.sh $(QUIET_GEN) \ @@ -3226,9 +3226,9 @@ endif NO_PERL_CPAN_FALLBACKS_SQ = $(subst ','\'',$(NO_PERL_CPAN_FALLBACKS)) endif -perl/build/lib/%.pm: perl/%.pm generate-perl.sh GIT-BUILD-OPTIONS GIT-VERSION-FILE GIT-PERL-DEFINES +perl/build/lib/%.pm: perl/%.pm tools/generate-perl.sh GIT-BUILD-OPTIONS GIT-VERSION-FILE GIT-PERL-DEFINES $(call mkdir_p_parent_template) - $(QUIET_GEN)$(SHELL_PATH) generate-perl.sh ./GIT-BUILD-OPTIONS ./GIT-VERSION-FILE GIT-PERL-HEADER "$<" "$@" + $(QUIET_GEN)$(SHELL_PATH) tools/generate-perl.sh ./GIT-BUILD-OPTIONS ./GIT-VERSION-FILE GIT-PERL-HEADER "$<" "$@" perl/build/man/man3/Git.3pm: perl/Git.pm $(call mkdir_p_parent_template) @@ -3936,7 +3936,7 @@ check-docs:: ### Make sure built-ins do not have dups and listed in git.c # check-builtins:: - ./check-builtins.sh + ./tools/check-builtins.sh ### Test suite coverage testing # diff --git a/check-builtins.sh b/check-builtins.sh deleted file mode 100755 index a0aaf3a347..0000000000 --- a/check-builtins.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/bin/sh - -{ - cat <<\EOF -sayIt: - $(foreach b,$(BUILT_INS),echo XXX $(b:$X=) YYY;) -EOF - cat Makefile -} | -make -f - sayIt 2>/dev/null | -sed -n -e 's/.*XXX \(.*\) YYY.*/\1/p' | -sort | -{ - bad=0 - while read builtin - do - base=$(expr "$builtin" : 'git-\(.*\)') - x=$(sed -ne 's/.*{ "'$base'", \(cmd_[^, ]*\).*/'$base' \1/p' git.c) - if test -z "$x" - then - echo "$base is builtin but not listed in git.c command list" - bad=1 - fi - for sfx in sh perl py - do - if test -f "$builtin.$sfx" - then - echo "$base is builtin but $builtin.$sfx still exists" - bad=1 - fi - done - done - exit $bad -} diff --git a/config.mak.dev b/config.mak.dev index e86b6e1b34..c8dcf78779 100644 --- a/config.mak.dev +++ b/config.mak.dev @@ -1,5 +1,5 @@ ifndef COMPILER_FEATURES -COMPILER_FEATURES := $(shell ./detect-compiler $(CC)) +COMPILER_FEATURES := $(shell ./tools/detect-compiler $(CC)) endif ifeq ($(filter no-error,$(DEVOPTS)),) diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index c6cfb874ef..81b4306e72 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -636,7 +636,7 @@ set(EXCLUSION_PROGS_CACHE ${EXCLUSION_PROGS} CACHE STRING "Programs not built" F if(NOT EXISTS ${CMAKE_BINARY_DIR}/command-list.h OR NOT EXCLUSION_PROGS_CACHE STREQUAL EXCLUSION_PROGS) list(REMOVE_ITEM EXCLUSION_PROGS empty) message("Generating command-list.h") - execute_process(COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/generate-cmdlist.sh" + execute_process(COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/tools/generate-cmdlist.sh" ${EXCLUSION_PROGS} "${CMAKE_SOURCE_DIR}" "${CMAKE_BINARY_DIR}/command-list.h") @@ -644,14 +644,14 @@ endif() if(NOT EXISTS ${CMAKE_BINARY_DIR}/config-list.h) message("Generating config-list.h") - execute_process(COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/generate-configlist.sh" + execute_process(COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/tools/generate-configlist.sh" "${CMAKE_SOURCE_DIR}" "${CMAKE_BINARY_DIR}/config-list.h") endif() if(NOT EXISTS ${CMAKE_BINARY_DIR}/hook-list.h) message("Generating hook-list.h") - execute_process(COMMAND "${SH_EXE}" ${CMAKE_SOURCE_DIR}/generate-hooklist.sh + execute_process(COMMAND "${SH_EXE}" ${CMAKE_SOURCE_DIR}/tools/generate-hooklist.sh "${CMAKE_SOURCE_DIR}" "${CMAKE_BINARY_DIR}/hook-list.h") endif() @@ -832,11 +832,11 @@ foreach(script ${git_shell_scripts}) endif() add_custom_command(OUTPUT "${CMAKE_BINARY_DIR}/${shell_gen_path}" - COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/generate-script.sh" + COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/tools/generate-script.sh" "${CMAKE_SOURCE_DIR}/${script}.sh" "${CMAKE_BINARY_DIR}/${shell_gen_path}" "${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS" - DEPENDS "${CMAKE_SOURCE_DIR}/generate-script.sh" + DEPENDS "${CMAKE_SOURCE_DIR}/tools/generate-script.sh" "${CMAKE_SOURCE_DIR}/${script}.sh" VERBATIM) list(APPEND shell_gen ${CMAKE_BINARY_DIR}/${shell_gen_path}) @@ -875,13 +875,13 @@ foreach(script ${git_perl_scripts} ${perl_modules}) file(MAKE_DIRECTORY "${CMAKE_BINARY_DIR}/${perl_gen_dir}") add_custom_command(OUTPUT "${CMAKE_BINARY_DIR}/${perl_gen_path}" - COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/generate-perl.sh" + COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/tools/generate-perl.sh" "${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS" "${CMAKE_BINARY_DIR}/GIT-VERSION-FILE" "${CMAKE_BINARY_DIR}/GIT-PERL-HEADER" "${CMAKE_SOURCE_DIR}/${script}" "${CMAKE_BINARY_DIR}/${perl_gen_path}" - DEPENDS "${CMAKE_SOURCE_DIR}/generate-perl.sh" + DEPENDS "${CMAKE_SOURCE_DIR}/tools/generate-perl.sh" "${CMAKE_SOURCE_DIR}/${script}" "${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS" "${CMAKE_BINARY_DIR}/GIT-VERSION-FILE" @@ -892,11 +892,11 @@ add_custom_target(perl-gen ALL DEPENDS ${perl_gen}) # Python script add_custom_command(OUTPUT "${CMAKE_BINARY_DIR}/git-p4" - COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/generate-python.sh" + COMMAND "${SH_EXE}" "${CMAKE_SOURCE_DIR}/tools/generate-python.sh" "${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS" "${CMAKE_SOURCE_DIR}/git-p4.py" "${CMAKE_BINARY_DIR}/git-p4" - DEPENDS "${CMAKE_SOURCE_DIR}/generate-python.sh" + DEPENDS "${CMAKE_SOURCE_DIR}/tools/generate-python.sh" "${CMAKE_SOURCE_DIR}/git-p4.py" "${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS" VERBATIM) diff --git a/contrib/subtree/meson.build b/contrib/subtree/meson.build index 161435abeb..804c315894 100644 --- a/contrib/subtree/meson.build +++ b/contrib/subtree/meson.build @@ -3,7 +3,7 @@ git_subtree = custom_target( output: 'git-subtree', command: [ shell, - meson.project_source_root() / 'generate-script.sh', + meson.project_source_root() / 'tools/generate-script.sh', '@INPUT@', '@OUTPUT@', meson.project_build_root() / 'GIT-BUILD-OPTIONS', diff --git a/detect-compiler b/detect-compiler deleted file mode 100755 index 124ebdd4c9..0000000000 --- a/detect-compiler +++ /dev/null @@ -1,58 +0,0 @@ -#!/bin/sh -# -# Probe the compiler for vintage, version, etc. This is used for setting -# optional make knobs under the DEVELOPER knob. - -CC="$*" - -# we get something like (this is at least true for gcc and clang) -# -# FreeBSD clang version 3.4.1 (tags/RELEASE...) -get_version_line() { - LANG=C LC_ALL=C $CC -v 2>&1 | sed -n '/ version /{p;q;}' -} - -get_family() { - get_version_line | sed 's/^\(.*\) version [0-9].*/\1/' -} - -get_version() { - # A string that begins with a digit up to the next SP - ver=$(get_version_line | sed 's/^.* version \([0-9][^ ]*\).*/\1/') - - # There are known -variant suffixes that do not affect the - # meaning of the main version number. Strip them. - ver=${ver%-win32} - ver=${ver%-posix} - - echo "$ver" -} - -print_flags() { - family=$1 - version=$(get_version | cut -f 1 -d .) - - # Print a feature flag not only for the current version, but also - # for any prior versions we encompass. This avoids needing to do - # numeric comparisons in make, which are awkward. - while test "$version" -gt 0 - do - echo $family$version - version=$((version - 1)) - done -} - -case "$(get_family)" in -gcc) - print_flags gcc - ;; -clang | *" clang") - print_flags clang - ;; -"Apple LLVM") - print_flags clang - ;; -*) - : unknown compiler family - ;; -esac diff --git a/generate-cmdlist.sh b/generate-cmdlist.sh deleted file mode 100755 index 0ed39c4c5d..0000000000 --- a/generate-cmdlist.sh +++ /dev/null @@ -1,120 +0,0 @@ -#!/bin/sh - -die () { - echo "$@" >&2 - exit 1 -} - -command_list () { - while read cmd rest - do - case "$cmd" in - "#"* | '') - # Ignore comments and allow empty lines - continue - ;; - *) - case "$exclude_programs" in - *":$cmd:"*) - ;; - *) - echo "$cmd $rest" - ;; - esac - esac - done <"$1" -} - -category_list () { - echo "$1" | - cut -d' ' -f2- | - tr ' ' '\012' | - grep -v '^$' | - LC_ALL=C sort -u -} - -define_categories () { - echo - echo "/* Command categories */" - bit=0 - echo "$1" | - while read cat - do - echo "#define CAT_$cat (1UL << $bit)" - bit=$(($bit+1)) - done - test "$bit" -gt 32 && die "Urgh.. too many categories?" -} - -define_category_names () { - echo - echo "/* Category names */" - echo "static const char *category_names[] = {" - bit=0 - echo "$1" | - while read cat - do - echo " \"$cat\", /* (1UL << $bit) */" - bit=$(($bit+1)) - done - echo " NULL" - echo "};" -} - -print_command_list () { - echo "static struct cmdname_help command_list[] = {" - - echo "$2" | - while read cmd rest - do - synopsis= - while read line - do - case "$line" in - "$cmd - "*) - synopsis=${line#$cmd - } - break - ;; - esac - done <"$1/Documentation/$cmd.adoc" - - printf '\t{ "%s", N_("%s"), 0' "$cmd" "$synopsis" - printf " | CAT_%s" $rest - echo " }," - done - echo "};" -} - -exclude_programs=: -while test "--exclude-program" = "$1" -do - shift - exclude_programs="$exclude_programs$1:" - shift -done - -if test "$#" -ne 2 -then - die "USAGE: $0 " -fi - -SOURCE_DIR="$1" -OUTPUT="$2" - -{ - commands="$(command_list "$SOURCE_DIR"/command-list.txt)" - categories="$(category_list "$commands")" - - echo "/* Automatically generated by generate-cmdlist.sh */ - struct cmdname_help { - const char *name; - const char *help; - uint32_t category; - }; - " - define_categories "$categories" - echo - define_category_names "$categories" - echo - print_command_list "$SOURCE_DIR" "$commands" -} >"$OUTPUT" diff --git a/generate-configlist.sh b/generate-configlist.sh deleted file mode 100755 index e28054f9e0..0000000000 --- a/generate-configlist.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/bin/sh - -SOURCE_DIR="$1" -OUTPUT="$2" -DEPFILE="$3" - -if test -z "$SOURCE_DIR" || ! test -d "$SOURCE_DIR" || test -z "$OUTPUT" -then - echo >&2 "USAGE: $0 []" - exit 1 -fi - -print_config_list () { - cat <"$OUTPUT" - -if test -n "$DEPFILE" -then - QUOTED_OUTPUT="$(printf '%s\n' "$OUTPUT" | sed 's,[&/\],\\&,g')" - { - printf '%s\n' "$SOURCE_DIR"/Documentation/*config.adoc \ - "$SOURCE_DIR"/Documentation/config/*.adoc | - sed -e 's/[# ]/\\&/g' -e "s/^/$QUOTED_OUTPUT: /" - printf '%s:\n' "$SOURCE_DIR"/Documentation/*config.adoc \ - "$SOURCE_DIR"/Documentation/config/*.adoc | - sed -e 's/[# ]/\\&/g' - } >"$DEPFILE" -fi diff --git a/generate-hooklist.sh b/generate-hooklist.sh deleted file mode 100755 index e0cdf26944..0000000000 --- a/generate-hooklist.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/sh -# -# Usage: ./generate-hooklist.sh >hook-list.h - -SOURCE_DIR="$1" -OUTPUT="$2" - -if test -z "$SOURCE_DIR" || ! test -d "$SOURCE_DIR" || test -z "$OUTPUT" -then - echo >&2 "USAGE: $0 " - exit 1 -fi - -{ - -cat <"$OUTPUT" diff --git a/generate-perl.sh b/generate-perl.sh deleted file mode 100755 index 796d835932..0000000000 --- a/generate-perl.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/bin/sh - -set -e - -if test $# -ne 5 -then - echo >&2 "USAGE: $0 " - exit 1 -fi - -GIT_BUILD_OPTIONS="$1" -GIT_VERSION_FILE="$2" -PERL_HEADER="$3" -INPUT="$4" -OUTPUT="$5" - -. "$GIT_BUILD_OPTIONS" -. "$GIT_VERSION_FILE" - -sed -e '1{' \ - -e " /^#!.*perl/!b" \ - -e " s|#!.*perl|#!$PERL_PATH|" \ - -e " r $PERL_HEADER" \ - -e ' G' \ - -e '}' \ - -e "s|@GIT_VERSION@|$GIT_VERSION|g" \ - -e "s|@LOCALEDIR@|$PERL_LOCALEDIR|g" \ - -e "s|@NO_GETTEXT@|$NO_GETTEXT|g" \ - -e "s|@NO_PERL_CPAN_FALLBACKS@|$NO_PERL_CPAN_FALLBACKS|g" \ - "$INPUT" >"$OUTPUT" - -case "$INPUT" in -*.perl|*git-contacts) - chmod a+x "$OUTPUT";; -*) - ;; -esac diff --git a/generate-python.sh b/generate-python.sh deleted file mode 100755 index 31ac115689..0000000000 --- a/generate-python.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/bin/sh - -set -e - -if test $# -ne 3 -then - echo >&2 "USAGE: $0 " - exit 1 -fi - -GIT_BUILD_OPTIONS="$1" -INPUT="$2" -OUTPUT="$3" - -. "$GIT_BUILD_OPTIONS" - -sed -e "1s|#!.*python|#!$PYTHON_PATH|" \ - "$INPUT" >"$OUTPUT+" -chmod a+x "$OUTPUT+" -mv "$OUTPUT+" "$OUTPUT" diff --git a/generate-script.sh b/generate-script.sh deleted file mode 100755 index a149e4f0ba..0000000000 --- a/generate-script.sh +++ /dev/null @@ -1,34 +0,0 @@ -#!/bin/sh - -set -e - -if test $# -ne 3 -then - echo >&2 "USAGE: $0 " - exit 1 -fi - -INPUT="$1" -OUTPUT="$2" -BUILD_OPTIONS="$3" - -. "$BUILD_OPTIONS" - -sed -e "1s|#!.*/sh|#!$SHELL_PATH|" \ - -e "s|@SHELL_PATH@|$SHELL_PATH|" \ - -e "s|@DIFF@|$DIFF|" \ - -e "s|@LOCALEDIR@|$LOCALEDIR|g" \ - -e "s/@USE_GETTEXT_SCHEME@/$USE_GETTEXT_SCHEME/g" \ - -e "$BROKEN_PATH_FIX" \ - -e "s|@GITWEBDIR@|$GITWEBDIR|g" \ - -e "s|@PERL_PATH@|$PERL_PATH|g" \ - -e "s|@PAGER_ENV@|$PAGER_ENV|g" \ - "$INPUT" >"$OUTPUT" - -case "$(basename "$INPUT")" in -git-mergetool--lib.sh|git-sh-i18n.sh|git-sh-setup.sh) - ;; -*) - chmod a+x "$OUTPUT" - ;; -esac diff --git a/meson.build b/meson.build index 1d66b5181e..604fe89d2d 100644 --- a/meson.build +++ b/meson.build @@ -554,7 +554,7 @@ libgit_sources = [ libgit_sources += custom_target( input: 'command-list.txt', output: 'command-list.h', - command: [shell, meson.current_source_dir() + '/generate-cmdlist.sh', meson.current_source_dir(), '@OUTPUT@'], + command: [shell, meson.current_source_dir() + '/tools/generate-cmdlist.sh', meson.current_source_dir(), '@OUTPUT@'], env: script_environment, ) @@ -723,10 +723,10 @@ endif builtin_sources += custom_target( output: 'config-list.h', depfile: 'config-list.h.d', - depend_files: [ 'generate-configlist.sh' ], + depend_files: [ 'tools/generate-configlist.sh' ], command: [ shell, - meson.current_source_dir() / 'generate-configlist.sh', + meson.current_source_dir() / 'tools/generate-configlist.sh', meson.current_source_dir(), '@OUTPUT@', '@DEPFILE@', @@ -739,7 +739,7 @@ builtin_sources += custom_target( output: 'hook-list.h', command: [ shell, - meson.current_source_dir() + '/generate-hooklist.sh', + meson.current_source_dir() + '/tools/generate-hooklist.sh', meson.current_source_dir(), '@OUTPUT@', ], @@ -1959,7 +1959,7 @@ foreach script : scripts_sh output: fs.stem(script), command: [ shell, - meson.project_source_root() / 'generate-script.sh', + meson.project_source_root() / 'tools/generate-script.sh', '@INPUT@', '@OUTPUT@', meson.project_build_root() / 'GIT-BUILD-OPTIONS', @@ -2008,7 +2008,7 @@ if perl_features_enabled generate_perl_command = [ shell, - meson.project_source_root() / 'generate-perl.sh', + meson.project_source_root() / 'tools/generate-perl.sh', meson.project_build_root() / 'GIT-BUILD-OPTIONS', git_version_file.full_path(), perl_header, @@ -2057,7 +2057,7 @@ if target_python.found() output: fs.stem(script), command: [ shell, - meson.project_source_root() / 'generate-python.sh', + meson.project_source_root() / 'tools/generate-python.sh', meson.project_build_root() / 'GIT-BUILD-OPTIONS', '@INPUT@', '@OUTPUT@', diff --git a/tools/check-builtins.sh b/tools/check-builtins.sh new file mode 100755 index 0000000000..a0aaf3a347 --- /dev/null +++ b/tools/check-builtins.sh @@ -0,0 +1,34 @@ +#!/bin/sh + +{ + cat <<\EOF +sayIt: + $(foreach b,$(BUILT_INS),echo XXX $(b:$X=) YYY;) +EOF + cat Makefile +} | +make -f - sayIt 2>/dev/null | +sed -n -e 's/.*XXX \(.*\) YYY.*/\1/p' | +sort | +{ + bad=0 + while read builtin + do + base=$(expr "$builtin" : 'git-\(.*\)') + x=$(sed -ne 's/.*{ "'$base'", \(cmd_[^, ]*\).*/'$base' \1/p' git.c) + if test -z "$x" + then + echo "$base is builtin but not listed in git.c command list" + bad=1 + fi + for sfx in sh perl py + do + if test -f "$builtin.$sfx" + then + echo "$base is builtin but $builtin.$sfx still exists" + bad=1 + fi + done + done + exit $bad +} diff --git a/tools/detect-compiler b/tools/detect-compiler new file mode 100755 index 0000000000..124ebdd4c9 --- /dev/null +++ b/tools/detect-compiler @@ -0,0 +1,58 @@ +#!/bin/sh +# +# Probe the compiler for vintage, version, etc. This is used for setting +# optional make knobs under the DEVELOPER knob. + +CC="$*" + +# we get something like (this is at least true for gcc and clang) +# +# FreeBSD clang version 3.4.1 (tags/RELEASE...) +get_version_line() { + LANG=C LC_ALL=C $CC -v 2>&1 | sed -n '/ version /{p;q;}' +} + +get_family() { + get_version_line | sed 's/^\(.*\) version [0-9].*/\1/' +} + +get_version() { + # A string that begins with a digit up to the next SP + ver=$(get_version_line | sed 's/^.* version \([0-9][^ ]*\).*/\1/') + + # There are known -variant suffixes that do not affect the + # meaning of the main version number. Strip them. + ver=${ver%-win32} + ver=${ver%-posix} + + echo "$ver" +} + +print_flags() { + family=$1 + version=$(get_version | cut -f 1 -d .) + + # Print a feature flag not only for the current version, but also + # for any prior versions we encompass. This avoids needing to do + # numeric comparisons in make, which are awkward. + while test "$version" -gt 0 + do + echo $family$version + version=$((version - 1)) + done +} + +case "$(get_family)" in +gcc) + print_flags gcc + ;; +clang | *" clang") + print_flags clang + ;; +"Apple LLVM") + print_flags clang + ;; +*) + : unknown compiler family + ;; +esac diff --git a/tools/generate-cmdlist.sh b/tools/generate-cmdlist.sh new file mode 100755 index 0000000000..0ed39c4c5d --- /dev/null +++ b/tools/generate-cmdlist.sh @@ -0,0 +1,120 @@ +#!/bin/sh + +die () { + echo "$@" >&2 + exit 1 +} + +command_list () { + while read cmd rest + do + case "$cmd" in + "#"* | '') + # Ignore comments and allow empty lines + continue + ;; + *) + case "$exclude_programs" in + *":$cmd:"*) + ;; + *) + echo "$cmd $rest" + ;; + esac + esac + done <"$1" +} + +category_list () { + echo "$1" | + cut -d' ' -f2- | + tr ' ' '\012' | + grep -v '^$' | + LC_ALL=C sort -u +} + +define_categories () { + echo + echo "/* Command categories */" + bit=0 + echo "$1" | + while read cat + do + echo "#define CAT_$cat (1UL << $bit)" + bit=$(($bit+1)) + done + test "$bit" -gt 32 && die "Urgh.. too many categories?" +} + +define_category_names () { + echo + echo "/* Category names */" + echo "static const char *category_names[] = {" + bit=0 + echo "$1" | + while read cat + do + echo " \"$cat\", /* (1UL << $bit) */" + bit=$(($bit+1)) + done + echo " NULL" + echo "};" +} + +print_command_list () { + echo "static struct cmdname_help command_list[] = {" + + echo "$2" | + while read cmd rest + do + synopsis= + while read line + do + case "$line" in + "$cmd - "*) + synopsis=${line#$cmd - } + break + ;; + esac + done <"$1/Documentation/$cmd.adoc" + + printf '\t{ "%s", N_("%s"), 0' "$cmd" "$synopsis" + printf " | CAT_%s" $rest + echo " }," + done + echo "};" +} + +exclude_programs=: +while test "--exclude-program" = "$1" +do + shift + exclude_programs="$exclude_programs$1:" + shift +done + +if test "$#" -ne 2 +then + die "USAGE: $0 " +fi + +SOURCE_DIR="$1" +OUTPUT="$2" + +{ + commands="$(command_list "$SOURCE_DIR"/command-list.txt)" + categories="$(category_list "$commands")" + + echo "/* Automatically generated by generate-cmdlist.sh */ + struct cmdname_help { + const char *name; + const char *help; + uint32_t category; + }; + " + define_categories "$categories" + echo + define_category_names "$categories" + echo + print_command_list "$SOURCE_DIR" "$commands" +} >"$OUTPUT" diff --git a/tools/generate-configlist.sh b/tools/generate-configlist.sh new file mode 100755 index 0000000000..e28054f9e0 --- /dev/null +++ b/tools/generate-configlist.sh @@ -0,0 +1,52 @@ +#!/bin/sh + +SOURCE_DIR="$1" +OUTPUT="$2" +DEPFILE="$3" + +if test -z "$SOURCE_DIR" || ! test -d "$SOURCE_DIR" || test -z "$OUTPUT" +then + echo >&2 "USAGE: $0 []" + exit 1 +fi + +print_config_list () { + cat <"$OUTPUT" + +if test -n "$DEPFILE" +then + QUOTED_OUTPUT="$(printf '%s\n' "$OUTPUT" | sed 's,[&/\],\\&,g')" + { + printf '%s\n' "$SOURCE_DIR"/Documentation/*config.adoc \ + "$SOURCE_DIR"/Documentation/config/*.adoc | + sed -e 's/[# ]/\\&/g' -e "s/^/$QUOTED_OUTPUT: /" + printf '%s:\n' "$SOURCE_DIR"/Documentation/*config.adoc \ + "$SOURCE_DIR"/Documentation/config/*.adoc | + sed -e 's/[# ]/\\&/g' + } >"$DEPFILE" +fi diff --git a/tools/generate-hooklist.sh b/tools/generate-hooklist.sh new file mode 100755 index 0000000000..e0cdf26944 --- /dev/null +++ b/tools/generate-hooklist.sh @@ -0,0 +1,33 @@ +#!/bin/sh +# +# Usage: ./generate-hooklist.sh >hook-list.h + +SOURCE_DIR="$1" +OUTPUT="$2" + +if test -z "$SOURCE_DIR" || ! test -d "$SOURCE_DIR" || test -z "$OUTPUT" +then + echo >&2 "USAGE: $0 " + exit 1 +fi + +{ + +cat <"$OUTPUT" diff --git a/tools/generate-perl.sh b/tools/generate-perl.sh new file mode 100755 index 0000000000..796d835932 --- /dev/null +++ b/tools/generate-perl.sh @@ -0,0 +1,37 @@ +#!/bin/sh + +set -e + +if test $# -ne 5 +then + echo >&2 "USAGE: $0 " + exit 1 +fi + +GIT_BUILD_OPTIONS="$1" +GIT_VERSION_FILE="$2" +PERL_HEADER="$3" +INPUT="$4" +OUTPUT="$5" + +. "$GIT_BUILD_OPTIONS" +. "$GIT_VERSION_FILE" + +sed -e '1{' \ + -e " /^#!.*perl/!b" \ + -e " s|#!.*perl|#!$PERL_PATH|" \ + -e " r $PERL_HEADER" \ + -e ' G' \ + -e '}' \ + -e "s|@GIT_VERSION@|$GIT_VERSION|g" \ + -e "s|@LOCALEDIR@|$PERL_LOCALEDIR|g" \ + -e "s|@NO_GETTEXT@|$NO_GETTEXT|g" \ + -e "s|@NO_PERL_CPAN_FALLBACKS@|$NO_PERL_CPAN_FALLBACKS|g" \ + "$INPUT" >"$OUTPUT" + +case "$INPUT" in +*.perl|*git-contacts) + chmod a+x "$OUTPUT";; +*) + ;; +esac diff --git a/tools/generate-python.sh b/tools/generate-python.sh new file mode 100755 index 0000000000..31ac115689 --- /dev/null +++ b/tools/generate-python.sh @@ -0,0 +1,20 @@ +#!/bin/sh + +set -e + +if test $# -ne 3 +then + echo >&2 "USAGE: $0 " + exit 1 +fi + +GIT_BUILD_OPTIONS="$1" +INPUT="$2" +OUTPUT="$3" + +. "$GIT_BUILD_OPTIONS" + +sed -e "1s|#!.*python|#!$PYTHON_PATH|" \ + "$INPUT" >"$OUTPUT+" +chmod a+x "$OUTPUT+" +mv "$OUTPUT+" "$OUTPUT" diff --git a/tools/generate-script.sh b/tools/generate-script.sh new file mode 100755 index 0000000000..a149e4f0ba --- /dev/null +++ b/tools/generate-script.sh @@ -0,0 +1,34 @@ +#!/bin/sh + +set -e + +if test $# -ne 3 +then + echo >&2 "USAGE: $0 " + exit 1 +fi + +INPUT="$1" +OUTPUT="$2" +BUILD_OPTIONS="$3" + +. "$BUILD_OPTIONS" + +sed -e "1s|#!.*/sh|#!$SHELL_PATH|" \ + -e "s|@SHELL_PATH@|$SHELL_PATH|" \ + -e "s|@DIFF@|$DIFF|" \ + -e "s|@LOCALEDIR@|$LOCALEDIR|g" \ + -e "s/@USE_GETTEXT_SCHEME@/$USE_GETTEXT_SCHEME/g" \ + -e "$BROKEN_PATH_FIX" \ + -e "s|@GITWEBDIR@|$GITWEBDIR|g" \ + -e "s|@PERL_PATH@|$PERL_PATH|g" \ + -e "s|@PAGER_ENV@|$PAGER_ENV|g" \ + "$INPUT" >"$OUTPUT" + +case "$(basename "$INPUT")" in +git-mergetool--lib.sh|git-sh-i18n.sh|git-sh-setup.sh) + ;; +*) + chmod a+x "$OUTPUT" + ;; +esac -- cgit v1.3 From baa61e46da8f501e3b2d1900486255a546d3535b Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 19 Mar 2026 06:33:25 +0100 Subject: git-compat-util.h: move warning infra to prepare for PCHs The "git-compat-util.h" header is supposed to be the first header included by every code compilation unit. As such, a subsequent commit will start to precompile this header to speed up compilation of Git. This will cause an issue though with the way that we have set up the "-Wsign-compare" warnings. It is expected that any compilation unit that fails with that compiler warning sets `DISABLE_SIGN_COMPARE_WARNINGS` before including "git-compat-util.h". If so, we'll disable the warning right away via a compiler pragma. But with precompiled headers we do not know ahead of time whether the code unit wants to disable those warnings, and thus we'll have to precompile the header without defining `DISABLE_SIGN_COMPARE_WARNINGS`. But as the pragma statement is wrapped by our include guards, the second include of that file will not have the desired effect of disabling the warnings anymore. We could fix this issue by declaring a new macro that compilation units are expected to invoke after having included the file. In retrospect, that would have been the better way to handle this as it allows for more flexibility: we could for example toggle the warning for specific code blocks, only. But changing this now would require a bunch of changes, and the churn feels excessive for what we gain. Instead, prepare for the precompiled headers by moving the code outside of the include guards. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- git-compat-util.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/git-compat-util.h b/git-compat-util.h index bebcf9f698..4b4ea2498f 100644 --- a/git-compat-util.h +++ b/git-compat-util.h @@ -34,10 +34,6 @@ struct strbuf; # define DISABLE_WARNING(warning) #endif -#ifdef DISABLE_SIGN_COMPARE_WARNINGS -DISABLE_WARNING(-Wsign-compare) -#endif - #undef FLEX_ARRAY #define FLEX_ARRAY /* empty - weather balloon to require C99 FAM */ @@ -1099,3 +1095,7 @@ extern int not_supposed_to_survive; #endif /* CHECK_ASSERTION_SIDE_EFFECTS */ #endif + +#ifdef DISABLE_SIGN_COMPARE_WARNINGS +DISABLE_WARNING(-Wsign-compare) +#endif -- cgit v1.3 From daa56fb7897ce8819e5a64f4ec7800b3eef03031 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 19 Mar 2026 06:33:26 +0100 Subject: meson: compile compatibility sources separately In the next commit we're about to introduce a precompiled header for "git-compat-util.h". The consequence of this change is that we'll implicitly include that header for every compilation unit that uses the precompiled headers. This is okay for our "normal" library sources and our builtins. But some of our compatibility sources do not include the header on purpose, and doing so would cause compilation errors. Prepare for this change by splitting out compatibility sources into their static library. Like this, we can selectively enable precompiled headers for the library sources. Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- meson.build | 79 +++++++++++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 34 deletions(-) diff --git a/meson.build b/meson.build index 604fe89d2d..cd00be1c23 100644 --- a/meson.build +++ b/meson.build @@ -271,6 +271,13 @@ version_gen_environment.set('GIT_VERSION', get_option('version')) compiler = meson.get_compiler('c') +compat_sources = [ + 'compat/nonblock.c', + 'compat/obstack.c', + 'compat/open.c', + 'compat/terminal.c', +] + libgit_sources = [ 'abspath.c', 'add-interactive.c', @@ -304,10 +311,6 @@ libgit_sources = [ 'commit.c', 'common-exit.c', 'common-init.c', - 'compat/nonblock.c', - 'compat/obstack.c', - 'compat/open.c', - 'compat/terminal.c', 'compiler-tricks/not-constant.c', 'config.c', 'connect.c', @@ -1163,7 +1166,7 @@ endif if not has_poll_h and not has_sys_poll_h libgit_c_args += '-DNO_POLL' - libgit_sources += 'compat/poll/poll.c' + compat_sources += 'compat/poll/poll.c' libgit_include_directories += 'compat/poll' endif @@ -1179,7 +1182,7 @@ endif # implementation to threat things like drive prefixes specially. if host_machine.system() == 'windows' or not compiler.has_header('libgen.h') libgit_c_args += '-DNO_LIBGEN_H' - libgit_sources += 'compat/basename.c' + compat_sources += 'compat/basename.c' endif if compiler.has_header('paths.h') @@ -1209,7 +1212,7 @@ if host_machine.system() != 'windows' foreach symbol : ['inet_ntop', 'inet_pton', 'hstrerror'] if not compiler.has_function(symbol, dependencies: networking_dependencies) libgit_c_args += '-DNO_' + symbol.to_upper() - libgit_sources += 'compat/' + symbol + '.c' + compat_sources += 'compat/' + symbol + '.c' endif endforeach endif @@ -1251,18 +1254,18 @@ else endif if host_machine.system() == 'darwin' - libgit_sources += 'compat/precompose_utf8.c' + compat_sources += 'compat/precompose_utf8.c' libgit_c_args += '-DPRECOMPOSE_UNICODE' libgit_c_args += '-DPROTECT_HFS_DEFAULT' endif # Configure general compatibility wrappers. if host_machine.system() == 'cygwin' - libgit_sources += [ + compat_sources += [ 'compat/win32/path-utils.c', ] elif host_machine.system() == 'windows' - libgit_sources += [ + compat_sources += [ 'compat/winansi.c', 'compat/win32/dirent.c', 'compat/win32/flush.c', @@ -1289,20 +1292,20 @@ elif host_machine.system() == 'windows' libgit_include_directories += 'compat/win32' if compiler.get_id() == 'msvc' libgit_include_directories += 'compat/vcbuild/include' - libgit_sources += 'compat/msvc.c' + compat_sources += 'compat/msvc.c' else - libgit_sources += 'compat/mingw.c' + compat_sources += 'compat/mingw.c' endif endif if host_machine.system() == 'linux' - libgit_sources += 'compat/linux/procinfo.c' + compat_sources += 'compat/linux/procinfo.c' elif host_machine.system() == 'windows' - libgit_sources += 'compat/win32/trace2_win32_process_info.c' + compat_sources += 'compat/win32/trace2_win32_process_info.c' elif host_machine.system() == 'darwin' - libgit_sources += 'compat/darwin/procinfo.c' + compat_sources += 'compat/darwin/procinfo.c' else - libgit_sources += 'compat/stub/procinfo.c' + compat_sources += 'compat/stub/procinfo.c' endif if host_machine.system() == 'cygwin' or host_machine.system() == 'windows' @@ -1315,13 +1318,13 @@ endif # Configure the simple-ipc subsystem required fro the fsmonitor. if host_machine.system() == 'windows' - libgit_sources += [ + compat_sources += [ 'compat/simple-ipc/ipc-shared.c', 'compat/simple-ipc/ipc-win32.c', ] libgit_c_args += '-DSUPPORTS_SIMPLE_IPC' else - libgit_sources += [ + compat_sources += [ 'compat/simple-ipc/ipc-shared.c', 'compat/simple-ipc/ipc-unix-socket.c', ] @@ -1339,7 +1342,7 @@ if fsmonitor_backend != '' libgit_c_args += '-DHAVE_FSMONITOR_DAEMON_BACKEND' libgit_c_args += '-DHAVE_FSMONITOR_OS_SETTINGS' - libgit_sources += [ + compat_sources += [ 'compat/fsmonitor/fsm-health-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-ipc-' + fsmonitor_backend + '.c', 'compat/fsmonitor/fsm-listen-' + fsmonitor_backend + '.c', @@ -1355,7 +1358,7 @@ if not get_option('b_sanitize').contains('address') and get_option('regex').allo if compiler.get_define('REG_ENHANCED', prefix: '#include ') != '' libgit_c_args += '-DUSE_ENHANCED_BASIC_REGULAR_EXPRESSIONS' - libgit_sources += 'compat/regcomp_enhanced.c' + compat_sources += 'compat/regcomp_enhanced.c' endif elif not get_option('regex').enabled() libgit_c_args += [ @@ -1364,7 +1367,7 @@ elif not get_option('regex').enabled() '-DNO_MBSUPPORT', ] build_options_config.set('NO_REGEX', '1') - libgit_sources += 'compat/regex/regex.c' + compat_sources += 'compat/regex/regex.c' libgit_include_directories += 'compat/regex' else error('Native regex support requested but not found') @@ -1428,7 +1431,7 @@ else if get_option('b_sanitize').contains('address') libgit_c_args += '-DNO_MMAP' - libgit_sources += 'compat/mmap.c' + compat_sources += 'compat/mmap.c' else checkfuncs += { 'mmap': ['mmap.c'] } endif @@ -1438,7 +1441,7 @@ foreach func, impls : checkfuncs if not compiler.has_function(func) libgit_c_args += '-DNO_' + func.to_upper() foreach impl : impls - libgit_sources += 'compat/' + impl + compat_sources += 'compat/' + impl endforeach endif endforeach @@ -1449,13 +1452,13 @@ endif if not compiler.has_function('strdup') libgit_c_args += '-DOVERRIDE_STRDUP' - libgit_sources += 'compat/strdup.c' + compat_sources += 'compat/strdup.c' endif if not compiler.has_function('qsort') libgit_c_args += '-DINTERNAL_QSORT' endif -libgit_sources += 'compat/qsort_s.c' +compat_sources += 'compat/qsort_s.c' if compiler.has_function('getdelim') libgit_c_args += '-DHAVE_GETDELIM' @@ -1511,7 +1514,7 @@ if meson.can_run_host_binaries() and compiler.run(''' } ''', name: 'fread reads directories').returncode() == 0 libgit_c_args += '-DFREAD_READS_DIRECTORIES' - libgit_sources += 'compat/fopen.c' + compat_sources += 'compat/fopen.c' endif if not meson.is_cross_build() and fs.exists('/dev/tty') @@ -1745,14 +1748,22 @@ else endif libgit = declare_dependency( - link_with: static_library('git', - sources: libgit_sources, - c_args: libgit_c_args + [ - '-DGIT_VERSION_H="' + version_def_h.full_path() + '"', - ], - dependencies: libgit_dependencies, - include_directories: libgit_include_directories, - ), + link_with: [ + static_library('compat', + sources: compat_sources, + c_args: libgit_c_args, + dependencies: libgit_dependencies, + include_directories: libgit_include_directories, + ), + static_library('git', + sources: libgit_sources, + c_args: libgit_c_args + [ + '-DGIT_VERSION_H="' + version_def_h.full_path() + '"', + ], + dependencies: libgit_dependencies, + include_directories: libgit_include_directories, + ), + ], compile_args: libgit_c_args, dependencies: libgit_dependencies, include_directories: libgit_include_directories, -- cgit v1.3 From 671df48df895fdf259b48a7f90b70b7c75fc4059 Mon Sep 17 00:00:00 2001 From: Patrick Steinhardt Date: Thu, 19 Mar 2026 06:33:27 +0100 Subject: meson: precompile "git-compat-util.h" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every compilation unit in Git is expected to include "git-compat-util.h" first, either directly or indirectly via "builtin.h". This header papers over differences between platforms so that we can expect the typical POSIX functions to exist. Furthermore, it provides functionality that we end up using everywhere. This header is thus quite heavy as a consequence. Preprocessing it as a standalone unit via `clang -E git-compat-util.h` yields over 23,000 lines of code overall. Naturally, it takes quite some time to compile all of this. Luckily, this is exactly the kind of use case that precompiled headers aim to solve: instead of recompiling it every single time, we compile it once and then link the result into the executable. If include guards are set up properly it means that the file won't need to be reprocessed. Set up such a precompiled header for "git-compat-util.h" and wire it up via Meson. This causes Meson to implicitly include the precompiled header in all compilation units. With GCC and Clang for example this is done via the "-include" statement [1]. This leads to a significant speedup when performing full builds: Benchmark 1: ninja (rev = HEAD~) Time (mean ± σ): 14.467 s ± 0.126 s [User: 248.133 s, System: 31.298 s] Range (min … max): 14.195 s … 14.633 s 10 runs Benchmark 2: ninja (rev = HEAD) Time (mean ± σ): 10.307 s ± 0.111 s [User: 173.290 s, System: 23.998 s] Range (min … max): 10.030 s … 10.433 s 10 runs Summary ninja (rev = HEAD) ran 1.40 ± 0.02 times faster than ninja (rev = HEAD~) [1]: https://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html Signed-off-by: Patrick Steinhardt Signed-off-by: Junio C Hamano --- meson.build | 2 ++ tools/precompiled.h | 1 + 2 files changed, 3 insertions(+) create mode 100644 tools/precompiled.h diff --git a/meson.build b/meson.build index cd00be1c23..2002f4795e 100644 --- a/meson.build +++ b/meson.build @@ -1760,6 +1760,7 @@ libgit = declare_dependency( c_args: libgit_c_args + [ '-DGIT_VERSION_H="' + version_def_h.full_path() + '"', ], + c_pch: 'tools/precompiled.h', dependencies: libgit_dependencies, include_directories: libgit_include_directories, ), @@ -1820,6 +1821,7 @@ test_dependencies = [ ] git_builtin = executable('git', sources: builtin_sources + 'git.c', + c_pch: 'tools/precompiled.h', dependencies: [libgit_commonmain], install: true, install_dir: git_exec_path, diff --git a/tools/precompiled.h b/tools/precompiled.h new file mode 100644 index 0000000000..b2bec0d2b4 --- /dev/null +++ b/tools/precompiled.h @@ -0,0 +1 @@ +#include "git-compat-util.h" -- cgit v1.3