From b6faef396f0380237deae144ee48caf5ee20ae1e Mon Sep 17 00:00:00 2001 From: Victoria Dye Date: Fri, 2 Sep 2022 15:56:42 +0000 Subject: scalar: fix command documentation section header Rename the last section header in 'contrib/scalar/scalar.txt' from "Scalar" to "GIT". The linting rules of the 'documentation' CI build enforce the existence of a "GIT" section in command documentation. Although 'scalar.txt' is not yet checked, it will be in a future patch. Here, changing the header name is more appropriate than making a Scalar-specific exception to the linting rule. The existing "Scalar" section contains only a link back to the main Git documentation, essentially the same as the "GIT" section in builtin documentation. Changing the section name further clarifies the Scalar-Git association and maintains consistency with the rest of Git. Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- contrib/scalar/scalar.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt index 1a12dc4507..f33436c7f6 100644 --- a/contrib/scalar/scalar.txt +++ b/contrib/scalar/scalar.txt @@ -161,6 +161,6 @@ SEE ALSO -------- linkgit:git-clone[1], linkgit:git-maintenance[1]. -Scalar +GIT --- -Associated with the linkgit:git[1] suite +Part of the linkgit:git[1] suite -- cgit v1.3 From 7b5c93c6c6847b4b6037e38418bc8bbb8c2eada8 Mon Sep 17 00:00:00 2001 From: Victoria Dye Date: Fri, 2 Sep 2022 15:56:43 +0000 Subject: scalar: include in standard Git build & installation Move 'scalar' out of 'contrib/' and into the root of the Git tree. The goal of this change is to build 'scalar' as part of the standard Git build & install processes. This patch includes both the physical move of Scalar's files out of 'contrib/' ('scalar.c', 'scalar.txt', and 't9xxx-scalar.sh'), and the changes to the build definitions in 'Makefile' and 'CMakelists.txt' to accommodate the new program. At a high level, Scalar is built so that: - there is a 'scalar-objs' target (similar to those created in 029bac01a8 (Makefile: add {program,xdiff,test,git,fuzz}-objs & objects targets, 2021-02-23)) for debugging purposes. - it appears in the root of the install directory (rather than the gitexecdir). - it is included in the 'bin-wrappers/' directory for use in tests. - it receives a platform-specific executable suffix (e.g., '.exe'), if applicable. - 'scalar.txt' is installed as 'man1' documentation. - the 'clean' target removes the 'scalar' executable. Additionally, update the root level '.gitignore' file to ignore the Scalar executable. Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- .gitignore | 1 + Documentation/Makefile | 1 + Documentation/scalar.txt | 166 +++++++ Makefile | 31 +- contrib/buildsystems/CMakeLists.txt | 9 +- contrib/scalar/.gitignore | 2 - contrib/scalar/Makefile | 35 -- contrib/scalar/scalar.c | 906 ------------------------------------ contrib/scalar/scalar.txt | 166 ------- contrib/scalar/t/Makefile | 81 ---- contrib/scalar/t/t9099-scalar.sh | 216 --------- scalar.c | 906 ++++++++++++++++++++++++++++++++++++ t/t9210-scalar.sh | 210 +++++++++ 13 files changed, 1308 insertions(+), 1422 deletions(-) create mode 100644 Documentation/scalar.txt delete mode 100644 contrib/scalar/.gitignore delete mode 100644 contrib/scalar/Makefile delete mode 100644 contrib/scalar/scalar.c delete mode 100644 contrib/scalar/scalar.txt delete mode 100644 contrib/scalar/t/Makefile delete mode 100755 contrib/scalar/t/t9099-scalar.sh create mode 100644 scalar.c create mode 100755 t/t9210-scalar.sh diff --git a/.gitignore b/.gitignore index 80b530bbed..3d1b880101 100644 --- a/.gitignore +++ b/.gitignore @@ -185,6 +185,7 @@ /git-whatchanged /git-worktree /git-write-tree +/scalar /git-core-*/?* /git.res /gitweb/GITWEB-BUILD-OPTIONS diff --git a/Documentation/Makefile b/Documentation/Makefile index bd6b6fcb93..16c9e06239 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -21,6 +21,7 @@ MAN1_TXT += $(filter-out \ MAN1_TXT += git.txt MAN1_TXT += gitk.txt MAN1_TXT += gitweb.txt +MAN1_TXT += scalar.txt # man5 / man7 guides (note: new guides should also be added to command-list.txt) MAN5_TXT += gitattributes.txt diff --git a/Documentation/scalar.txt b/Documentation/scalar.txt new file mode 100644 index 0000000000..f33436c7f6 --- /dev/null +++ b/Documentation/scalar.txt @@ -0,0 +1,166 @@ +scalar(1) +========= + +NAME +---- +scalar - A tool for managing large Git repositories + +SYNOPSIS +-------- +[verse] +scalar clone [--single-branch] [--branch ] [--full-clone] [] +scalar list +scalar register [] +scalar unregister [] +scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [] +scalar reconfigure [ --all | ] +scalar diagnose [] +scalar delete + +DESCRIPTION +----------- + +Scalar is a repository management tool that optimizes Git for use in large +repositories. Scalar improves performance by configuring advanced Git settings, +maintaining repositories in the background, and helping to reduce data sent +across the network. + +An important Scalar concept is the enlistment: this is the top-level directory +of the project. It usually contains the subdirectory `src/` which is a Git +worktree. This encourages the separation between tracked files (inside `src/`) +and untracked files, such as build artifacts (outside `src/`). When registering +an existing Git worktree with Scalar whose name is not `src`, the enlistment +will be identical to the worktree. + +The `scalar` command implements various subcommands, and different options +depending on the subcommand. With the exception of `clone`, `list` and +`reconfigure --all`, all subcommands expect to be run in an enlistment. + +The following options can be specified _before_ the subcommand: + +-C :: + Before running the subcommand, change the working directory. This + option imitates the same option of linkgit:git[1]. + +-c =:: + For the duration of running the specified subcommand, configure this + setting. This option imitates the same option of linkgit:git[1]. + +COMMANDS +-------- + +Clone +~~~~~ + +clone [] []:: + Clones the specified repository, similar to linkgit:git-clone[1]. By + default, only commit and tree objects are cloned. Once finished, the + worktree is located at `/src`. ++ +The sparse-checkout feature is enabled (except when run with `--full-clone`) +and the only files present are those in the top-level directory. Use +`git sparse-checkout set` to expand the set of directories you want to see, +or `git sparse-checkout disable` to expand to all files (see +linkgit:git-sparse-checkout[1] for more details). You can explore the +subdirectories outside your sparse-checkout by using `git ls-tree +HEAD[:]`. + +-b :: +--branch :: + Instead of checking out the branch pointed to by the cloned + repository's HEAD, check out the `` branch instead. + +--[no-]single-branch:: + Clone only the history leading to the tip of a single branch, either + specified by the `--branch` option or the primary branch remote's + `HEAD` points at. ++ +Further fetches into the resulting repository will only update the +remote-tracking branch for the branch this option was used for the initial +cloning. If the HEAD at the remote did not point at any branch when +`--single-branch` clone was made, no remote-tracking branch is created. + +--[no-]full-clone:: + A sparse-checkout is initialized by default. This behavior can be + turned off via `--full-clone`. + +List +~~~~ + +list:: + List enlistments that are currently registered by Scalar. This + subcommand does not need to be run inside an enlistment. + +Register +~~~~~~~~ + +register []:: + Adds the enlistment's repository to the list of registered repositories + and starts background maintenance. If `` is not provided, + then the enlistment associated with the current working directory is + registered. ++ +Note: when this subcommand is called in a worktree that is called `src/`, its +parent directory is considered to be the Scalar enlistment. If the worktree is +_not_ called `src/`, it itself will be considered to be the Scalar enlistment. + +Unregister +~~~~~~~~~~ + +unregister []:: + Remove the specified repository from the list of repositories + registered with Scalar and stop the scheduled background maintenance. + +Run +~~~ + +scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) []:: + Run the given maintenance task (or all tasks, if `all` was specified). + Except for `all` and `config`, this subcommand simply hands off to + linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and + `pack-files` to `incremental-repack`). ++ +These tasks are run automatically as part of the scheduled maintenance, +as soon as the repository is registered with Scalar. It should therefore +not be necessary to run this subcommand manually. ++ +The `config` task is specific to Scalar and configures all those +opinionated default settings that make Git work more efficiently with +large repositories. As this task is run as part of `scalar clone` +automatically, explicit invocations of this task are rarely needed. + +Reconfigure +~~~~~~~~~~~ + +After a Scalar upgrade, or when the configuration of a Scalar enlistment +was somehow corrupted or changed by mistake, this subcommand allows to +reconfigure the enlistment. + +With the `--all` option, all enlistments currently registered with Scalar +will be reconfigured. Use this option after each Scalar upgrade. + +Diagnose +~~~~~~~~ + +diagnose []:: + When reporting issues with Scalar, it is often helpful to provide the + information gathered by this command, including logs and certain + statistics describing the data shape of the current enlistment. ++ +The output of this command is a `.zip` file that is written into +a directory adjacent to the worktree in the `src` directory. + +Delete +~~~~~~ + +delete :: + This subcommand lets you delete an existing Scalar enlistment from your + local file system, unregistering the repository. + +SEE ALSO +-------- +linkgit:git-clone[1], linkgit:git-maintenance[1]. + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/Makefile b/Makefile index eac30126e2..e03f32ec1e 100644 --- a/Makefile +++ b/Makefile @@ -608,7 +608,9 @@ FUZZ_OBJS = FUZZ_PROGRAMS = GIT_OBJS = LIB_OBJS = +SCALAR_OBJS = OBJECTS = +OTHER_PROGRAMS = PROGRAM_OBJS = PROGRAMS = EXCLUDED_PROGRAMS = @@ -821,10 +823,12 @@ BUILT_INS += git-switch$X BUILT_INS += git-whatchanged$X # what 'all' will build but not install in gitexecdir -OTHER_PROGRAMS = git$X +OTHER_PROGRAMS += git$X +OTHER_PROGRAMS += scalar$X # what test wrappers are needed and 'install' will install, in bindir BINDIR_PROGRAMS_NEED_X += git +BINDIR_PROGRAMS_NEED_X += scalar BINDIR_PROGRAMS_NEED_X += git-receive-pack BINDIR_PROGRAMS_NEED_X += git-shell BINDIR_PROGRAMS_NEED_X += git-upload-archive @@ -2222,7 +2226,7 @@ profile-fast: profile-clean all:: $(ALL_COMMANDS_TO_INSTALL) $(SCRIPT_LIB) $(OTHER_PROGRAMS) GIT-BUILD-OPTIONS ifneq (,$X) - $(QUIET_BUILT_IN)$(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_COMMANDS_TO_INSTALL) git$X)), test -d '$p' -o '$p' -ef '$p$X' || $(RM) '$p';) + $(QUIET_BUILT_IN)$(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_COMMANDS_TO_INSTALL) $(OTHER_PROGRAMS))), test -d '$p' -o '$p' -ef '$p$X' || $(RM) '$p';) endif all:: @@ -2545,7 +2549,12 @@ GIT_OBJS += git.o .PHONY: git-objs git-objs: $(GIT_OBJS) +SCALAR_OBJS += scalar.o +.PHONY: scalar-objs +scalar-objs: $(SCALAR_OBJS) + OBJECTS += $(GIT_OBJS) +OBJECTS += $(SCALAR_OBJS) OBJECTS += $(PROGRAM_OBJS) OBJECTS += $(TEST_OBJS) OBJECTS += $(XDIFF_OBJS) @@ -2556,10 +2565,6 @@ ifndef NO_CURL OBJECTS += http.o http-walker.o remote-curl.o endif -SCALAR_SOURCES := contrib/scalar/scalar.c -SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o) -OBJECTS += $(SCALAR_OBJECTS) - .PHONY: objects objects: $(OBJECTS) @@ -2691,7 +2696,7 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \ $(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS) -contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS) +scalar$X: scalar.o GIT-LDFLAGS $(GITLIBS) $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \ $(filter %.o,$^) $(LIBS) @@ -2747,8 +2752,7 @@ XGETTEXT_FLAGS_SH = $(XGETTEXT_FLAGS) --language=Shell \ XGETTEXT_FLAGS_PERL = $(XGETTEXT_FLAGS) --language=Perl \ --keyword=__ --keyword=N__ --keyword="__n:1,2" MSGMERGE_FLAGS = --add-location --backup=off --update -LOCALIZED_C = $(sort $(FOUND_C_SOURCES) $(FOUND_H_SOURCES) $(SCALAR_SOURCES) \ - $(GENERATED_H)) +LOCALIZED_C = $(sort $(FOUND_C_SOURCES) $(FOUND_H_SOURCES) $(GENERATED_H)) LOCALIZED_SH = $(sort $(SCRIPT_SH) git-sh-setup.sh) LOCALIZED_PERL = $(sort $(SCRIPT_PERL)) @@ -3062,7 +3066,7 @@ bin-wrappers/%: wrap-for-bin.sh $(call mkdir_p_parent_template) $(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \ -e 's|@@BUILD_DIR@@|$(shell pwd)|' \ - -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \ + -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%,$(@F))$(if $(filter-out $(BINDIR_PROGRAMS_NO_X),$(@F)),$(X),)|' < $< > $@ && \ chmod +x $@ # GNU make supports exporting all variables by "export" without parameters. @@ -3276,14 +3280,14 @@ ifndef NO_TCLTK $(MAKE) -C git-gui gitexecdir='$(gitexec_instdir_SQ)' install endif ifneq (,$X) - $(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_COMMANDS_TO_INSTALL) git$X)), test '$(DESTDIR_SQ)$(gitexec_instdir_SQ)/$p' -ef '$(DESTDIR_SQ)$(gitexec_instdir_SQ)/$p$X' || $(RM) '$(DESTDIR_SQ)$(gitexec_instdir_SQ)/$p';) + $(foreach p,$(patsubst %$X,%,$(filter %$X,$(ALL_COMMANDS_TO_INSTALL) $(OTHER_PROGRAMS))), test '$(DESTDIR_SQ)$(gitexec_instdir_SQ)/$p' -ef '$(DESTDIR_SQ)$(gitexec_instdir_SQ)/$p$X' || $(RM) '$(DESTDIR_SQ)$(gitexec_instdir_SQ)/$p';) endif bindir=$$(cd '$(DESTDIR_SQ)$(bindir_SQ)' && pwd) && \ execdir=$$(cd '$(DESTDIR_SQ)$(gitexec_instdir_SQ)' && pwd) && \ destdir_from_execdir_SQ=$$(echo '$(gitexecdir_relative_SQ)' | sed -e 's|[^/][^/]*|..|g') && \ { test "$$bindir/" = "$$execdir/" || \ - for p in git$X $(filter $(install_bindir_programs),$(ALL_PROGRAMS)); do \ + for p in $(OTHER_PROGRAMS) $(filter $(install_bindir_programs),$(ALL_PROGRAMS)); do \ $(RM) "$$execdir/$$p" && \ test -n "$(INSTALL_SYMLINKS)" && \ ln -s "$$destdir_from_execdir_SQ/$(bindir_relative_SQ)/$$p" "$$execdir/$$p" || \ @@ -3458,7 +3462,7 @@ clean: profile-clean coverage-clean cocciclean $(RM) git.res $(RM) $(OBJECTS) $(RM) $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(REFTABLE_TEST_LIB) - $(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X + $(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) $(RM) $(TEST_PROGRAMS) $(RM) $(FUZZ_PROGRAMS) $(RM) $(SP_OBJ) @@ -3509,6 +3513,7 @@ ALL_COMMANDS += git-citool ALL_COMMANDS += git-gui ALL_COMMANDS += gitk ALL_COMMANDS += gitweb +ALL_COMMANDS += scalar .PHONY: check-docs check-docs:: diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 2237109b57..bae203c1fb 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -610,7 +610,7 @@ unset(CMAKE_REQUIRED_INCLUDES) #programs set(PROGRAMS_BUILT git git-daemon git-http-backend git-sh-i18n--envsubst - git-shell) + git-shell scalar) if(NOT CURL_FOUND) list(APPEND excluded_progs git-http-fetch git-http-push) @@ -757,6 +757,9 @@ target_link_libraries(git-sh-i18n--envsubst common-main) add_executable(git-shell ${CMAKE_SOURCE_DIR}/shell.c) target_link_libraries(git-shell common-main) +add_executable(scalar ${CMAKE_SOURCE_DIR}/scalar.c) +target_link_libraries(scalar common-main) + if(CURL_FOUND) add_library(http_obj OBJECT ${CMAKE_SOURCE_DIR}/http.c) @@ -903,7 +906,7 @@ list(TRANSFORM git_perl_scripts PREPEND "${CMAKE_BINARY_DIR}/") #install foreach(program ${PROGRAMS_BUILT}) -if(program STREQUAL "git" OR program STREQUAL "git-shell") +if(program MATCHES "^(git|git-shell|scalar)$") install(TARGETS ${program} RUNTIME DESTINATION bin) else() @@ -977,7 +980,7 @@ endif() #wrapper scripts set(wrapper_scripts - git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext) + git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar) set(wrapper_test_scripts test-fake-ssh test-tool) diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore deleted file mode 100644 index ff3d47e84d..0000000000 --- a/contrib/scalar/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -/*.exe -/scalar diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile deleted file mode 100644 index 37f283f35d..0000000000 --- a/contrib/scalar/Makefile +++ /dev/null @@ -1,35 +0,0 @@ -# The default target of this Makefile is... -all:: - -# Import tree-wide shared Makefile behavior and libraries -include ../../shared.mak - -include ../../config.mak.uname --include ../../config.mak.autogen --include ../../config.mak - -TARGETS = scalar$(X) scalar.o -GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a - -all:: scalar$(X) ../../bin-wrappers/scalar - -$(GITLIBS): - $(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@) - -$(TARGETS): $(GITLIBS) scalar.c - $(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@) - -clean: - $(RM) $(TARGETS) ../../bin-wrappers/scalar - -../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile - @mkdir -p ../../bin-wrappers - $(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \ - -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \ - -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \ - chmod +x $@ - -test: all - $(MAKE) -C t - -.PHONY: $(GITLIBS) all clean test FORCE diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c deleted file mode 100644 index 642d16124e..0000000000 --- a/contrib/scalar/scalar.c +++ /dev/null @@ -1,906 +0,0 @@ -/* - * The Scalar command-line interface. - */ - -#include "cache.h" -#include "gettext.h" -#include "parse-options.h" -#include "config.h" -#include "run-command.h" -#include "simple-ipc.h" -#include "fsmonitor-ipc.h" -#include "fsmonitor-settings.h" -#include "refs.h" -#include "dir.h" -#include "packfile.h" -#include "help.h" - -static void setup_enlistment_directory(int argc, const char **argv, - const char * const *usagestr, - const struct option *options, - struct strbuf *enlistment_root) -{ - struct strbuf path = STRBUF_INIT; - int enlistment_is_repo_parent = 0; - size_t len; - - if (startup_info->have_repository) - BUG("gitdir already set up?!?"); - - if (argc > 1) - usage_with_options(usagestr, options); - - /* find the worktree, determine its corresponding root */ - if (argc == 1) { - strbuf_add_absolute_path(&path, argv[0]); - if (!is_directory(path.buf)) - die(_("'%s' does not exist"), path.buf); - if (chdir(path.buf) < 0) - die_errno(_("could not switch to '%s'"), path.buf); - } else if (strbuf_getcwd(&path) < 0) - die(_("need a working directory")); - - strbuf_trim_trailing_dir_sep(&path); - - /* check if currently in enlistment root with src/ workdir */ - len = path.len; - strbuf_addstr(&path, "/src"); - if (is_nonbare_repository_dir(&path)) { - enlistment_is_repo_parent = 1; - if (chdir(path.buf) < 0) - die_errno(_("could not switch to '%s'"), path.buf); - } - strbuf_setlen(&path, len); - - setup_git_directory(); - - if (!the_repository->worktree) - die(_("Scalar enlistments require a worktree")); - - if (enlistment_root) { - if (enlistment_is_repo_parent) - strbuf_addbuf(enlistment_root, &path); - else - strbuf_addstr(enlistment_root, the_repository->worktree); - } - - strbuf_release(&path); -} - -static int run_git(const char *arg, ...) -{ - struct strvec argv = STRVEC_INIT; - va_list args; - const char *p; - int res; - - va_start(args, arg); - strvec_push(&argv, arg); - while ((p = va_arg(args, const char *))) - strvec_push(&argv, p); - va_end(args); - - res = run_command_v_opt(argv.v, RUN_GIT_CMD); - - strvec_clear(&argv); - return res; -} - -struct scalar_config { - const char *key; - const char *value; - int overwrite_on_reconfigure; -}; - -static int set_scalar_config(const struct scalar_config *config, int reconfigure) -{ - char *value = NULL; - int res; - - if ((reconfigure && config->overwrite_on_reconfigure) || - git_config_get_string(config->key, &value)) { - trace2_data_string("scalar", the_repository, config->key, "created"); - res = git_config_set_gently(config->key, config->value); - } else { - trace2_data_string("scalar", the_repository, config->key, "exists"); - res = 0; - } - - free(value); - return res; -} - -static int have_fsmonitor_support(void) -{ - return fsmonitor_ipc__is_supported() && - fsm_settings__get_reason(the_repository) == FSMONITOR_REASON_OK; -} - -static int set_recommended_config(int reconfigure) -{ - struct scalar_config config[] = { - /* Required */ - { "am.keepCR", "true", 1 }, - { "core.FSCache", "true", 1 }, - { "core.multiPackIndex", "true", 1 }, - { "core.preloadIndex", "true", 1 }, -#ifndef WIN32 - { "core.untrackedCache", "true", 1 }, -#else - /* - * Unfortunately, Scalar's Functional Tests demonstrated - * that the untracked cache feature is unreliable on Windows - * (which is a bummer because that platform would benefit the - * most from it). For some reason, freshly created files seem - * not to update the directory's `lastModified` time - * immediately, but the untracked cache would need to rely on - * that. - * - * Therefore, with a sad heart, we disable this very useful - * feature on Windows. - */ - { "core.untrackedCache", "false", 1 }, -#endif - { "core.logAllRefUpdates", "true", 1 }, - { "credential.https://dev.azure.com.useHttpPath", "true", 1 }, - { "credential.validate", "false", 1 }, /* GCM4W-only */ - { "gc.auto", "0", 1 }, - { "gui.GCWarning", "false", 1 }, - { "index.threads", "true", 1 }, - { "index.version", "4", 1 }, - { "merge.stat", "false", 1 }, - { "merge.renames", "true", 1 }, - { "pack.useBitmaps", "false", 1 }, - { "pack.useSparse", "true", 1 }, - { "receive.autoGC", "false", 1 }, - { "feature.manyFiles", "false", 1 }, - { "feature.experimental", "false", 1 }, - { "fetch.unpackLimit", "1", 1 }, - { "fetch.writeCommitGraph", "false", 1 }, -#ifdef WIN32 - { "http.sslBackend", "schannel", 1 }, -#endif - /* Optional */ - { "status.aheadBehind", "false" }, - { "commitGraph.generationVersion", "1" }, - { "core.autoCRLF", "false" }, - { "core.safeCRLF", "false" }, - { "fetch.showForcedUpdates", "false" }, - { NULL, NULL }, - }; - int i; - char *value; - - for (i = 0; config[i].key; i++) { - if (set_scalar_config(config + i, reconfigure)) - return error(_("could not configure %s=%s"), - config[i].key, config[i].value); - } - - if (have_fsmonitor_support()) { - struct scalar_config fsmonitor = { "core.fsmonitor", "true" }; - if (set_scalar_config(&fsmonitor, reconfigure)) - return error(_("could not configure %s=%s"), - fsmonitor.key, fsmonitor.value); - } - - /* - * The `log.excludeDecoration` setting is special because it allows - * for multiple values. - */ - if (git_config_get_string("log.excludeDecoration", &value)) { - trace2_data_string("scalar", the_repository, - "log.excludeDecoration", "created"); - if (git_config_set_multivar_gently("log.excludeDecoration", - "refs/prefetch/*", - CONFIG_REGEX_NONE, 0)) - return error(_("could not configure " - "log.excludeDecoration")); - } else { - trace2_data_string("scalar", the_repository, - "log.excludeDecoration", "exists"); - free(value); - } - - return 0; -} - -static int toggle_maintenance(int enable) -{ - return run_git("maintenance", enable ? "start" : "unregister", NULL); -} - -static int add_or_remove_enlistment(int add) -{ - int res; - - if (!the_repository->worktree) - die(_("Scalar enlistments require a worktree")); - - res = run_git("config", "--global", "--get", "--fixed-value", - "scalar.repo", the_repository->worktree, NULL); - - /* - * If we want to add and the setting is already there, then do nothing. - * If we want to remove and the setting is not there, then do nothing. - */ - if ((add && !res) || (!add && res)) - return 0; - - return run_git("config", "--global", add ? "--add" : "--unset", - add ? "--no-fixed-value" : "--fixed-value", - "scalar.repo", the_repository->worktree, NULL); -} - -static int start_fsmonitor_daemon(void) -{ - assert(have_fsmonitor_support()); - - if (fsmonitor_ipc__get_state() != IPC_STATE__LISTENING) - return run_git("fsmonitor--daemon", "start", NULL); - - return 0; -} - -static int stop_fsmonitor_daemon(void) -{ - assert(have_fsmonitor_support()); - - if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) - return run_git("fsmonitor--daemon", "stop", NULL); - - return 0; -} - -static int register_dir(void) -{ - if (add_or_remove_enlistment(1)) - return error(_("could not add enlistment")); - - if (set_recommended_config(0)) - return error(_("could not set recommended config")); - - if (toggle_maintenance(1)) - return error(_("could not turn on maintenance")); - - if (have_fsmonitor_support() && start_fsmonitor_daemon()) { - return error(_("could not start the FSMonitor daemon")); - } - - return 0; -} - -static int unregister_dir(void) -{ - int res = 0; - - if (toggle_maintenance(0)) - res = error(_("could not turn off maintenance")); - - if (add_or_remove_enlistment(0)) - res = error(_("could not remove enlistment")); - - return res; -} - -/* printf-style interface, expects `=` argument */ -static int set_config(const char *fmt, ...) -{ - struct strbuf buf = STRBUF_INIT; - char *value; - int res; - va_list args; - - va_start(args, fmt); - strbuf_vaddf(&buf, fmt, args); - va_end(args); - - value = strchr(buf.buf, '='); - if (value) - *(value++) = '\0'; - res = git_config_set_gently(buf.buf, value); - strbuf_release(&buf); - - return res; -} - -static char *remote_default_branch(const char *url) -{ - struct child_process cp = CHILD_PROCESS_INIT; - struct strbuf out = STRBUF_INIT; - - cp.git_cmd = 1; - strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL); - if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) { - const char *line = out.buf; - - while (*line) { - const char *eol = strchrnul(line, '\n'), *p; - size_t len = eol - line; - char *branch; - - if (!skip_prefix(line, "ref: ", &p) || - !strip_suffix_mem(line, &len, "\tHEAD")) { - line = eol + (*eol == '\n'); - continue; - } - - eol = line + len; - if (skip_prefix(p, "refs/heads/", &p)) { - branch = xstrndup(p, eol - p); - strbuf_release(&out); - return branch; - } - - error(_("remote HEAD is not a branch: '%.*s'"), - (int)(eol - p), p); - strbuf_release(&out); - return NULL; - } - } - warning(_("failed to get default branch name from remote; " - "using local default")); - strbuf_reset(&out); - - child_process_init(&cp); - cp.git_cmd = 1; - strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL); - if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) { - strbuf_trim(&out); - return strbuf_detach(&out, NULL); - } - - strbuf_release(&out); - error(_("failed to get default branch name")); - return NULL; -} - -static int delete_enlistment(struct strbuf *enlistment) -{ -#ifdef WIN32 - struct strbuf parent = STRBUF_INIT; - size_t offset; - char *path_sep; -#endif - - if (unregister_dir()) - return error(_("failed to unregister repository")); - -#ifdef WIN32 - /* - * Change the current directory to one outside of the enlistment so - * that we may delete everything underneath it. - */ - offset = offset_1st_component(enlistment->buf); - path_sep = find_last_dir_sep(enlistment->buf + offset); - strbuf_add(&parent, enlistment->buf, - path_sep ? path_sep - enlistment->buf : offset); - if (chdir(parent.buf) < 0) { - int res = error_errno(_("could not switch to '%s'"), parent.buf); - strbuf_release(&parent); - return res; - } - strbuf_release(&parent); -#endif - - if (have_fsmonitor_support() && stop_fsmonitor_daemon()) - return error(_("failed to stop the FSMonitor daemon")); - - if (remove_dir_recursively(enlistment, 0)) - return error(_("failed to delete enlistment directory")); - - return 0; -} - -/* - * Dummy implementation; Using `get_version_info()` would cause a link error - * without this. - */ -void load_builtin_commands(const char *prefix, struct cmdnames *cmds) -{ - die("not implemented"); -} - -static int cmd_clone(int argc, const char **argv) -{ - const char *branch = NULL; - int full_clone = 0, single_branch = 0; - struct option clone_options[] = { - OPT_STRING('b', "branch", &branch, N_(""), - N_("branch to checkout after clone")), - OPT_BOOL(0, "full-clone", &full_clone, - N_("when cloning, create full working directory")), - OPT_BOOL(0, "single-branch", &single_branch, - N_("only download metadata for the branch that will " - "be checked out")), - OPT_END(), - }; - const char * const clone_usage[] = { - N_("scalar clone [] [--] []"), - NULL - }; - const char *url; - char *enlistment = NULL, *dir = NULL; - struct strbuf buf = STRBUF_INIT; - int res; - - argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0); - - if (argc == 2) { - url = argv[0]; - enlistment = xstrdup(argv[1]); - } else if (argc == 1) { - url = argv[0]; - - strbuf_addstr(&buf, url); - /* Strip trailing slashes, if any */ - while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1])) - strbuf_setlen(&buf, buf.len - 1); - /* Strip suffix `.git`, if any */ - strbuf_strip_suffix(&buf, ".git"); - - enlistment = find_last_dir_sep(buf.buf); - if (!enlistment) { - die(_("cannot deduce worktree name from '%s'"), url); - } - enlistment = xstrdup(enlistment + 1); - } else { - usage_msg_opt(_("You must specify a repository to clone."), - clone_usage, clone_options); - } - - if (is_directory(enlistment)) - die(_("directory '%s' exists already"), enlistment); - - dir = xstrfmt("%s/src", enlistment); - - strbuf_reset(&buf); - if (branch) - strbuf_addf(&buf, "init.defaultBranch=%s", branch); - else { - char *b = repo_default_branch_name(the_repository, 1); - strbuf_addf(&buf, "init.defaultBranch=%s", b); - free(b); - } - - if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL))) - goto cleanup; - - if (chdir(dir) < 0) { - res = error_errno(_("could not switch to '%s'"), dir); - goto cleanup; - } - - setup_git_directory(); - - /* common-main already logs `argv` */ - trace2_def_repo(the_repository); - - if (!branch && !(branch = remote_default_branch(url))) { - res = error(_("failed to get default branch for '%s'"), url); - goto cleanup; - } - - if (set_config("remote.origin.url=%s", url) || - set_config("remote.origin.fetch=" - "+refs/heads/%s:refs/remotes/origin/%s", - single_branch ? branch : "*", - single_branch ? branch : "*") || - set_config("remote.origin.promisor=true") || - set_config("remote.origin.partialCloneFilter=blob:none")) { - res = error(_("could not configure remote in '%s'"), dir); - goto cleanup; - } - - if (!full_clone && - (res = run_git("sparse-checkout", "init", "--cone", NULL))) - goto cleanup; - - if (set_recommended_config(0)) - return error(_("could not configure '%s'"), dir); - - if ((res = run_git("fetch", "--quiet", "origin", NULL))) { - warning(_("partial clone failed; attempting full clone")); - - if (set_config("remote.origin.promisor") || - set_config("remote.origin.partialCloneFilter")) { - res = error(_("could not configure for full clone")); - goto cleanup; - } - - if ((res = run_git("fetch", "--quiet", "origin", NULL))) - goto cleanup; - } - - if ((res = set_config("branch.%s.remote=origin", branch))) - goto cleanup; - if ((res = set_config("branch.%s.merge=refs/heads/%s", - branch, branch))) - goto cleanup; - - strbuf_reset(&buf); - strbuf_addf(&buf, "origin/%s", branch); - res = run_git("checkout", "-f", "-t", buf.buf, NULL); - if (res) - goto cleanup; - - res = register_dir(); - -cleanup: - free(enlistment); - free(dir); - strbuf_release(&buf); - return res; -} - -static int cmd_diagnose(int argc, const char **argv) -{ - struct option options[] = { - OPT_END(), - }; - const char * const usage[] = { - N_("scalar diagnose []"), - NULL - }; - struct strbuf diagnostics_root = STRBUF_INIT; - int res = 0; - - argc = parse_options(argc, argv, NULL, options, - usage, 0); - - setup_enlistment_directory(argc, argv, usage, options, &diagnostics_root); - strbuf_addstr(&diagnostics_root, "/.scalarDiagnostics"); - - res = run_git("diagnose", "--mode=all", "-s", "%Y%m%d_%H%M%S", - "-o", diagnostics_root.buf, NULL); - - strbuf_release(&diagnostics_root); - return res; -} - -static int cmd_list(int argc, const char **argv) -{ - if (argc != 1) - die(_("`scalar list` does not take arguments")); - - if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0) - return -1; - return 0; -} - -static int cmd_register(int argc, const char **argv) -{ - struct option options[] = { - OPT_END(), - }; - const char * const usage[] = { - N_("scalar register []"), - NULL - }; - - argc = parse_options(argc, argv, NULL, options, - usage, 0); - - setup_enlistment_directory(argc, argv, usage, options, NULL); - - return register_dir(); -} - -static int get_scalar_repos(const char *key, const char *value, void *data) -{ - struct string_list *list = data; - - if (!strcmp(key, "scalar.repo")) - string_list_append(list, value); - - return 0; -} - -static int cmd_reconfigure(int argc, const char **argv) -{ - int all = 0; - struct option options[] = { - OPT_BOOL('a', "all", &all, - N_("reconfigure all registered enlistments")), - OPT_END(), - }; - const char * const usage[] = { - N_("scalar reconfigure [--all | ]"), - NULL - }; - struct string_list scalar_repos = STRING_LIST_INIT_DUP; - int i, res = 0; - struct repository r = { NULL }; - struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT; - - argc = parse_options(argc, argv, NULL, options, - usage, 0); - - if (!all) { - setup_enlistment_directory(argc, argv, usage, options, NULL); - - return set_recommended_config(1); - } - - if (argc > 0) - usage_msg_opt(_("--all or , but not both"), - usage, options); - - git_config(get_scalar_repos, &scalar_repos); - - for (i = 0; i < scalar_repos.nr; i++) { - const char *dir = scalar_repos.items[i].string; - - strbuf_reset(&commondir); - strbuf_reset(&gitdir); - - if (chdir(dir) < 0) { - warning_errno(_("could not switch to '%s'"), dir); - res = -1; - } else if (discover_git_directory(&commondir, &gitdir) < 0) { - warning_errno(_("git repository gone in '%s'"), dir); - res = -1; - } else { - git_config_clear(); - - the_repository = &r; - r.commondir = commondir.buf; - r.gitdir = gitdir.buf; - - if (set_recommended_config(1) < 0) - res = -1; - } - } - - string_list_clear(&scalar_repos, 1); - strbuf_release(&commondir); - strbuf_release(&gitdir); - - return res; -} - -static int cmd_run(int argc, const char **argv) -{ - struct option options[] = { - OPT_END(), - }; - struct { - const char *arg, *task; - } tasks[] = { - { "config", NULL }, - { "commit-graph", "commit-graph" }, - { "fetch", "prefetch" }, - { "loose-objects", "loose-objects" }, - { "pack-files", "incremental-repack" }, - { NULL, NULL } - }; - struct strbuf buf = STRBUF_INIT; - const char *usagestr[] = { NULL, NULL }; - int i; - - strbuf_addstr(&buf, N_("scalar run []\nTasks:\n")); - for (i = 0; tasks[i].arg; i++) - strbuf_addf(&buf, "\t%s\n", tasks[i].arg); - usagestr[0] = buf.buf; - - argc = parse_options(argc, argv, NULL, options, - usagestr, 0); - - if (!argc) - usage_with_options(usagestr, options); - - if (!strcmp("all", argv[0])) { - i = -1; - } else { - for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++) - ; /* keep looking for the task */ - - if (i > 0 && !tasks[i].arg) { - error(_("no such task: '%s'"), argv[0]); - usage_with_options(usagestr, options); - } - } - - argc--; - argv++; - setup_enlistment_directory(argc, argv, usagestr, options, NULL); - strbuf_release(&buf); - - if (i == 0) - return register_dir(); - - if (i > 0) - return run_git("maintenance", "run", - "--task", tasks[i].task, NULL); - - if (register_dir()) - return -1; - for (i = 1; tasks[i].arg; i++) - if (run_git("maintenance", "run", - "--task", tasks[i].task, NULL)) - return -1; - return 0; -} - -static int remove_deleted_enlistment(struct strbuf *path) -{ - int res = 0; - strbuf_realpath_forgiving(path, path->buf, 1); - - if (run_git("config", "--global", - "--unset", "--fixed-value", - "scalar.repo", path->buf, NULL) < 0) - res = -1; - - if (run_git("config", "--global", - "--unset", "--fixed-value", - "maintenance.repo", path->buf, NULL) < 0) - res = -1; - - return res; -} - -static int cmd_unregister(int argc, const char **argv) -{ - struct option options[] = { - OPT_END(), - }; - const char * const usage[] = { - N_("scalar unregister []"), - NULL - }; - - argc = parse_options(argc, argv, NULL, options, - usage, 0); - - /* - * Be forgiving when the enlistment or worktree does not even exist any - * longer; This can be the case if a user deleted the worktree by - * mistake and _still_ wants to unregister the thing. - */ - if (argc == 1) { - struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT; - - strbuf_addf(&src_path, "%s/src/.git", argv[0]); - strbuf_addf(&workdir_path, "%s/.git", argv[0]); - if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) { - /* remove possible matching registrations */ - int res = -1; - - strbuf_strip_suffix(&src_path, "/.git"); - res = remove_deleted_enlistment(&src_path) && res; - - strbuf_strip_suffix(&workdir_path, "/.git"); - res = remove_deleted_enlistment(&workdir_path) && res; - - strbuf_release(&src_path); - strbuf_release(&workdir_path); - return res; - } - strbuf_release(&src_path); - strbuf_release(&workdir_path); - } - - setup_enlistment_directory(argc, argv, usage, options, NULL); - - return unregister_dir(); -} - -static int cmd_delete(int argc, const char **argv) -{ - char *cwd = xgetcwd(); - struct option options[] = { - OPT_END(), - }; - const char * const usage[] = { - N_("scalar delete "), - NULL - }; - struct strbuf enlistment = STRBUF_INIT; - int res = 0; - - argc = parse_options(argc, argv, NULL, options, - usage, 0); - - if (argc != 1) - usage_with_options(usage, options); - - setup_enlistment_directory(argc, argv, usage, options, &enlistment); - - if (dir_inside_of(cwd, enlistment.buf) >= 0) - res = error(_("refusing to delete current working directory")); - else { - close_object_store(the_repository->objects); - res = delete_enlistment(&enlistment); - } - strbuf_release(&enlistment); - free(cwd); - - return res; -} - -static int cmd_version(int argc, const char **argv) -{ - int verbose = 0, build_options = 0; - struct option options[] = { - OPT__VERBOSE(&verbose, N_("include Git version")), - OPT_BOOL(0, "build-options", &build_options, - N_("include Git's build options")), - OPT_END(), - }; - const char * const usage[] = { - N_("scalar verbose [-v | --verbose] [--build-options]"), - NULL - }; - struct strbuf buf = STRBUF_INIT; - - argc = parse_options(argc, argv, NULL, options, - usage, 0); - - if (argc != 0) - usage_with_options(usage, options); - - get_version_info(&buf, build_options); - fprintf(stderr, "%s\n", buf.buf); - strbuf_release(&buf); - - return 0; -} - -static struct { - const char *name; - int (*fn)(int, const char **); -} builtins[] = { - { "clone", cmd_clone }, - { "list", cmd_list }, - { "register", cmd_register }, - { "unregister", cmd_unregister }, - { "run", cmd_run }, - { "reconfigure", cmd_reconfigure }, - { "delete", cmd_delete }, - { "version", cmd_version }, - { "diagnose", cmd_diagnose }, - { NULL, NULL}, -}; - -int cmd_main(int argc, const char **argv) -{ - struct strbuf scalar_usage = STRBUF_INIT; - int i; - - while (argc > 1 && *argv[1] == '-') { - if (!strcmp(argv[1], "-C")) { - if (argc < 3) - die(_("-C requires a ")); - if (chdir(argv[2]) < 0) - die_errno(_("could not change to '%s'"), - argv[2]); - argc -= 2; - argv += 2; - } else if (!strcmp(argv[1], "-c")) { - if (argc < 3) - die(_("-c requires a = argument")); - git_config_push_parameter(argv[2]); - argc -= 2; - argv += 2; - } else - break; - } - - if (argc > 1) { - argv++; - argc--; - - for (i = 0; builtins[i].name; i++) - if (!strcmp(builtins[i].name, argv[0])) - return !!builtins[i].fn(argc, argv); - } - - strbuf_addstr(&scalar_usage, - N_("scalar [-C ] [-c =] " - " []\n\nCommands:\n")); - for (i = 0; builtins[i].name; i++) - strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name); - - usage(scalar_usage.buf); -} diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt deleted file mode 100644 index f33436c7f6..0000000000 --- a/contrib/scalar/scalar.txt +++ /dev/null @@ -1,166 +0,0 @@ -scalar(1) -========= - -NAME ----- -scalar - A tool for managing large Git repositories - -SYNOPSIS --------- -[verse] -scalar clone [--single-branch] [--branch ] [--full-clone] [] -scalar list -scalar register [] -scalar unregister [] -scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [] -scalar reconfigure [ --all | ] -scalar diagnose [] -scalar delete - -DESCRIPTION ------------ - -Scalar is a repository management tool that optimizes Git for use in large -repositories. Scalar improves performance by configuring advanced Git settings, -maintaining repositories in the background, and helping to reduce data sent -across the network. - -An important Scalar concept is the enlistment: this is the top-level directory -of the project. It usually contains the subdirectory `src/` which is a Git -worktree. This encourages the separation between tracked files (inside `src/`) -and untracked files, such as build artifacts (outside `src/`). When registering -an existing Git worktree with Scalar whose name is not `src`, the enlistment -will be identical to the worktree. - -The `scalar` command implements various subcommands, and different options -depending on the subcommand. With the exception of `clone`, `list` and -`reconfigure --all`, all subcommands expect to be run in an enlistment. - -The following options can be specified _before_ the subcommand: - --C :: - Before running the subcommand, change the working directory. This - option imitates the same option of linkgit:git[1]. - --c =:: - For the duration of running the specified subcommand, configure this - setting. This option imitates the same option of linkgit:git[1]. - -COMMANDS --------- - -Clone -~~~~~ - -clone [] []:: - Clones the specified repository, similar to linkgit:git-clone[1]. By - default, only commit and tree objects are cloned. Once finished, the - worktree is located at `/src`. -+ -The sparse-checkout feature is enabled (except when run with `--full-clone`) -and the only files present are those in the top-level directory. Use -`git sparse-checkout set` to expand the set of directories you want to see, -or `git sparse-checkout disable` to expand to all files (see -linkgit:git-sparse-checkout[1] for more details). You can explore the -subdirectories outside your sparse-checkout by using `git ls-tree -HEAD[:]`. - --b :: ---branch :: - Instead of checking out the branch pointed to by the cloned - repository's HEAD, check out the `` branch instead. - ---[no-]single-branch:: - Clone only the history leading to the tip of a single branch, either - specified by the `--branch` option or the primary branch remote's - `HEAD` points at. -+ -Further fetches into the resulting repository will only update the -remote-tracking branch for the branch this option was used for the initial -cloning. If the HEAD at the remote did not point at any branch when -`--single-branch` clone was made, no remote-tracking branch is created. - ---[no-]full-clone:: - A sparse-checkout is initialized by default. This behavior can be - turned off via `--full-clone`. - -List -~~~~ - -list:: - List enlistments that are currently registered by Scalar. This - subcommand does not need to be run inside an enlistment. - -Register -~~~~~~~~ - -register []:: - Adds the enlistment's repository to the list of registered repositories - and starts background maintenance. If `` is not provided, - then the enlistment associated with the current working directory is - registered. -+ -Note: when this subcommand is called in a worktree that is called `src/`, its -parent directory is considered to be the Scalar enlistment. If the worktree is -_not_ called `src/`, it itself will be considered to be the Scalar enlistment. - -Unregister -~~~~~~~~~~ - -unregister []:: - Remove the specified repository from the list of repositories - registered with Scalar and stop the scheduled background maintenance. - -Run -~~~ - -scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) []:: - Run the given maintenance task (or all tasks, if `all` was specified). - Except for `all` and `config`, this subcommand simply hands off to - linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and - `pack-files` to `incremental-repack`). -+ -These tasks are run automatically as part of the scheduled maintenance, -as soon as the repository is registered with Scalar. It should therefore -not be necessary to run this subcommand manually. -+ -The `config` task is specific to Scalar and configures all those -opinionated default settings that make Git work more efficiently with -large repositories. As this task is run as part of `scalar clone` -automatically, explicit invocations of this task are rarely needed. - -Reconfigure -~~~~~~~~~~~ - -After a Scalar upgrade, or when the configuration of a Scalar enlistment -was somehow corrupted or changed by mistake, this subcommand allows to -reconfigure the enlistment. - -With the `--all` option, all enlistments currently registered with Scalar -will be reconfigured. Use this option after each Scalar upgrade. - -Diagnose -~~~~~~~~ - -diagnose []:: - When reporting issues with Scalar, it is often helpful to provide the - information gathered by this command, including logs and certain - statistics describing the data shape of the current enlistment. -+ -The output of this command is a `.zip` file that is written into -a directory adjacent to the worktree in the `src` directory. - -Delete -~~~~~~ - -delete :: - This subcommand lets you delete an existing Scalar enlistment from your - local file system, unregistering the repository. - -SEE ALSO --------- -linkgit:git-clone[1], linkgit:git-maintenance[1]. - -GIT ---- -Part of the linkgit:git[1] suite diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefile deleted file mode 100644 index 1ed174a8cf..0000000000 --- a/contrib/scalar/t/Makefile +++ /dev/null @@ -1,81 +0,0 @@ -# Import tree-wide shared Makefile behavior and libraries -include ../../../shared.mak - -# Run scalar tests -# -# Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin -# - --include ../../../config.mak.autogen --include ../../../config.mak - -SHELL_PATH ?= $(SHELL) -PERL_PATH ?= /usr/bin/perl -RM ?= rm -f -PROVE ?= prove -DEFAULT_TEST_TARGET ?= test -TEST_LINT ?= test-lint - -ifdef TEST_OUTPUT_DIRECTORY -TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results -else -TEST_RESULTS_DIRECTORY = ../../../t/test-results -endif - -# Shell quote; -SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH)) -PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH)) -TEST_RESULTS_DIRECTORY_SQ = $(subst ','\'',$(TEST_RESULTS_DIRECTORY)) - -T = $(sort $(wildcard t[0-9][0-9][0-9][0-9]-*.sh)) - -all: $(DEFAULT_TEST_TARGET) - -test: $(TEST_LINT) - $(MAKE) aggregate-results-and-cleanup - -prove: $(TEST_LINT) - @echo "*** prove ***"; GIT_CONFIG=.git/config $(PROVE) --exec '$(SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS) - $(MAKE) clean-except-prove-cache - -$(T): - @echo "*** $@ ***"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS) - -clean-except-prove-cache: - $(RM) -r 'trash directory'.* - $(RM) -r valgrind/bin - -clean: clean-except-prove-cache - $(RM) .prove - -test-lint: test-lint-duplicates test-lint-executable test-lint-shell-syntax - -test-lint-duplicates: - @dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \ - test -z "$$dups" || { \ - echo >&2 "duplicate test numbers:" $$dups; exit 1; } - -test-lint-executable: - @bad=`for i in $(T); do test -x "$$i" || echo $$i; done` && \ - test -z "$$bad" || { \ - echo >&2 "non-executable tests:" $$bad; exit 1; } - -test-lint-shell-syntax: - @'$(PERL_PATH_SQ)' ../../../t/check-non-portable-shell.pl $(T) - -aggregate-results-and-cleanup: $(T) - $(MAKE) aggregate-results - $(MAKE) clean - -aggregate-results: - for f in '$(TEST_RESULTS_DIRECTORY_SQ)'/t*-*.counts; do \ - echo "$$f"; \ - done | '$(SHELL_PATH_SQ)' ../../../t/aggregate-results.sh - -valgrind: - $(MAKE) GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind" - -test-results: - mkdir -p test-results - -.PHONY: $(T) aggregate-results clean valgrind diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh deleted file mode 100755 index dfb949f52e..0000000000 --- a/contrib/scalar/t/t9099-scalar.sh +++ /dev/null @@ -1,216 +0,0 @@ -#!/bin/sh - -test_description='test the `scalar` command' - -TEST_DIRECTORY=$PWD/../../../t -export TEST_DIRECTORY - -# Make it work with --no-bin-wrappers -PATH=$PWD/..:$PATH - -. ../../../t/test-lib.sh - -GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt,launchctl:true,schtasks:true" -export GIT_TEST_MAINT_SCHEDULER - -test_expect_success 'scalar shows a usage' ' - test_expect_code 129 scalar -h -' - -test_expect_success 'scalar invoked on enlistment root' ' - test_when_finished rm -rf test src deeper && - - for enlistment_root in test src deeper/test - do - git init ${enlistment_root}/src && - - # Register - scalar register ${enlistment_root} && - scalar list >out && - grep "$(pwd)/${enlistment_root}/src\$" out && - - # Delete (including enlistment root) - scalar delete $enlistment_root && - test_path_is_missing $enlistment_root && - scalar list >out && - ! grep "^$(pwd)/${enlistment_root}/src\$" out || return 1 - done -' - -test_expect_success 'scalar invoked on enlistment src repo' ' - test_when_finished rm -rf test src deeper && - - for enlistment_root in test src deeper/test - do - git init ${enlistment_root}/src && - - # Register - scalar register ${enlistment_root}/src && - scalar list >out && - grep "$(pwd)/${enlistment_root}/src\$" out && - - # Delete (will not include enlistment root) - scalar delete ${enlistment_root}/src && - test_path_is_dir $enlistment_root && - scalar list >out && - ! grep "^$(pwd)/${enlistment_root}/src\$" out || return 1 - done -' - -test_expect_success 'scalar invoked when enlistment root and repo are the same' ' - test_when_finished rm -rf test src deeper && - - for enlistment_root in test src deeper/test - do - git init ${enlistment_root} && - - # Register - scalar register ${enlistment_root} && - scalar list >out && - grep "$(pwd)/${enlistment_root}\$" out && - - # Delete (will not include enlistment root) - scalar delete ${enlistment_root} && - test_path_is_missing $enlistment_root && - scalar list >out && - ! grep "^$(pwd)/${enlistment_root}\$" out && - - # Make sure we did not accidentally delete the trash dir - test_path_is_dir "$TRASH_DIRECTORY" || return 1 - done -' - -test_expect_success 'scalar repo search respects GIT_CEILING_DIRECTORIES' ' - test_when_finished rm -rf test && - - git init test/src && - mkdir -p test/src/deep && - GIT_CEILING_DIRECTORIES="$(pwd)/test/src" && - ! scalar register test/src/deep 2>err && - grep "not a git repository" err -' - -test_expect_success 'scalar enlistments need a worktree' ' - test_when_finished rm -rf bare test && - - git init --bare bare/src && - ! scalar register bare/src 2>err && - grep "Scalar enlistments require a worktree" err && - - git init test/src && - ! scalar register test/src/.git 2>err && - grep "Scalar enlistments require a worktree" err -' - -test_expect_success FSMONITOR_DAEMON 'scalar register starts fsmon daemon' ' - git init test/src && - test_must_fail git -C test/src fsmonitor--daemon status && - scalar register test/src && - git -C test/src fsmonitor--daemon status && - test_cmp_config -C test/src true core.fsmonitor -' - -test_expect_success 'scalar unregister' ' - git init vanish/src && - scalar register vanish/src && - git config --get --global --fixed-value \ - maintenance.repo "$(pwd)/vanish/src" && - scalar list >scalar.repos && - grep -F "$(pwd)/vanish/src" scalar.repos && - rm -rf vanish/src/.git && - scalar unregister vanish && - test_must_fail git config --get --global --fixed-value \ - maintenance.repo "$(pwd)/vanish/src" && - scalar list >scalar.repos && - ! grep -F "$(pwd)/vanish/src" scalar.repos -' - -test_expect_success 'set up repository to clone' ' - test_commit first && - test_commit second && - test_commit third && - git switch -c parallel first && - mkdir -p 1/2 && - test_commit 1/2/3 && - git config uploadPack.allowFilter true && - git config uploadPack.allowAnySHA1InWant true -' - -test_expect_success 'scalar clone' ' - second=$(git rev-parse --verify second:second.t) && - scalar clone "file://$(pwd)" cloned --single-branch && - ( - cd cloned/src && - - git config --get --global --fixed-value maintenance.repo \ - "$(pwd)" && - - git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual && - echo "refs/remotes/origin/parallel" >expect && - test_cmp expect actual && - - test_path_is_missing 1/2 && - test_must_fail git rev-list --missing=print $second && - git rev-list $second && - git cat-file blob $second >actual && - echo "second" >expect && - test_cmp expect actual - ) -' - -test_expect_success 'scalar reconfigure' ' - git init one/src && - scalar register one && - git -C one/src config core.preloadIndex false && - scalar reconfigure one && - test true = "$(git -C one/src config core.preloadIndex)" && - git -C one/src config core.preloadIndex false && - scalar reconfigure -a && - test true = "$(git -C one/src config core.preloadIndex)" -' - -test_expect_success 'scalar delete without enlistment shows a usage' ' - test_expect_code 129 scalar delete -' - -test_expect_success 'scalar delete with enlistment' ' - scalar delete cloned && - test_path_is_missing cloned -' - -test_expect_success 'scalar supports -c/-C' ' - test_when_finished "scalar delete sub" && - git init sub && - scalar -C sub -c status.aheadBehind=bogus register && - test -z "$(git -C sub config --local status.aheadBehind)" && - test true = "$(git -C sub config core.preloadIndex)" -' - -test_expect_success '`scalar [...] ` errors out when dir is missing' ' - ! scalar run config cloned 2>err && - grep "cloned. does not exist" err -' - -SQ="'" -test_expect_success UNZIP 'scalar diagnose' ' - scalar clone "file://$(pwd)" cloned --single-branch && - git repack && - echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && - test_commit -C cloned/src loose && - scalar diagnose cloned >out 2>err && - grep "Available space" out && - sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && - zip_path=$(cat zip_path) && - test -n "$zip_path" && - "$GIT_UNZIP" -v "$zip_path" && - folder=${zip_path%.zip} && - test_path_is_missing "$folder" && - "$GIT_UNZIP" -p "$zip_path" diagnostics.log >out && - test_file_not_empty out && - "$GIT_UNZIP" -p "$zip_path" packs-local.txt >out && - grep "$(pwd)/.git/objects" out && - "$GIT_UNZIP" -p "$zip_path" objects-local.txt >out && - grep "^Total: [1-9]" out -' - -test_done diff --git a/scalar.c b/scalar.c new file mode 100644 index 0000000000..642d16124e --- /dev/null +++ b/scalar.c @@ -0,0 +1,906 @@ +/* + * The Scalar command-line interface. + */ + +#include "cache.h" +#include "gettext.h" +#include "parse-options.h" +#include "config.h" +#include "run-command.h" +#include "simple-ipc.h" +#include "fsmonitor-ipc.h" +#include "fsmonitor-settings.h" +#include "refs.h" +#include "dir.h" +#include "packfile.h" +#include "help.h" + +static void setup_enlistment_directory(int argc, const char **argv, + const char * const *usagestr, + const struct option *options, + struct strbuf *enlistment_root) +{ + struct strbuf path = STRBUF_INIT; + int enlistment_is_repo_parent = 0; + size_t len; + + if (startup_info->have_repository) + BUG("gitdir already set up?!?"); + + if (argc > 1) + usage_with_options(usagestr, options); + + /* find the worktree, determine its corresponding root */ + if (argc == 1) { + strbuf_add_absolute_path(&path, argv[0]); + if (!is_directory(path.buf)) + die(_("'%s' does not exist"), path.buf); + if (chdir(path.buf) < 0) + die_errno(_("could not switch to '%s'"), path.buf); + } else if (strbuf_getcwd(&path) < 0) + die(_("need a working directory")); + + strbuf_trim_trailing_dir_sep(&path); + + /* check if currently in enlistment root with src/ workdir */ + len = path.len; + strbuf_addstr(&path, "/src"); + if (is_nonbare_repository_dir(&path)) { + enlistment_is_repo_parent = 1; + if (chdir(path.buf) < 0) + die_errno(_("could not switch to '%s'"), path.buf); + } + strbuf_setlen(&path, len); + + setup_git_directory(); + + if (!the_repository->worktree) + die(_("Scalar enlistments require a worktree")); + + if (enlistment_root) { + if (enlistment_is_repo_parent) + strbuf_addbuf(enlistment_root, &path); + else + strbuf_addstr(enlistment_root, the_repository->worktree); + } + + strbuf_release(&path); +} + +static int run_git(const char *arg, ...) +{ + struct strvec argv = STRVEC_INIT; + va_list args; + const char *p; + int res; + + va_start(args, arg); + strvec_push(&argv, arg); + while ((p = va_arg(args, const char *))) + strvec_push(&argv, p); + va_end(args); + + res = run_command_v_opt(argv.v, RUN_GIT_CMD); + + strvec_clear(&argv); + return res; +} + +struct scalar_config { + const char *key; + const char *value; + int overwrite_on_reconfigure; +}; + +static int set_scalar_config(const struct scalar_config *config, int reconfigure) +{ + char *value = NULL; + int res; + + if ((reconfigure && config->overwrite_on_reconfigure) || + git_config_get_string(config->key, &value)) { + trace2_data_string("scalar", the_repository, config->key, "created"); + res = git_config_set_gently(config->key, config->value); + } else { + trace2_data_string("scalar", the_repository, config->key, "exists"); + res = 0; + } + + free(value); + return res; +} + +static int have_fsmonitor_support(void) +{ + return fsmonitor_ipc__is_supported() && + fsm_settings__get_reason(the_repository) == FSMONITOR_REASON_OK; +} + +static int set_recommended_config(int reconfigure) +{ + struct scalar_config config[] = { + /* Required */ + { "am.keepCR", "true", 1 }, + { "core.FSCache", "true", 1 }, + { "core.multiPackIndex", "true", 1 }, + { "core.preloadIndex", "true", 1 }, +#ifndef WIN32 + { "core.untrackedCache", "true", 1 }, +#else + /* + * Unfortunately, Scalar's Functional Tests demonstrated + * that the untracked cache feature is unreliable on Windows + * (which is a bummer because that platform would benefit the + * most from it). For some reason, freshly created files seem + * not to update the directory's `lastModified` time + * immediately, but the untracked cache would need to rely on + * that. + * + * Therefore, with a sad heart, we disable this very useful + * feature on Windows. + */ + { "core.untrackedCache", "false", 1 }, +#endif + { "core.logAllRefUpdates", "true", 1 }, + { "credential.https://dev.azure.com.useHttpPath", "true", 1 }, + { "credential.validate", "false", 1 }, /* GCM4W-only */ + { "gc.auto", "0", 1 }, + { "gui.GCWarning", "false", 1 }, + { "index.threads", "true", 1 }, + { "index.version", "4", 1 }, + { "merge.stat", "false", 1 }, + { "merge.renames", "true", 1 }, + { "pack.useBitmaps", "false", 1 }, + { "pack.useSparse", "true", 1 }, + { "receive.autoGC", "false", 1 }, + { "feature.manyFiles", "false", 1 }, + { "feature.experimental", "false", 1 }, + { "fetch.unpackLimit", "1", 1 }, + { "fetch.writeCommitGraph", "false", 1 }, +#ifdef WIN32 + { "http.sslBackend", "schannel", 1 }, +#endif + /* Optional */ + { "status.aheadBehind", "false" }, + { "commitGraph.generationVersion", "1" }, + { "core.autoCRLF", "false" }, + { "core.safeCRLF", "false" }, + { "fetch.showForcedUpdates", "false" }, + { NULL, NULL }, + }; + int i; + char *value; + + for (i = 0; config[i].key; i++) { + if (set_scalar_config(config + i, reconfigure)) + return error(_("could not configure %s=%s"), + config[i].key, config[i].value); + } + + if (have_fsmonitor_support()) { + struct scalar_config fsmonitor = { "core.fsmonitor", "true" }; + if (set_scalar_config(&fsmonitor, reconfigure)) + return error(_("could not configure %s=%s"), + fsmonitor.key, fsmonitor.value); + } + + /* + * The `log.excludeDecoration` setting is special because it allows + * for multiple values. + */ + if (git_config_get_string("log.excludeDecoration", &value)) { + trace2_data_string("scalar", the_repository, + "log.excludeDecoration", "created"); + if (git_config_set_multivar_gently("log.excludeDecoration", + "refs/prefetch/*", + CONFIG_REGEX_NONE, 0)) + return error(_("could not configure " + "log.excludeDecoration")); + } else { + trace2_data_string("scalar", the_repository, + "log.excludeDecoration", "exists"); + free(value); + } + + return 0; +} + +static int toggle_maintenance(int enable) +{ + return run_git("maintenance", enable ? "start" : "unregister", NULL); +} + +static int add_or_remove_enlistment(int add) +{ + int res; + + if (!the_repository->worktree) + die(_("Scalar enlistments require a worktree")); + + res = run_git("config", "--global", "--get", "--fixed-value", + "scalar.repo", the_repository->worktree, NULL); + + /* + * If we want to add and the setting is already there, then do nothing. + * If we want to remove and the setting is not there, then do nothing. + */ + if ((add && !res) || (!add && res)) + return 0; + + return run_git("config", "--global", add ? "--add" : "--unset", + add ? "--no-fixed-value" : "--fixed-value", + "scalar.repo", the_repository->worktree, NULL); +} + +static int start_fsmonitor_daemon(void) +{ + assert(have_fsmonitor_support()); + + if (fsmonitor_ipc__get_state() != IPC_STATE__LISTENING) + return run_git("fsmonitor--daemon", "start", NULL); + + return 0; +} + +static int stop_fsmonitor_daemon(void) +{ + assert(have_fsmonitor_support()); + + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + return run_git("fsmonitor--daemon", "stop", NULL); + + return 0; +} + +static int register_dir(void) +{ + if (add_or_remove_enlistment(1)) + return error(_("could not add enlistment")); + + if (set_recommended_config(0)) + return error(_("could not set recommended config")); + + if (toggle_maintenance(1)) + return error(_("could not turn on maintenance")); + + if (have_fsmonitor_support() && start_fsmonitor_daemon()) { + return error(_("could not start the FSMonitor daemon")); + } + + return 0; +} + +static int unregister_dir(void) +{ + int res = 0; + + if (toggle_maintenance(0)) + res = error(_("could not turn off maintenance")); + + if (add_or_remove_enlistment(0)) + res = error(_("could not remove enlistment")); + + return res; +} + +/* printf-style interface, expects `=` argument */ +static int set_config(const char *fmt, ...) +{ + struct strbuf buf = STRBUF_INIT; + char *value; + int res; + va_list args; + + va_start(args, fmt); + strbuf_vaddf(&buf, fmt, args); + va_end(args); + + value = strchr(buf.buf, '='); + if (value) + *(value++) = '\0'; + res = git_config_set_gently(buf.buf, value); + strbuf_release(&buf); + + return res; +} + +static char *remote_default_branch(const char *url) +{ + struct child_process cp = CHILD_PROCESS_INIT; + struct strbuf out = STRBUF_INIT; + + cp.git_cmd = 1; + strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL); + if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) { + const char *line = out.buf; + + while (*line) { + const char *eol = strchrnul(line, '\n'), *p; + size_t len = eol - line; + char *branch; + + if (!skip_prefix(line, "ref: ", &p) || + !strip_suffix_mem(line, &len, "\tHEAD")) { + line = eol + (*eol == '\n'); + continue; + } + + eol = line + len; + if (skip_prefix(p, "refs/heads/", &p)) { + branch = xstrndup(p, eol - p); + strbuf_release(&out); + return branch; + } + + error(_("remote HEAD is not a branch: '%.*s'"), + (int)(eol - p), p); + strbuf_release(&out); + return NULL; + } + } + warning(_("failed to get default branch name from remote; " + "using local default")); + strbuf_reset(&out); + + child_process_init(&cp); + cp.git_cmd = 1; + strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL); + if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) { + strbuf_trim(&out); + return strbuf_detach(&out, NULL); + } + + strbuf_release(&out); + error(_("failed to get default branch name")); + return NULL; +} + +static int delete_enlistment(struct strbuf *enlistment) +{ +#ifdef WIN32 + struct strbuf parent = STRBUF_INIT; + size_t offset; + char *path_sep; +#endif + + if (unregister_dir()) + return error(_("failed to unregister repository")); + +#ifdef WIN32 + /* + * Change the current directory to one outside of the enlistment so + * that we may delete everything underneath it. + */ + offset = offset_1st_component(enlistment->buf); + path_sep = find_last_dir_sep(enlistment->buf + offset); + strbuf_add(&parent, enlistment->buf, + path_sep ? path_sep - enlistment->buf : offset); + if (chdir(parent.buf) < 0) { + int res = error_errno(_("could not switch to '%s'"), parent.buf); + strbuf_release(&parent); + return res; + } + strbuf_release(&parent); +#endif + + if (have_fsmonitor_support() && stop_fsmonitor_daemon()) + return error(_("failed to stop the FSMonitor daemon")); + + if (remove_dir_recursively(enlistment, 0)) + return error(_("failed to delete enlistment directory")); + + return 0; +} + +/* + * Dummy implementation; Using `get_version_info()` would cause a link error + * without this. + */ +void load_builtin_commands(const char *prefix, struct cmdnames *cmds) +{ + die("not implemented"); +} + +static int cmd_clone(int argc, const char **argv) +{ + const char *branch = NULL; + int full_clone = 0, single_branch = 0; + struct option clone_options[] = { + OPT_STRING('b', "branch", &branch, N_(""), + N_("branch to checkout after clone")), + OPT_BOOL(0, "full-clone", &full_clone, + N_("when cloning, create full working directory")), + OPT_BOOL(0, "single-branch", &single_branch, + N_("only download metadata for the branch that will " + "be checked out")), + OPT_END(), + }; + const char * const clone_usage[] = { + N_("scalar clone [] [--] []"), + NULL + }; + const char *url; + char *enlistment = NULL, *dir = NULL; + struct strbuf buf = STRBUF_INIT; + int res; + + argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0); + + if (argc == 2) { + url = argv[0]; + enlistment = xstrdup(argv[1]); + } else if (argc == 1) { + url = argv[0]; + + strbuf_addstr(&buf, url); + /* Strip trailing slashes, if any */ + while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1])) + strbuf_setlen(&buf, buf.len - 1); + /* Strip suffix `.git`, if any */ + strbuf_strip_suffix(&buf, ".git"); + + enlistment = find_last_dir_sep(buf.buf); + if (!enlistment) { + die(_("cannot deduce worktree name from '%s'"), url); + } + enlistment = xstrdup(enlistment + 1); + } else { + usage_msg_opt(_("You must specify a repository to clone."), + clone_usage, clone_options); + } + + if (is_directory(enlistment)) + die(_("directory '%s' exists already"), enlistment); + + dir = xstrfmt("%s/src", enlistment); + + strbuf_reset(&buf); + if (branch) + strbuf_addf(&buf, "init.defaultBranch=%s", branch); + else { + char *b = repo_default_branch_name(the_repository, 1); + strbuf_addf(&buf, "init.defaultBranch=%s", b); + free(b); + } + + if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL))) + goto cleanup; + + if (chdir(dir) < 0) { + res = error_errno(_("could not switch to '%s'"), dir); + goto cleanup; + } + + setup_git_directory(); + + /* common-main already logs `argv` */ + trace2_def_repo(the_repository); + + if (!branch && !(branch = remote_default_branch(url))) { + res = error(_("failed to get default branch for '%s'"), url); + goto cleanup; + } + + if (set_config("remote.origin.url=%s", url) || + set_config("remote.origin.fetch=" + "+refs/heads/%s:refs/remotes/origin/%s", + single_branch ? branch : "*", + single_branch ? branch : "*") || + set_config("remote.origin.promisor=true") || + set_config("remote.origin.partialCloneFilter=blob:none")) { + res = error(_("could not configure remote in '%s'"), dir); + goto cleanup; + } + + if (!full_clone && + (res = run_git("sparse-checkout", "init", "--cone", NULL))) + goto cleanup; + + if (set_recommended_config(0)) + return error(_("could not configure '%s'"), dir); + + if ((res = run_git("fetch", "--quiet", "origin", NULL))) { + warning(_("partial clone failed; attempting full clone")); + + if (set_config("remote.origin.promisor") || + set_config("remote.origin.partialCloneFilter")) { + res = error(_("could not configure for full clone")); + goto cleanup; + } + + if ((res = run_git("fetch", "--quiet", "origin", NULL))) + goto cleanup; + } + + if ((res = set_config("branch.%s.remote=origin", branch))) + goto cleanup; + if ((res = set_config("branch.%s.merge=refs/heads/%s", + branch, branch))) + goto cleanup; + + strbuf_reset(&buf); + strbuf_addf(&buf, "origin/%s", branch); + res = run_git("checkout", "-f", "-t", buf.buf, NULL); + if (res) + goto cleanup; + + res = register_dir(); + +cleanup: + free(enlistment); + free(dir); + strbuf_release(&buf); + return res; +} + +static int cmd_diagnose(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar diagnose []"), + NULL + }; + struct strbuf diagnostics_root = STRBUF_INIT; + int res = 0; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + setup_enlistment_directory(argc, argv, usage, options, &diagnostics_root); + strbuf_addstr(&diagnostics_root, "/.scalarDiagnostics"); + + res = run_git("diagnose", "--mode=all", "-s", "%Y%m%d_%H%M%S", + "-o", diagnostics_root.buf, NULL); + + strbuf_release(&diagnostics_root); + return res; +} + +static int cmd_list(int argc, const char **argv) +{ + if (argc != 1) + die(_("`scalar list` does not take arguments")); + + if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0) + return -1; + return 0; +} + +static int cmd_register(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar register []"), + NULL + }; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + setup_enlistment_directory(argc, argv, usage, options, NULL); + + return register_dir(); +} + +static int get_scalar_repos(const char *key, const char *value, void *data) +{ + struct string_list *list = data; + + if (!strcmp(key, "scalar.repo")) + string_list_append(list, value); + + return 0; +} + +static int cmd_reconfigure(int argc, const char **argv) +{ + int all = 0; + struct option options[] = { + OPT_BOOL('a', "all", &all, + N_("reconfigure all registered enlistments")), + OPT_END(), + }; + const char * const usage[] = { + N_("scalar reconfigure [--all | ]"), + NULL + }; + struct string_list scalar_repos = STRING_LIST_INIT_DUP; + int i, res = 0; + struct repository r = { NULL }; + struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + if (!all) { + setup_enlistment_directory(argc, argv, usage, options, NULL); + + return set_recommended_config(1); + } + + if (argc > 0) + usage_msg_opt(_("--all or , but not both"), + usage, options); + + git_config(get_scalar_repos, &scalar_repos); + + for (i = 0; i < scalar_repos.nr; i++) { + const char *dir = scalar_repos.items[i].string; + + strbuf_reset(&commondir); + strbuf_reset(&gitdir); + + if (chdir(dir) < 0) { + warning_errno(_("could not switch to '%s'"), dir); + res = -1; + } else if (discover_git_directory(&commondir, &gitdir) < 0) { + warning_errno(_("git repository gone in '%s'"), dir); + res = -1; + } else { + git_config_clear(); + + the_repository = &r; + r.commondir = commondir.buf; + r.gitdir = gitdir.buf; + + if (set_recommended_config(1) < 0) + res = -1; + } + } + + string_list_clear(&scalar_repos, 1); + strbuf_release(&commondir); + strbuf_release(&gitdir); + + return res; +} + +static int cmd_run(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + struct { + const char *arg, *task; + } tasks[] = { + { "config", NULL }, + { "commit-graph", "commit-graph" }, + { "fetch", "prefetch" }, + { "loose-objects", "loose-objects" }, + { "pack-files", "incremental-repack" }, + { NULL, NULL } + }; + struct strbuf buf = STRBUF_INIT; + const char *usagestr[] = { NULL, NULL }; + int i; + + strbuf_addstr(&buf, N_("scalar run []\nTasks:\n")); + for (i = 0; tasks[i].arg; i++) + strbuf_addf(&buf, "\t%s\n", tasks[i].arg); + usagestr[0] = buf.buf; + + argc = parse_options(argc, argv, NULL, options, + usagestr, 0); + + if (!argc) + usage_with_options(usagestr, options); + + if (!strcmp("all", argv[0])) { + i = -1; + } else { + for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++) + ; /* keep looking for the task */ + + if (i > 0 && !tasks[i].arg) { + error(_("no such task: '%s'"), argv[0]); + usage_with_options(usagestr, options); + } + } + + argc--; + argv++; + setup_enlistment_directory(argc, argv, usagestr, options, NULL); + strbuf_release(&buf); + + if (i == 0) + return register_dir(); + + if (i > 0) + return run_git("maintenance", "run", + "--task", tasks[i].task, NULL); + + if (register_dir()) + return -1; + for (i = 1; tasks[i].arg; i++) + if (run_git("maintenance", "run", + "--task", tasks[i].task, NULL)) + return -1; + return 0; +} + +static int remove_deleted_enlistment(struct strbuf *path) +{ + int res = 0; + strbuf_realpath_forgiving(path, path->buf, 1); + + if (run_git("config", "--global", + "--unset", "--fixed-value", + "scalar.repo", path->buf, NULL) < 0) + res = -1; + + if (run_git("config", "--global", + "--unset", "--fixed-value", + "maintenance.repo", path->buf, NULL) < 0) + res = -1; + + return res; +} + +static int cmd_unregister(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar unregister []"), + NULL + }; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + /* + * Be forgiving when the enlistment or worktree does not even exist any + * longer; This can be the case if a user deleted the worktree by + * mistake and _still_ wants to unregister the thing. + */ + if (argc == 1) { + struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT; + + strbuf_addf(&src_path, "%s/src/.git", argv[0]); + strbuf_addf(&workdir_path, "%s/.git", argv[0]); + if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) { + /* remove possible matching registrations */ + int res = -1; + + strbuf_strip_suffix(&src_path, "/.git"); + res = remove_deleted_enlistment(&src_path) && res; + + strbuf_strip_suffix(&workdir_path, "/.git"); + res = remove_deleted_enlistment(&workdir_path) && res; + + strbuf_release(&src_path); + strbuf_release(&workdir_path); + return res; + } + strbuf_release(&src_path); + strbuf_release(&workdir_path); + } + + setup_enlistment_directory(argc, argv, usage, options, NULL); + + return unregister_dir(); +} + +static int cmd_delete(int argc, const char **argv) +{ + char *cwd = xgetcwd(); + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + N_("scalar delete "), + NULL + }; + struct strbuf enlistment = STRBUF_INIT; + int res = 0; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + if (argc != 1) + usage_with_options(usage, options); + + setup_enlistment_directory(argc, argv, usage, options, &enlistment); + + if (dir_inside_of(cwd, enlistment.buf) >= 0) + res = error(_("refusing to delete current working directory")); + else { + close_object_store(the_repository->objects); + res = delete_enlistment(&enlistment); + } + strbuf_release(&enlistment); + free(cwd); + + return res; +} + +static int cmd_version(int argc, const char **argv) +{ + int verbose = 0, build_options = 0; + struct option options[] = { + OPT__VERBOSE(&verbose, N_("include Git version")), + OPT_BOOL(0, "build-options", &build_options, + N_("include Git's build options")), + OPT_END(), + }; + const char * const usage[] = { + N_("scalar verbose [-v | --verbose] [--build-options]"), + NULL + }; + struct strbuf buf = STRBUF_INIT; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + if (argc != 0) + usage_with_options(usage, options); + + get_version_info(&buf, build_options); + fprintf(stderr, "%s\n", buf.buf); + strbuf_release(&buf); + + return 0; +} + +static struct { + const char *name; + int (*fn)(int, const char **); +} builtins[] = { + { "clone", cmd_clone }, + { "list", cmd_list }, + { "register", cmd_register }, + { "unregister", cmd_unregister }, + { "run", cmd_run }, + { "reconfigure", cmd_reconfigure }, + { "delete", cmd_delete }, + { "version", cmd_version }, + { "diagnose", cmd_diagnose }, + { NULL, NULL}, +}; + +int cmd_main(int argc, const char **argv) +{ + struct strbuf scalar_usage = STRBUF_INIT; + int i; + + while (argc > 1 && *argv[1] == '-') { + if (!strcmp(argv[1], "-C")) { + if (argc < 3) + die(_("-C requires a ")); + if (chdir(argv[2]) < 0) + die_errno(_("could not change to '%s'"), + argv[2]); + argc -= 2; + argv += 2; + } else if (!strcmp(argv[1], "-c")) { + if (argc < 3) + die(_("-c requires a = argument")); + git_config_push_parameter(argv[2]); + argc -= 2; + argv += 2; + } else + break; + } + + if (argc > 1) { + argv++; + argc--; + + for (i = 0; builtins[i].name; i++) + if (!strcmp(builtins[i].name, argv[0])) + return !!builtins[i].fn(argc, argv); + } + + strbuf_addstr(&scalar_usage, + N_("scalar [-C ] [-c =] " + " []\n\nCommands:\n")); + for (i = 0; builtins[i].name; i++) + strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name); + + usage(scalar_usage.buf); +} diff --git a/t/t9210-scalar.sh b/t/t9210-scalar.sh new file mode 100755 index 0000000000..14ca575a21 --- /dev/null +++ b/t/t9210-scalar.sh @@ -0,0 +1,210 @@ +#!/bin/sh + +test_description='test the `scalar` command' + +. ./test-lib.sh + +GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt,launchctl:true,schtasks:true" +export GIT_TEST_MAINT_SCHEDULER + +test_expect_success 'scalar shows a usage' ' + test_expect_code 129 scalar -h +' + +test_expect_success 'scalar invoked on enlistment root' ' + test_when_finished rm -rf test src deeper && + + for enlistment_root in test src deeper/test + do + git init ${enlistment_root}/src && + + # Register + scalar register ${enlistment_root} && + scalar list >out && + grep "$(pwd)/${enlistment_root}/src\$" out && + + # Delete (including enlistment root) + scalar delete $enlistment_root && + test_path_is_missing $enlistment_root && + scalar list >out && + ! grep "^$(pwd)/${enlistment_root}/src\$" out || return 1 + done +' + +test_expect_success 'scalar invoked on enlistment src repo' ' + test_when_finished rm -rf test src deeper && + + for enlistment_root in test src deeper/test + do + git init ${enlistment_root}/src && + + # Register + scalar register ${enlistment_root}/src && + scalar list >out && + grep "$(pwd)/${enlistment_root}/src\$" out && + + # Delete (will not include enlistment root) + scalar delete ${enlistment_root}/src && + test_path_is_dir $enlistment_root && + scalar list >out && + ! grep "^$(pwd)/${enlistment_root}/src\$" out || return 1 + done +' + +test_expect_success 'scalar invoked when enlistment root and repo are the same' ' + test_when_finished rm -rf test src deeper && + + for enlistment_root in test src deeper/test + do + git init ${enlistment_root} && + + # Register + scalar register ${enlistment_root} && + scalar list >out && + grep "$(pwd)/${enlistment_root}\$" out && + + # Delete (will not include enlistment root) + scalar delete ${enlistment_root} && + test_path_is_missing $enlistment_root && + scalar list >out && + ! grep "^$(pwd)/${enlistment_root}\$" out && + + # Make sure we did not accidentally delete the trash dir + test_path_is_dir "$TRASH_DIRECTORY" || return 1 + done +' + +test_expect_success 'scalar repo search respects GIT_CEILING_DIRECTORIES' ' + test_when_finished rm -rf test && + + git init test/src && + mkdir -p test/src/deep && + GIT_CEILING_DIRECTORIES="$(pwd)/test/src" && + ! scalar register test/src/deep 2>err && + grep "not a git repository" err +' + +test_expect_success 'scalar enlistments need a worktree' ' + test_when_finished rm -rf bare test && + + git init --bare bare/src && + ! scalar register bare/src 2>err && + grep "Scalar enlistments require a worktree" err && + + git init test/src && + ! scalar register test/src/.git 2>err && + grep "Scalar enlistments require a worktree" err +' + +test_expect_success FSMONITOR_DAEMON 'scalar register starts fsmon daemon' ' + git init test/src && + test_must_fail git -C test/src fsmonitor--daemon status && + scalar register test/src && + git -C test/src fsmonitor--daemon status && + test_cmp_config -C test/src true core.fsmonitor +' + +test_expect_success 'scalar unregister' ' + git init vanish/src && + scalar register vanish/src && + git config --get --global --fixed-value \ + maintenance.repo "$(pwd)/vanish/src" && + scalar list >scalar.repos && + grep -F "$(pwd)/vanish/src" scalar.repos && + rm -rf vanish/src/.git && + scalar unregister vanish && + test_must_fail git config --get --global --fixed-value \ + maintenance.repo "$(pwd)/vanish/src" && + scalar list >scalar.repos && + ! grep -F "$(pwd)/vanish/src" scalar.repos +' + +test_expect_success 'set up repository to clone' ' + test_commit first && + test_commit second && + test_commit third && + git switch -c parallel first && + mkdir -p 1/2 && + test_commit 1/2/3 && + git config uploadPack.allowFilter true && + git config uploadPack.allowAnySHA1InWant true +' + +test_expect_success 'scalar clone' ' + second=$(git rev-parse --verify second:second.t) && + scalar clone "file://$(pwd)" cloned --single-branch && + ( + cd cloned/src && + + git config --get --global --fixed-value maintenance.repo \ + "$(pwd)" && + + git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual && + echo "refs/remotes/origin/parallel" >expect && + test_cmp expect actual && + + test_path_is_missing 1/2 && + test_must_fail git rev-list --missing=print $second && + git rev-list $second && + git cat-file blob $second >actual && + echo "second" >expect && + test_cmp expect actual + ) +' + +test_expect_success 'scalar reconfigure' ' + git init one/src && + scalar register one && + git -C one/src config core.preloadIndex false && + scalar reconfigure one && + test true = "$(git -C one/src config core.preloadIndex)" && + git -C one/src config core.preloadIndex false && + scalar reconfigure -a && + test true = "$(git -C one/src config core.preloadIndex)" +' + +test_expect_success 'scalar delete without enlistment shows a usage' ' + test_expect_code 129 scalar delete +' + +test_expect_success 'scalar delete with enlistment' ' + scalar delete cloned && + test_path_is_missing cloned +' + +test_expect_success 'scalar supports -c/-C' ' + test_when_finished "scalar delete sub" && + git init sub && + scalar -C sub -c status.aheadBehind=bogus register && + test -z "$(git -C sub config --local status.aheadBehind)" && + test true = "$(git -C sub config core.preloadIndex)" +' + +test_expect_success '`scalar [...] ` errors out when dir is missing' ' + ! scalar run config cloned 2>err && + grep "cloned. does not exist" err +' + +SQ="'" +test_expect_success UNZIP 'scalar diagnose' ' + scalar clone "file://$(pwd)" cloned --single-branch && + git repack && + echo "$(pwd)/.git/objects/" >>cloned/src/.git/objects/info/alternates && + test_commit -C cloned/src loose && + scalar diagnose cloned >out 2>err && + grep "Available space" out && + sed -n "s/.*$SQ\\(.*\\.zip\\)$SQ.*/\\1/p" zip_path && + zip_path=$(cat zip_path) && + test -n "$zip_path" && + "$GIT_UNZIP" -v "$zip_path" && + folder=${zip_path%.zip} && + test_path_is_missing "$folder" && + "$GIT_UNZIP" -p "$zip_path" diagnostics.log >out && + test_file_not_empty out && + "$GIT_UNZIP" -p "$zip_path" packs-local.txt >out && + grep "$(pwd)/.git/objects" out && + "$GIT_UNZIP" -p "$zip_path" objects-local.txt >out && + grep "^Total: [1-9]" out +' + +test_done -- cgit v1.3 From dd9603e22822fab19c0557dad0cf1825de325403 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Fri, 2 Sep 2022 15:56:44 +0000 Subject: git help: special-case `scalar` With this commit, `git help scalar` will open the appropriate manual or HTML page (instead of looking for `gitscalar`). Signed-off-by: Johannes Schindelin Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- builtin/help.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/builtin/help.c b/builtin/help.c index 09ac4289f1..6f2796f211 100644 --- a/builtin/help.c +++ b/builtin/help.c @@ -440,6 +440,8 @@ static const char *cmd_to_page(const char *git_cmd) return git_cmd; else if (is_git_command(git_cmd)) return xstrfmt("git-%s", git_cmd); + else if (!strcmp("scalar", git_cmd)) + return xstrdup(git_cmd); else return xstrfmt("git%s", git_cmd); } -- cgit v1.3 From 951759d3a5cb20ffeff4836aa0c4b793fa142b51 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Fri, 2 Sep 2022 15:56:45 +0000 Subject: scalar: implement the `help` subcommand It is merely handing off to `git help scalar`. Signed-off-by: Johannes Schindelin Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- scalar.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/scalar.c b/scalar.c index 642d16124e..c5c1ce6891 100644 --- a/scalar.c +++ b/scalar.c @@ -819,6 +819,25 @@ static int cmd_delete(int argc, const char **argv) return res; } +static int cmd_help(int argc, const char **argv) +{ + struct option options[] = { + OPT_END(), + }; + const char * const usage[] = { + "scalar help", + NULL + }; + + argc = parse_options(argc, argv, NULL, options, + usage, 0); + + if (argc != 0) + usage_with_options(usage, options); + + return run_git("help", "scalar", NULL); +} + static int cmd_version(int argc, const char **argv) { int verbose = 0, build_options = 0; @@ -858,6 +877,7 @@ static struct { { "run", cmd_run }, { "reconfigure", cmd_reconfigure }, { "delete", cmd_delete }, + { "help", cmd_help }, { "version", cmd_version }, { "diagnose", cmd_diagnose }, { NULL, NULL}, -- cgit v1.3 From cc75e556a9de5d62c1ca52db900b729fc830f378 Mon Sep 17 00:00:00 2001 From: Victoria Dye Date: Fri, 2 Sep 2022 15:56:46 +0000 Subject: scalar: add to 'git help -a' command list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add 'scalar' as a 'mainporcelain' command in the Git command list. Update the regex in 'cmd-list.perl' used to match the first line of command documentation to find 'scalar(1)'. Helped-by: Ævar Arnfjörð Bjarmason Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- Documentation/cmd-list.perl | 2 +- command-list.txt | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/Documentation/cmd-list.perl b/Documentation/cmd-list.perl index af5da45d28..9515a499a3 100755 --- a/Documentation/cmd-list.perl +++ b/Documentation/cmd-list.perl @@ -10,7 +10,7 @@ sub format_one { $state = 0; open I, '<', "$name.txt" or die "No such file $name.txt"; while () { - if (/^git[a-z0-9-]*\(([0-9])\)$/) { + if (/^(git|scalar)[a-z0-9-]*\(([0-9])\)$/) { $mansection = $1; next; } diff --git a/command-list.txt b/command-list.txt index f96bdabd7d..93f94e42ab 100644 --- a/command-list.txt +++ b/command-list.txt @@ -235,3 +235,4 @@ gittutorial guide gittutorial-2 guide gitweb ancillaryinterrogators gitworkflows guide +scalar mainporcelain -- cgit v1.3 From 14b4e7e5a455b771f9dee74a5b4eb9a10f2a1cd4 Mon Sep 17 00:00:00 2001 From: Victoria Dye Date: Fri, 2 Sep 2022 15:56:47 +0000 Subject: scalar-clone: add test coverage Create a new test file ('t9211-scalar-clone.sh') to exercise the options and behavior of the 'scalar clone' command. Each test clones to a unique target location and cleans up the cloned repo only when the test passes. This ensures that failed tests' artifacts are captured in CI artifacts for further debugging. Helped-by: Johannes Schindelin Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- t/t9211-scalar-clone.sh | 151 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100755 t/t9211-scalar-clone.sh diff --git a/t/t9211-scalar-clone.sh b/t/t9211-scalar-clone.sh new file mode 100755 index 0000000000..dd33d87e9b --- /dev/null +++ b/t/t9211-scalar-clone.sh @@ -0,0 +1,151 @@ +#!/bin/sh + +test_description='test the `scalar clone` subcommand' + +. ./test-lib.sh + +GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt,launchctl:true,schtasks:true" +export GIT_TEST_MAINT_SCHEDULER + +test_expect_success 'set up repository to clone' ' + rm -rf .git && + git init to-clone && + ( + cd to-clone && + git branch -m base && + + test_commit first && + test_commit second && + test_commit third && + + git switch -c parallel first && + mkdir -p 1/2 && + test_commit 1/2/3 && + + git switch base && + + # By default, permit + git config uploadpack.allowfilter true && + git config uploadpack.allowanysha1inwant true + ) +' + +cleanup_clone () { + rm -rf "$1" +} + +test_expect_success 'creates content in enlistment root' ' + enlistment=cloned && + + scalar clone "file://$(pwd)/to-clone" $enlistment && + ls -A $enlistment >enlistment-root && + test_line_count = 1 enlistment-root && + test_path_is_dir $enlistment/src && + test_path_is_dir $enlistment/src/.git && + + cleanup_clone $enlistment +' + +test_expect_success 'with spaces' ' + enlistment="cloned with space" && + + scalar clone "file://$(pwd)/to-clone" "$enlistment" && + test_path_is_dir "$enlistment" && + test_path_is_dir "$enlistment/src" && + test_path_is_dir "$enlistment/src/.git" && + + cleanup_clone "$enlistment" +' + +test_expect_success 'partial clone if supported by server' ' + enlistment=partial-clone && + + scalar clone "file://$(pwd)/to-clone" $enlistment && + + ( + cd $enlistment/src && + + # Two promisor packs: one for refs, the other for blobs + ls .git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 2 promisorlist + ) && + + cleanup_clone $enlistment +' + +test_expect_success 'fall back on full clone if partial unsupported' ' + enlistment=no-partial-support && + + test_config -C to-clone uploadpack.allowfilter false && + test_config -C to-clone uploadpack.allowanysha1inwant false && + + scalar clone "file://$(pwd)/to-clone" $enlistment 2>err && + grep "filtering not recognized by server, ignoring" err && + + ( + cd $enlistment/src && + + # Still get a refs promisor file, but none for blobs + ls .git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 1 promisorlist + ) && + + cleanup_clone $enlistment +' + +test_expect_success 'initializes sparse-checkout by default' ' + enlistment=sparse && + + scalar clone "file://$(pwd)/to-clone" $enlistment && + ( + cd $enlistment/src && + test_cmp_config true core.sparseCheckout && + test_cmp_config true core.sparseCheckoutCone + ) && + + cleanup_clone $enlistment +' + +test_expect_success '--full-clone does not create sparse-checkout' ' + enlistment=full-clone && + + scalar clone --full-clone "file://$(pwd)/to-clone" $enlistment && + ( + cd $enlistment/src && + test_cmp_config "" --default "" core.sparseCheckout && + test_cmp_config "" --default "" core.sparseCheckoutCone + ) && + + cleanup_clone $enlistment +' + +test_expect_success '--single-branch clones HEAD only' ' + enlistment=single-branch && + + scalar clone --single-branch "file://$(pwd)/to-clone" $enlistment && + ( + cd $enlistment/src && + git for-each-ref refs/remotes/origin >out && + test_line_count = 1 out && + grep "refs/remotes/origin/base" out + ) && + + cleanup_clone $enlistment +' + +test_expect_success '--no-single-branch clones all branches' ' + enlistment=no-single-branch && + + scalar clone --no-single-branch "file://$(pwd)/to-clone" $enlistment && + ( + cd $enlistment/src && + git for-each-ref refs/remotes/origin >out && + test_line_count = 2 out && + grep "refs/remotes/origin/base" out && + grep "refs/remotes/origin/parallel" out + ) && + + cleanup_clone $enlistment +' + +test_done -- cgit v1.3 From e2809233d19e0faacedf59f229ec292cb9e7c7ef Mon Sep 17 00:00:00 2001 From: Victoria Dye Date: Fri, 2 Sep 2022 15:56:48 +0000 Subject: t/perf: add Scalar performance tests Create 'p9210-scalar.sh' for testing Scalar performance and comparing performance of Git operations in Scalar registrations and standard repositories. Example results: Test this tree ------------------------------------------------------------------------ 9210.2: scalar clone 14.82(18.00+3.63) 9210.3: git clone 26.15(36.67+6.90) 9210.4: git status (scalar) 0.04(0.01+0.01) 9210.5: git status (non-scalar) 0.10(0.02+0.11) 9210.6: test_commit --append --no-tag A (scalar) 0.08(0.02+0.03) 9210.7: test_commit --append --no-tag A (non-scalar) 0.13(0.03+0.11) Helped-by: Johannes Schindelin Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- t/perf/p9210-scalar.sh | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 t/perf/p9210-scalar.sh diff --git a/t/perf/p9210-scalar.sh b/t/perf/p9210-scalar.sh new file mode 100755 index 0000000000..265f7cd1fe --- /dev/null +++ b/t/perf/p9210-scalar.sh @@ -0,0 +1,39 @@ +#!/bin/sh + +test_description='test scalar performance' +. ./perf-lib.sh + +test_perf_large_repo "$TRASH_DIRECTORY/to-clone" + +test_expect_success 'enable server-side partial clone' ' + git -C to-clone config uploadpack.allowFilter true && + git -C to-clone config uploadpack.allowAnySHA1InWant true +' + +test_perf 'scalar clone' ' + rm -rf scalar-clone && + scalar clone "file://$(pwd)/to-clone" scalar-clone +' + +test_perf 'git clone' ' + rm -rf git-clone && + git clone "file://$(pwd)/to-clone" git-clone +' + +test_compare_perf () { + command=$1 + shift + args=$* + test_perf "$command $args (scalar)" " + $command -C scalar-clone/src $args + " + + test_perf "$command $args (non-scalar)" " + $command -C git-clone $args + " +} + +test_compare_perf git status +test_compare_perf test_commit --append --no-tag A + +test_done -- cgit v1.3 From ba1b117eec2f73c84f73f827e6f3ac8b82b35585 Mon Sep 17 00:00:00 2001 From: Victoria Dye Date: Fri, 2 Sep 2022 15:56:49 +0000 Subject: t/perf: add 'GIT_PERF_USE_SCALAR' run option Add a 'GIT_PERF_USE_SCALAR' environment variable (and corresponding perf config 'useScalar') to register a repository created with any of: * test_perf_fresh_repo * test_perf_default_repo * test_perf_large_repo as a Scalar enlistment. This is intended to allow a developer to test the impact of Scalar on already-defined performance scenarios. Suggested-by: Derrick Stolee Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- t/perf/README | 4 ++++ t/perf/perf-lib.sh | 13 ++++++++++++- t/perf/run | 3 +++ 3 files changed, 19 insertions(+), 1 deletion(-) diff --git a/t/perf/README b/t/perf/README index fb9127a66f..8f217d7be7 100644 --- a/t/perf/README +++ b/t/perf/README @@ -95,6 +95,10 @@ You can set the following variables (also in your config.mak): Git (e.g., performance of index-pack as the number of threads changes). These can be enabled with GIT_PERF_EXTRA. + GIT_PERF_USE_SCALAR + Boolean indicating whether to register test repo(s) with Scalar + before executing tests. + You can also pass the options taken by ordinary git tests; the most useful one is: diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 27c2801792..e7786775a9 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -49,6 +49,9 @@ export TEST_DIRECTORY TRASH_DIRECTORY GIT_BUILD_DIR GIT_TEST_CMP MODERN_GIT=$GIT_BUILD_DIR/bin-wrappers/git export MODERN_GIT +MODERN_SCALAR=$GIT_BUILD_DIR/bin-wrappers/scalar +export MODERN_SCALAR + perf_results_dir=$TEST_RESULTS_DIR test -n "$GIT_PERF_SUBSECTION" && perf_results_dir="$perf_results_dir/$GIT_PERF_SUBSECTION" mkdir -p "$perf_results_dir" @@ -120,6 +123,10 @@ test_perf_create_repo_from () { # status" due to a locked index. Since we have # a copy it's fine to remove the lock. rm .git/index.lock + fi && + if test_bool_env GIT_PERF_USE_SCALAR false + then + "$MODERN_SCALAR" register fi ) || error "failed to copy repository '$source' to '$repo'" } @@ -130,7 +137,11 @@ test_perf_fresh_repo () { "$MODERN_GIT" init -q "$repo" && ( cd "$repo" && - test_perf_do_repo_symlink_config_ + test_perf_do_repo_symlink_config_ && + if test_bool_env GIT_PERF_USE_SCALAR false + then + "$MODERN_SCALAR" register + fi ) } diff --git a/t/perf/run b/t/perf/run index 55219aa405..33da4d2aba 100755 --- a/t/perf/run +++ b/t/perf/run @@ -171,6 +171,9 @@ run_subsection () { get_var_from_env_or_config "GIT_PERF_MAKE_COMMAND" "perf" "makeCommand" get_var_from_env_or_config "GIT_PERF_MAKE_OPTS" "perf" "makeOpts" + get_var_from_env_or_config "GIT_PERF_USE_SCALAR" "perf" "useScalar" "--bool" + export GIT_PERF_USE_SCALAR + get_var_from_env_or_config "GIT_PERF_REPO_NAME" "perf" "repoName" export GIT_PERF_REPO_NAME -- cgit v1.3 From 9eb7a73158bdc91892a6b9a0b43b8f954b1e39e2 Mon Sep 17 00:00:00 2001 From: Victoria Dye Date: Fri, 2 Sep 2022 15:56:50 +0000 Subject: Documentation/technical: include Scalar technical doc Include 'Documentation/technical/scalar.txt' alongside the other HTML technical docs when installing them. Now that the document is intended as a widely-accessible reference, remove the internal work-in-progress roadmap from the document. Those details should no longer be needed to guide Scalar's development and, if they were left, they could fall out-of-date and be misleading to readers. Signed-off-by: Victoria Dye Signed-off-by: Junio C Hamano --- Documentation/Makefile | 1 + Documentation/technical/scalar.txt | 61 -------------------------------------- 2 files changed, 1 insertion(+), 61 deletions(-) diff --git a/Documentation/Makefile b/Documentation/Makefile index 16c9e06239..9ec53afdf1 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -116,6 +116,7 @@ TECH_DOCS += technical/parallel-checkout TECH_DOCS += technical/partial-clone TECH_DOCS += technical/racy-git TECH_DOCS += technical/reftable +TECH_DOCS += technical/scalar TECH_DOCS += technical/send-pack-pipeline TECH_DOCS += technical/shallow TECH_DOCS += technical/trivial-merge diff --git a/Documentation/technical/scalar.txt b/Documentation/technical/scalar.txt index 0600150b3a..921cb104c3 100644 --- a/Documentation/technical/scalar.txt +++ b/Documentation/technical/scalar.txt @@ -64,64 +64,3 @@ some "global" `git` options (e.g., `-c` and `-C`). Because `scalar` is not invoked as a Git subcommand (like `git scalar`), it is built and installed as its own executable in the `bin/` directory, alongside `git`, `git-gui`, etc. - -Roadmap -------- - -NOTE: this section will be removed once the remaining tasks outlined in this -roadmap are complete. - -Scalar is a large enough project that it is being upstreamed incrementally, -living in `contrib/` until it is feature-complete. So far, the following patch -series have been accepted: - -- `scalar-the-beginning`: The initial patch series which sets up - `contrib/scalar/` and populates it with a minimal `scalar` command that - demonstrates the fundamental ideas. - -- `scalar-c-and-C`: The `scalar` command learns about two options that can be - specified before the command, `-c =` and `-C `. - -- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand. - -- `scalar-generalize-diagnose`: Move the functionality of `scalar diagnose` - into `git diagnose` and `git bugreport --diagnose`. - -- 'scalar-add-fsmonitor: Enable the built-in FSMonitor in Scalar - enlistments. At the end of this series, Scalar should be feature-complete - from the perspective of a user. - -Roughly speaking (and subject to change), the following series are needed to -"finish" this initial version of Scalar: - -- Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of - `git`. This includes a variety of related updates, including: - - building & installing Scalar in the Git root-level 'make [install]'. - - builing & testing Scalar as part of CI. - - moving and expanding test coverage of Scalar (including perf tests). - - implementing 'scalar help'/'git help scalar' to display scalar - documentation. - -Finally, there are two additional patch series that exist in Microsoft's fork of -Git, but there is no current plan to upstream them. There are some interesting -ideas there, but the implementation is too specific to Azure Repos and/or VFS -for Git to be of much help in general. - -These still exist mainly because the GVFS protocol is what Azure Repos has -instead of partial clone, while Git is focused on improving partial clone: - -- `scalar-with-gvfs`: The primary purpose of this patch series is to support - existing Scalar users whose repositories are hosted in Azure Repos (which does - not support Git's partial clones, but supports its predecessor, the GVFS - protocol, which is used by Scalar to emulate the partial clone). - - Since the GVFS protocol will never be supported by core Git, this patch series - will remain in Microsoft's fork of Git. - -- `run-scalar-functional-tests`: The Scalar project developed a quite - comprehensive set of integration tests (or, "Functional Tests"). They are the - sole remaining part of the original C#-based Scalar project, and this patch - adds a GitHub workflow that runs them all. - - Since the tests partially depend on features that are only provided in the - `scalar-with-gvfs` patch series, this patch cannot be upstreamed. -- cgit v1.3