From 87264b7dde2be9e555243f0dce649b785407827e Mon Sep 17 00:00:00 2001 From: "brian m. carlson" Date: Thu, 9 Oct 2025 21:56:18 +0000 Subject: docs: update pack index v3 format Our current pack index v3 format uses 4-byte integers to find the trailer of the file. This effectively means that the file cannot be much larger than 2^32. While this might at first seem to be okay, we expect that each object will have at least 64 bytes worth of data, which means that no more than about 67 million objects can be stored. Again, this might seem fine, but unfortunately, we know of many users who attempt to create repos with extremely large numbers of commits to get a "high score," and we've already seen repositories with at least 55 million commits. In the interests of gracefully handling repositories even for these well-intentioned but ultimately misguided users, let's change these lengths to 8 bytes. For the checksums at the end of the file, we're producing 32-byte SHA-256 checksums because that's what we already do with pack index v2 and SHA-256. Truncating SHA-256 doesn't pose any actual security problems other than those related to the reduced size, but our pack checksum must already be 32 bytes (since SHA-256 packs have 32-byte checksums) and it simplifies the code to use the existing hashfile logic for these cases for the index checksum as well. In addition, even though we may not need cryptographic security for the index checksum, we'd like to avoid arguments from auditors and such for organizations that may have compliance or security requirements. Using the simple, boring choice of the full SHA-256 hash avoids all possible discussion related to hash truncation and removes impediments for these organizations. Note that we do not yet have a pack index v3 implementation in Git, so it should be fine to change this format. However, such an implementation has been written for future inclusion following this format. Signed-off-by: brian m. carlson Signed-off-by: Junio C Hamano --- Documentation/technical/hash-function-transition.adoc | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) (limited to 'Documentation/technical') diff --git a/Documentation/technical/hash-function-transition.adoc b/Documentation/technical/hash-function-transition.adoc index f047fd80ca..274dc993d4 100644 --- a/Documentation/technical/hash-function-transition.adoc +++ b/Documentation/technical/hash-function-transition.adoc @@ -227,9 +227,9 @@ network byte order): ** 4-byte length in bytes of shortened object names. This is the shortest possible length needed to make names in the shortened object name table unambiguous. - ** 4-byte integer, recording where tables relating to this format + ** 8-byte integer, recording where tables relating to this format are stored in this index file, as an offset from the beginning. - * 4-byte offset to the trailer from the beginning of this file. + * 8-byte offset to the trailer from the beginning of this file. * Zero or more additional key/value pairs (4-byte key, 4-byte value). Only one key is supported: 'PSRC'. See the "Loose objects and unreachable objects" section for supported values and how this @@ -276,10 +276,14 @@ network byte order): up to and not including the table of CRC32 values. - Zero or more NUL bytes. - The trailer consists of the following: - * A copy of the 20-byte SHA-256 checksum at the end of the + * A copy of the full main hash checksum at the end of the corresponding packfile. - * 20-byte SHA-256 checksum of all of the above. + * Full main hash checksum of all of the above. + +The "full main hash" is a full-length hash of the main (not compatibility) +algorithm in the repository. Thus, if the main algorithm is SHA-256, this is +a 32-byte SHA-256 hash and for SHA-1, it's a 20-byte SHA-1 hash. Loose object index ~~~~~~~~~~~~~~~~~~ -- cgit v1.3-6-g1900 From 6947ed321d271533237768354b64c145a8df1551 Mon Sep 17 00:00:00 2001 From: "brian m. carlson" Date: Thu, 9 Oct 2025 21:56:19 +0000 Subject: docs: update offset order for pack index v3 The current design of pack index v3 has items in two different orders: sorted shortened object ID order and pack order. The shortened object IDs and the pack index offset values are in the former order and everything else is in the latter. This, however, poses some problems. We have many parts of the packfile code that expect to find out data about an object knowing only its index in pack order. With the current design, to find the pack offset after having looked up the index in pack order, we must then look up the full object ID and use that to look up the shortened object ID to find the pack offset, which is inconvenient, inefficient, and leads to poor cache usage. Instead, let's change the offset values to be looked up by pack order. This works better because once we know the pack order offset, we can find the full object name and its location in the pack with a simple index into their respective tables. This makes many operations much more efficient, especially with the functions we already have, and it avoids the need for the revindex with pack index v3. Signed-off-by: brian m. carlson Signed-off-by: Junio C Hamano --- Documentation/technical/hash-function-transition.adoc | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) (limited to 'Documentation/technical') diff --git a/Documentation/technical/hash-function-transition.adoc b/Documentation/technical/hash-function-transition.adoc index 274dc993d4..adb0c61e53 100644 --- a/Documentation/technical/hash-function-transition.adoc +++ b/Documentation/technical/hash-function-transition.adoc @@ -260,12 +260,10 @@ network byte order): compressed data to be copied directly from pack to pack during repacking without undetected data corruption. - * A table of 4-byte offset values. For an object in the table of - sorted shortened object names, the value at the corresponding - index in this table indicates where that object can be found in - the pack file. These are usually 31-bit pack file offsets, but - large offsets are encoded as an index into the next table with the - most significant bit set. + * A table of 4-byte offset values. The index of this table in pack order + indicates where that object can be found in the pack file. These are + usually 31-bit pack file offsets, but large offsets are encoded as + an index into the next table with the most significant bit set. * A table of 8-byte offset entries (empty for pack files less than 2 GiB). Pack files are organized with heavily used objects toward -- cgit v1.3-6-g1900 From d477892b30b25333badb829190eb349fb671458c Mon Sep 17 00:00:00 2001 From: "brian m. carlson" Date: Thu, 9 Oct 2025 21:56:20 +0000 Subject: docs: reflect actual double signature for tags The documentation for the hash function transition reflects the original design where the SHA-256 signature would always be placed in a header. However, due to a missed patch in Git 2.29, we shipped SHA-256 support such that the signature for the current algorithm is always an in-body signature and the opposite algorithm is always in a header. Since the documentation is inaccurate, update it to reflect the correct information. Signed-off-by: brian m. carlson Signed-off-by: Junio C Hamano --- .../technical/hash-function-transition.adoc | 24 ++++++++++++---------- 1 file changed, 13 insertions(+), 11 deletions(-) (limited to 'Documentation/technical') diff --git a/Documentation/technical/hash-function-transition.adoc b/Documentation/technical/hash-function-transition.adoc index adb0c61e53..2359d7d106 100644 --- a/Documentation/technical/hash-function-transition.adoc +++ b/Documentation/technical/hash-function-transition.adoc @@ -429,17 +429,19 @@ ordinary unsigned commit. Signed Tags ~~~~~~~~~~~ -We add a new field "gpgsig-sha256" to the tag object format to allow -signing tags without relying on SHA-1. Its signed payload is the -SHA-256 content of the tag with its gpgsig-sha256 field and "-----BEGIN PGP -SIGNATURE-----" delimited in-body signature removed. - -This means tags can be signed - -1. using SHA-1 only, as in existing signed tag objects -2. using both SHA-1 and SHA-256, by using gpgsig-sha256 and an in-body - signature. -3. using only SHA-256, by only using the gpgsig-sha256 field. +We add new fields "gpgsig" and "gpgsig-sha256" to the tag object format to +allow signing tags in both formats. The in-body signature is used for the +signature in the current hash algorithm and the header is used for the +signature in the other algorithm. Thus, a dual-signature tag will contain both +an in-body signature and a gpgsig-sha256 header for the SHA-1 format of an +object or both an in-body signature and a gpgsig header for the SHA-256 format +of and object. + +The signed payload of the tag is the content of the tag in the current +algorithm with both its gpgsig and gpgsig-sha256 fields and +"-----BEGIN PGP SIGNATURE-----" delimited in-body signature removed. + +This means tags can be signed using one or both algorithms. Mergetag embedding ~~~~~~~~~~~~~~~~~~ -- cgit v1.3-6-g1900