rn-132: add pack-refs article

Author: chriscool

rev_news/drafts/edition-132.md

@@ -25,9 +25,84 @@ This edition covers what happened during the months of January and February 2026
 ### Reviews
 -->
 
-<!---
+
 ### Support
--->
+
+* [Slow git pack-refs --all](https://www.google.com/search?q=https://lore.kernel.org/git/CH3PR12MB9026B5872FD42F031970074BC2B3A%40CH3PR12MB9026.namprd12.prod.outlook.com)
+
+  Martin Fick started the discussion by reporting a significant
+  performance issue where `git pack-refs --all` was taking over five
+  minutes to complete on a large repository (~3M refs) hosted on an
+  NFS filesystem. This delay was particularly problematic for Gerrit
+  servers because Git holds the `packed-refs.lock` for nearly the
+  entire duration, blocking other reference updates. Martin noted that
+  JGit was able to perform the same operation in under 20 seconds on
+  the same repository, suggesting the bottleneck was specific to the
+  Git implementation.
+
+  The `packed-refs` file is used by Git to store a large number of
+  references in a single sorted file to avoid the overhead of many
+  small "loose" reference files. However, updating this file requires
+  rewriting it entirely, and Git typically verifies that objects exist
+  and "peels" tags (finding the underlying object a tag points to)
+  during this process.
+
+  brian m. carlson replied to Martin, suggesting that the slowdown
+  might be occurring in `should_pack_ref()` because Git needs to
+  verify that the object at the end of a reference actually
+  exists. Brian also pointed out that NFS was likely a major factor,
+  as the network latency involved in opening many pack files and
+  checking loose objects can be substantial. He suggested setting
+  `receive.unpackLimit` to 1 to reduce the number of loose objects
+  created in the first place.
+
+  Peff (alias Jeff King) explained that the `packed-refs` file stores
+  "tag-peeling" information, which requires Git to open each object
+  for newly written refs via `peel_object()` to read its header and
+  determine its type. Peff noted that this logic resides in
+  `write_with_updates()` within `packed-backend.c`.
+
+  Martin conducted further testing using `strace` and `drop_caches` to
+  eliminate filesystem caching interference. He discovered that while
+  the actual `write()` calls were fast, there were long gaps—up to
+  four minutes in total—where the program was not making any system
+  calls. Martin hypothesized that this "hidden" time was spent by the
+  kernel handling page faults for `mmap()`ed memory over NFS.
+
+  Patrick Steinhardt concurred that NFS was frequently a source of
+  such performance issues, mentioning that GitLab had eventually
+  sunsetted the use of NFS for this reason. Patrick suggested using
+  `perf(1)` to generate a flame graph to see exactly where the CPU was
+  spending time.
+
+  Martin provided a summary of a flame graph, which showed about
+  one-third of the time spent in `_memcmp_sse4_1` and another third in
+  `unpack_object_header_buffer()`, both accompanied by high page fault
+  rates. He also noticed significant time spent in a function he
+  identified as `packed_refs_store_create()`.
+
+  Peff corrected the function name to `packed_ref_store_create()` and
+  noted that Git might be performing an extra linear pass through the
+  `packed-refs` file if it lacks certain header tags. He discovered
+  that JGit-generated files were missing the `sorted` and
+  `fully-peeled` traits in the header. Without the `sorted` tag, Git
+  reads the entire file linearly to verify its order before it can
+  perform binary searches. Peff suggested that JGit should be updated
+  to write these markers.
+
+  In a final breakthrough, Martin tested adding these tags
+  manually. He found that while the `sorted` tag did not provide a
+  major boost, adding the `fully-peeled` tag was the "trigger" that
+  dropped the execution time from over five minutes to under four
+  seconds. The absence of the `fully-peeled` tag was forcing Git to
+  re-peel every reference by looking up the objects in the pack files
+  over the slow NFS connection.
+
+  The discussion concluded with the identification of a specific
+  interoperability issue between JGit and Git. By identifying that the
+  missing `fully-peeled` header was causing redundant, expensive I/O
+  operations, Martin was able to plan a fix for JGit that would
+  resolve the performance bottleneck on his production servers.
 
 <!---
 ## Developer Spotlight: