diff --git a/docs/internals/data-structures.rst b/docs/internals/data-structures.rst index 007e871338..03fe44d71b 100644 --- a/docs/internals/data-structures.rst +++ b/docs/internals/data-structures.rst @@ -553,16 +553,25 @@ run, its age is reset to 0, otherwise its age is incremented by one. If a file was not seen in BORG_FILES_CACHE_TTL backups, its cache entry is removed. -The files cache is a python dictionary, storing python objects, which -generates a lot of overhead. +The files cache is a python dictionary. To keep the memory overhead of python +objects low, the value is not kept as a python tuple, but in a "compressed" form: + +- the chunks list is reduced from (256bit chunk id, 32bit size) tuples to bare + 32bit indexes into the chunks index (see ``ChunkIndex.k_to_idx``). The chunk + id and size are looked up from the chunks index again when the entry is used. + This only works while that chunks index is in memory. +- the resulting entry is then msgpacked, so one dict value is a single ``bytes`` + object instead of a nested structure of python objects. Borg can also work without using the files cache (saves memory if you have a lot of files or not much RAM free), then all files are assumed to have changed. This is usually much slower than with files cache. The on-disk format of the files cache is a stream of msgpacked tuples (key, value). +There, the chunks list is stored in its uncompressed form (chunk id and size), as +the chunks index indexes are only valid for one specific in-memory chunks index. Loading the files cache involves reading the file, one msgpack object at a time, -unpacking it, and msgpacking the value (in an effort to save memory). +unpacking it, and compressing the entry as described above. .. _index: @@ -574,13 +583,40 @@ It is used to determine whether we already have a specific chunk. The chunks index is a key -> value mapping and contains: -* key: +* key (32 bytes): - chunk id_hash -* value: - - - reference count (always MAX_VALUE as we do not refcount anymore) - - size (0 for prev. existing objects, we can't query their plaintext size) +* value (48 bytes, ``ChunkIndexEntry`` in ``borg.hashindex``): + + - flags (32bit): ``F_USED`` (chunk is used / referenced), ``F_COMPRESS`` (chunk + shall get re-compressed), ``F_PENDING`` (the chunk is still buffered in the pack + writer, so its pack location is not resolved yet). The upper 8 bits are reserved + for system flags (currently ``F_NEW``) and are not visible to users of the index. + - size (32bit): plaintext chunk size, 0 if not known (see below) + - pack_id (32 bytes): id of the pack file the chunk's blob is stored in + - obj_offset (32bit): byte offset of the blob inside that pack file + - obj_size (32bit): blob length (header + encrypted_meta + encrypted_data) + +The last 3 values are the chunk's location, see :ref:`pack-index-entry`: reading a +chunk is one ranged read of ``[obj_offset, obj_offset + obj_size)`` from +``packs/``. + +So a chunks index entry is 32 + 48 == 80 bytes, and that is also exactly what it +needs on disk (the serialized format is just key/value pairs, no padding, plus a +small header). In memory, there is some additional overhead, see below. + +Not all of that is persisted, though: when an index fragment is written, flags and +size are zeroed (only the chunk id and the pack location are of interest there). +Thus, a chunks index that was just built from the repository has size == 0 for all +its entries, no matter whether it came from the index fragments or from the slow +rebuild (which reads the pack headers, where only the stored blob size is known, +not the plaintext size). + +The plaintext size of an entry is only filled in while borg is running, for the +chunks it actually processes: by ``borg create`` when it adds or re-uses a chunk, +or when the files cache entries of a previous archive are loaded (their chunks +lists have the plaintext sizes). So code using the chunks index must be prepared +to see size == 0 and must not assume it is the real chunk size. The chunks index is a HashIndex_. @@ -594,21 +630,31 @@ Here is the estimated memory usage of Borg - it's complicated:: chunk_size ~= 2 ^ HASH_MASK_BITS (for buzhash chunker, BLOCK_SIZE for fixed chunker) chunk_count ~= total_file_size / chunk_size - chunks_index_usage = chunk_count * 40 + chunks_index_usage = chunk_count * 100 - files_cache_usage = total_file_count * 240 + chunk_count * 165 + files_cache_usage = total_file_count * 230 + chunk_count * 6 mem_usage ~= chunks_index_usage + files_cache_usage - = chunk_count * 205 + total_file_count * 240 + = chunk_count * 106 + total_file_count * 230 -Due to the hashtables, the best/usual/worst cases for memory allocation can -be estimated like that:: +All units are Bytes. - mem_allocation = mem_usage / load_factor # l_f = 0.25 .. 0.75 +The 100 Bytes per chunks index entry are the 80 Bytes of the entry itself plus +the overhead of the hash table it lives in (see HashIndex_): the keys/values +arrays are over-allocated by up to 30%, and the bucket table adds another 4 Bytes +per bucket at a load factor of 0.25 .. 0.5. So, depending on where between two +resizes the index currently is, the real value is somewhere between 88 and 120 +Bytes per entry - 100 is a good average. - mem_allocation_peak = mem_allocation * (1 + growth_factor) # g_f = 1.1 .. 2 +The files cache numbers are for CPython on a 64bit platform: the ~230 Bytes per file +cover the dict slot, the 32 Bytes path hash (as a python ``bytes`` object) and the +fixed part of the msgpacked value; the ~6 Bytes per chunk are one msgpacked 32bit +index into the chunks index. -All units are Bytes. +Both data structures grow by re-allocating and copying, so there are short-time +peaks in memory usage while a resize happens (worst case about 2x the values +computed above for the structure being resized). Usually this does not happen for +all data structures at the same time, though. It is assuming every chunk is referenced exactly once (if you have a lot of duplicate chunks, you will have fewer chunks than estimated above). @@ -617,26 +663,24 @@ It is also assuming that typical chunk size is 2^HASH_MASK_BITS (if you have a lot of files smaller than this statistical medium chunk size, you will have more chunks than estimated above, because 1 file is at least 1 chunk). -The chunks index and files cache are all implemented as hash tables. -A hash table must have a significant amount of unused entries to be fast - -the so-called load factor gives the used/unused elements ratio. - -When a hash table gets full (load factor getting too high), it needs to be -grown (allocate new, bigger hash table, copy all elements over to it, free old -hash table) - this will lead to short-time peaks in memory usage each time this -happens. Usually does not happen for all hashtables at the same time, though. -For small hash tables, we start with a growth factor of 2, which comes down to -~1.1x for big hash tables. +The chunks index and files cache are both implemented as hash tables (the chunks +index as a HashIndex_, the files cache as a python dict). A hash table must have a +significant amount of unused entries to be fast - the so-called load factor gives +the used/unused elements ratio. E.g. backing up a total count of 1 Mi (IEC binary prefix i.e. 2^20) files with a total size of 1TiB. a) with ``create --chunker-params buzhash,10,23,16,4095`` (custom): - mem_usage = 2.8GiB + chunk_count = 16 Mi, chunks_index_usage = 1.56GiB, files_cache_usage = 0.32GiB + + mem_usage = 1.9GiB b) with ``create --chunker-params buzhash,19,23,21,4095`` (default): - mem_usage = 0.31GiB + chunk_count = 512 Ki, chunks_index_usage = 0.05GiB, files_cache_usage = 0.23GiB + + mem_usage = 0.28GiB .. note:: There is also the ``--files-cache=disabled`` option to disable the files cache. You'll save some memory, but it will need to read / chunk all the files as @@ -647,65 +691,69 @@ b) with ``create --chunker-params buzhash,19,23,21,4095`` (default): HashIndex --------- -The chunks index is implemented as a hash table, with -only one slot per bucket, spreading hash collisions to the following -buckets. As a consequence the hash is just a start position for a linear -search. If a key is looked up that is not in the table, then the hash table -is searched from the start position (the hash) until the first empty -bucket is reached. +The chunks index is implemented on top of ``borghash.HashTableNT``, which comes from +the separate `borghash `_ package (Cython). +``borg.hashindex.ChunkIndex`` only adds the borg specific parts on top of it: the +``ChunkIndexEntry`` namedtuple / struct format and the handling of the system flags. -This particular mode of operation is open addressing with linear probing. +``HashTableNT`` packs/unpacks the namedtuple value to/from ``bytes`` using a +``struct.Struct`` and delegates the actual storage to ``borghash.HashTable``, which +is a fixed key size / fixed value size ``bytes -> bytes`` mapping. -When the hash table is filled to 75%, its size is grown. When it's -emptied to 25%, its size is shrunken. Operations on it have a variable -complexity between constant and linear with low factor, and memory overhead -varies between 33% and 300%. +Internally, ``HashTable`` is not one, but three arrays: -If an element is deleted, and the slot behind the deleted element is not empty, -then the element will leave a tombstone, a bucket marked as deleted. Tombstones -are only removed by insertions using the tombstone's bucket, or by resizing -the table. They present the same load to the hash table as a real entry, -but do not count towards the regular load factor. +- the *bucket table*, an array of ``uint32_t`` indexes into the keys/values arrays. + ``0xffffffff`` marks an empty bucket, ``0xfffffffe`` marks a deleted bucket + (tombstone); everything ``>= 0xffffff00`` is reserved, so the usable index range + (and thus the maximum number of entries) is a bit below 4Gi. +- the *keys* array, holding ``key_size`` (32 for the chunks index) Bytes per entry. +- the *values* array, holding ``value_size`` (48 for the chunks index) Bytes per entry. -Thus, if the number of empty slots becomes too low (recall that linear probing -for an element not in the index stops at the first empty slot), the hash table -is rebuilt. The maximum *effective* load factor, i.e. including tombstones, is 93%. +Keys and values are appended to their arrays in insertion order, so the index of a +key in the keys array is stable while the hash table is in memory. The files cache +uses that to "compress" chunk ids to 32bit numbers, see ``ChunkIndex.k_to_idx``. -Data in a HashIndex is always stored in little-endian format, which increases -efficiency for almost everyone, since basically no one uses big-endian processors -any more. +The bucket table has only one slot per bucket, spreading hash collisions to the +following buckets. As a consequence the hash is just a start position for a linear +search. If a key is looked up that is not in the table, then the bucket table is +searched from the start position (the hash) until the first empty bucket is reached. -HashIndex does not use a hashing function, because all keys (save manifest) are -outputs of a cryptographic hash or MAC and thus already have excellent distribution. -Thus, HashIndex simply uses the first 32 bits of the key as its "hash". +This particular mode of operation is open addressing with linear probing. -The format is easy to read and write, because the buckets array has the same layout -in memory and on disk. Only the header formats differ. The on-disk header is -``struct HashHeader``: +The bucket table is grown (by 2x) when the number of used buckets plus tombstones +exceeds 50% of its capacity, and shrunken (to 40%, but never below 1000 buckets) +when the number of used buckets drops below 10% of its capacity. So its load factor +usually is between 0.25 and 0.5. That is cheap, because a bucket is only 4 Bytes - +the bulk of the data is in the keys/values arrays, which are not hash tables and +thus do not need any unused space for speed. They are just grown by 1.3x whenever +they are full. -- First, the HashIndex magic, the eight byte ASCII string "BORG_IDX". -- Second, the signed 32-bit number of entries (i.e. buckets which are not deleted and not empty). -- Third, the signed 32-bit number of buckets, i.e. the length of the buckets array - contained in the file, and the modulus for index calculation. -- Fourth, the signed 8-bit length of keys. -- Fifth, the signed 8-bit length of values. This has to be at least four bytes. +If an element is deleted, its bucket is marked with a tombstone (the keys/values +array slots are zeroed, but not reclaimed until the next rebuild). Tombstones are +only removed by resizing / rebuilding the bucket table. They present the same load +to the hash table as a real entry (recall that linear probing for an element not in +the index stops at the first empty bucket), which is why they count towards the load +factor that triggers the growth. -All fields are packed. +Data in a HashIndex is stored in little-endian format, which increases efficiency +for almost everyone, since basically no one uses big-endian processors any more. -The HashIndex is *not* a general purpose data structure. -The value size must be at least 4 bytes, and these first bytes are used for in-band -signalling in the data structure itself. +HashIndex does not use a hashing function, because all keys (save manifest) are +outputs of a cryptographic hash or MAC and thus already have excellent distribution. +Thus, HashIndex simply uses the first 32 bits of the key as its "hash". -The constant MAX_VALUE (defined as 2**32-1025 = 4294966271) defines the valid range for -these 4 bytes when interpreted as an uint32_t from 0 to MAX_VALUE (inclusive). -The following reserved values beyond MAX_VALUE are currently in use (byte order is LE): +The on-disk format does not mirror the in-memory layout - neither the bucket table +nor the unused space of the keys/values arrays are written. A serialized HashIndex is: -- 0xffffffff marks empty buckets in the hash table -- 0xfffffffe marks deleted buckets in the hash table +- First, a header: the eight byte ASCII string "BORGHASH", an ``uint32`` format + version and an ``uint32`` length of the metadata block (all little-endian). +- Second, the metadata block, a JSON object with the key size, value size, byte + order, the value namedtuple's name / fields / struct format, the bucket table + capacity and the number of entries ("used"). +- Third, "used" times a (key, value) pair, without any padding or separators. -HashIndex is implemented in C and wrapped with Cython in a class-based interface. -The Cython wrapper checks every passed value against these reserved values and -raises an AssertionError if they are used. +So the on-disk size is ``entries * (key_size + value_size)`` plus a small header, +i.e. exactly 80 Bytes per entry for the chunks index. .. _data-encryption: diff --git a/src/borg/cache.py b/src/borg/cache.py index 8560b20026..e500b116a2 100644 --- a/src/borg/cache.py +++ b/src/borg/cache.py @@ -989,9 +989,7 @@ class AdHocWithFilesCache(FilesCacheMixin, ChunksMixin): """ An ad-hoc chunks and files cache. - Chunks: it does not maintain accurate reference count. - Chunks that were not added during the current lifetime won't have correct size set (0 bytes) - and will have an infinite reference count (MAX_VALUE). + Chunks: Chunks that were not added during the current lifetime won't have correct size set (0 bytes). Files: if a previous_archive_id is given, ad-hoc build a in-memory files cache from that archive. """