Add a Cobblestone recipe implementing c2sp.org/chunked-encryption by alex · Pull Request #15144 · pyca/cryptography

alex · 2026-07-03T17:02:49Z

Resolves #2358

This adds cryptography.cobblestone, a top-level recipe (à la Fernet) for streaming authenticated encryption of large messages, implementing the C2SP chunked-encryption specification's two named instantiations (per C2SP/C2SP#296): Cobblestone-128 (SHA-512 + AES-128-GCM, recommended) and Cobblestone-256 (SHA-512 + AES-256-GCM, for environments that mandate 256-bit keys).

from cryptography.cobblestone import (
    Cobblestone128Decryptor, Cobblestone128Encryptor
)

key = Cobblestone128Encryptor.generate_key()
enc = Cobblestone128Encryptor(key, context=b"myapp backup encryption")
ciphertext = enc.update(b"a secret message") + enc.finalize()
dec = Cobblestone128Decryptor(key, context=b"myapp backup encryption")
assert dec.update(ciphertext) + dec.finalize() == b"a secret message"

Design notes:

The core is implemented in Rust, on top of the existing AesGcm AEAD and HkdfExpand implementations. The four classes are simple compositions over internal ChunkedEncryptor/ChunkedDecryptor types parameterized by the AEAD (IANA name, key/tag lengths). For each message a fresh AEAD key, base nonce, and 32-byte key commitment are derived with HKDF-Expand-SHA-512 from the input key, a random 24-byte salt, and a caller-provided context; the message is encrypted in 16 KiB chunks, with the chunk counter XOR'd into the base nonce. The ciphertext is salt || commitment || chunks.
Buffering is minimal: full 16 KiB chunks are encrypted/decrypted directly from the caller's input into the output, so only sub-chunk remainders are copied into the internal buffer. update_into() variants let callers supply their own output buffers.
The decrypter checks the commitment before decrypting anything, releases plaintext only for authenticated chunks, and enforces the short-final-chunk truncation rule at finalize().
Auth failures raise cryptography.exceptions.InvalidTag; any further use of a finalized or failed context raises AlreadyFinalized.

Tests cover round-trip, streaming-granularity, tampering, reordering, truncation, extension, and API misuse cases for both variants, plus a Rust unit test for the chunk counter limit (unreachable from Python). Note: the vendored FiloSottile/chunked test vectors (#15145) predate C2SP/C2SP#296 — they were generated with HKDF-SHA-256 — so the vector tests are removed until the reference implementation regenerates them for Cobblestone; they should be re-added (and the vendored file refreshed) once that happens.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

These are the test vectors from the chunked reference implementation (https://github.com/FiloSottile/chunked), for use by the chunked encryption recipe being added in #15144. Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8 Co-authored-by: Claude <noreply@anthropic.com>

This adds cryptography.chunked_encryption, a top-level recipe (like Fernet) for streaming authenticated encryption of large messages, implementing the C2SP chunked-encryption specification (https://c2sp.org/chunked-encryption) instantiated with SHA-256 and AES-128-GCM. The core is implemented in Rust: for each message a fresh key, base nonce, and key commitment are derived with HKDF-Expand-SHA-256 from the input key, a random 24-byte salt, and a caller-provided context; the message is encrypted in 16 KiB chunks with AES-128-GCM, with the chunk counter XOR'd into the base nonce. Full chunks are encrypted/decrypted directly from the caller's input, so only sub-chunk remainders are buffered internally, and update_into variants allow callers to supply output buffers. The internals are parameterized over the AEAD so that additional AEADs could be supported later, but the public API is AES-128-GCM only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

- Use the existing AesGcm AEAD implementation for chunk encryption/decryption instead of using OpenSSL's EVP interface directly (AESGCM.decrypt_into is now pub(crate) for this). - Use the existing HkdfExpand implementation for key derivation. - Fold the error state into the finalized state: a context that hit an error raises AlreadyFinalized on further use. - Replace the let-else in Decrypter::update_impl with a match that yields the cipher and buffer fields. - Replace the in-test reference implementation with the test vectors from the chunked reference implementation (https://github.com/FiloSottile/chunked), vendored into cryptography_vectors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

- Extract the chunk counter/nonce logic into a ChunkNonces struct with Rust unit tests covering the 2**38-chunk limit, which can't be reached from Python tests. - Track the Decrypter state as Option<DecrypterState> and process both active states in a single exhaustive match, removing the unreachable!() arms; finalize() now consumes the state on all paths. - Drop the Encrypter's error poisoning: the only errors it guarded against are the capacity check (which fails before any state is modified) and internal OpenSSL errors. - Test that a Decrypter is unusable after update_into() raises InvalidTag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

…ariant Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

C2SP/C2SP#296 names the chunked-encryption spec's two instantiations: Cobblestone-128 (SHA-512 and AES-128-GCM, recommended) and Cobblestone-256 (SHA-512 and AES-256-GCM, compliance-oriented). Replace the Encrypter/Decrypter classes with Cobblestone128Encryptor, Cobblestone128Decryptor, Cobblestone256Encryptor, and Cobblestone256Decryptor, sharing the core implementation, and switch the HKDF-Expand hash from SHA-256 to SHA-512 per the updated spec. The vendored reference test vectors predate this spec change (they were generated with HKDF-SHA-256), so the vector tests are removed until the reference implementation regenerates them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

The four Cobblestone classes are now plain compositions over internal ChunkedEncryptor/ChunkedDecryptor types. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

alex · 2026-07-14T01:35:09Z

+A message is encrypted in 16 KiB chunks, each of which is individually
+authenticated, so decryption can also be performed as a stream:
+decrypted data is returned incrementally, and only authenticated
+plaintext is ever returned. Reordering, truncating, or extending the
+ciphertext is detected. The scheme is also key committing: a ciphertext
+can only be decrypted with the key it was encrypted with.


None of this information is useful, delete this paragraph.

alex · 2026-07-14T01:35:56Z

+        :meth:`generate_key`, :func:`os.urandom`, or a key derivation
+        function — never a password). A single key may be used to


you don't need to mention os.urandom, e.g. the output of generate key -- never a password is enough

alex · 2026-07-14T01:36:34Z

+            for a given call is included in the :class:`ValueError`
+            raised if the buffer is too small.


Delete everything after hte semicolon

alex · 2026-07-14T01:36:56Z

+        ciphertext. This must always be called, even if ``update``
+        returned all but the last few bytes of ciphertext, and the


delete everything between the first and second commas, it makes no sense

alex · 2026-07-14T01:38:05Z

+    @staticmethod
+    def generate_key() -> bytes: ...


Decryptors shouldn't have a generate key method, only the encryptor side

alex · 2026-07-14T01:42:27Z

+}
+
+#[cfg(test)]
+mod tests {


in general we should only have rust tests for things that are unreachable from python in normal operations, but needed for coverage. do these meet that?

Partially: the counter-limit test does meet the bar (hitting the 2³⁸-chunk limit requires a 4 PiB message, and it's what covers the check_capacity/message_too_large_error error lines), but the nonce XOR test didn't — that path is fully exercised by the Python round-trip tests. Dropped the XOR test and kept only the limit test, with a comment explaining why it exists.

Generated by Claude Code

…e Rust tests - Simplify the docs per review: drop the intro properties paragraph, the os.urandom mention, and the redundant clauses in the update_into and finalize descriptions. - Remove generate_key from the Decryptor classes; key generation is an encryption-side operation. - Drop the nonce XOR unit test: that path is fully covered by the Python round-trip tests. The remaining Rust test covers only the chunk counter limit, which is unreachable from Python. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

alex · 2026-07-14T02:02:51Z

        assert isinstance(key, bytes)
        assert len(key) == key_len
-        assert len(decryptor_cls.generate_key()) == key_len
+        assert not hasattr(decryptor_cls, "generate_key")


this is a very stupid assert, delete it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

reaperhulk · 2026-07-16T18:54:18Z

+    messages produced by :class:`Cobblestone256Encryptor` with a
+    32-byte key.
+
+Implementation


This whole section doesn’t feel useful since we already state that it’s an implementation of c2sp chunked encryption at the top

Per review: the module, docs page, stubs, and Rust module are renamed from chunked_encryption to cobblestone (the classes were already named for the Cobblestone instantiations), and the docs' Implementation section is dropped since the page already states it implements the C2SP chunked-encryption specification. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

alex · 2026-07-17T00:00:45Z

At this point I think the only other thing we want is the wycheproof tests to land so we can use them.

alex · 2026-07-17T00:05:14Z

The consuming test is written and validated: I extracted the vectors from C2SP/wycheproof#265 as it stands and ran them against this branch — all 70 pass (35 per variant: valid decryptions checked against msgLength/msgSha512, invalid ones rejected, InvalidKeySize rejected at construction, and failed contexts keep failing). Once #265 merges I'll push tests/wycheproof/test_cobblestone.py along with a WYCHEPROOF_REF bump so CI picks the files up.

Generated by Claude Code

alex mentioned this pull request Jul 3, 2026

Add test vectors for c2sp.org/chunked-encryption #15145

Merged

alex force-pushed the claude/chunked-encryption-rust-dw7f80 branch from d94f3cf to 896f9cc Compare July 3, 2026 18:11

alex commented Jul 3, 2026

View reviewed changes

Comment thread docs/chunked-encryption.rst Outdated

Comment thread src/rust/src/cobblestone.rs

claude added 7 commits July 13, 2026 00:10

Add PiB to the docs spelling wordlist

107572a

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

Address review feedback: drop overhead paragraph, document buffer inv…

65967e0

…ariant Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

alex force-pushed the claude/chunked-encryption-rust-dw7f80 branch from 6e59943 to 7d06bbe Compare July 13, 2026 00:14

alex commented Jul 14, 2026

View reviewed changes

Remove hasattr assert from test_generate_key

41c9269

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uyk58oD6F8BJwKHE4qCMA8

reaperhulk reviewed Jul 16, 2026

View reviewed changes

alex changed the title ~~Add a chunked encryption recipe implementing c2sp.org/chunked-encryption~~ Add a Cobblestone recipe implementing c2sp.org/chunked-encryption Jul 16, 2026

		:meth:`generate_key`, :func:`os.urandom`, or a key derivation
		function — never a password). A single key may be used to

		for a given call is included in the :class:`ValueError`
		raised if the buffer is too small.

		ciphertext. This must always be called, even if ``update``
		returned all but the last few bytes of ciphertext, and the

Uh oh!

Conversation

alex commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex commented Jul 17, 2026

Uh oh!

alex commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

alex commented Jul 3, 2026 •

edited

Loading