feat: add gossipsub message payload event for raw wire-bytes capture#877
Merged
Conversation
Adds LIBP2P_TRACE_GOSSIPSUB_MESSAGE_PAYLOAD end-to-end: proto event and payload messages, clmimicry handler (opt-in via gossipSubMessagePayloadEnabled, off by default), event-ingester registration, ClickHouse table libp2p_gossipsub_message_payload, and the consumoor route. The table's sorting key and distributed sharding key are content-derived (message IDs are content hashes), so identical messages captured by multiple vantage points deduplicate on merge. Payload bytes are stored exactly as received off the wire, so capture works for any topic without a decoder, including validation-rejected messages. Claude-Session: https://claude.ai/code/session_01KDvdo7uVQpxKWgk2gLnWpE
validate() already rejects events missing the wallclock fields or message ID, so the sentinel-zero fallback branches were unreachable and read as if missing data could silently insert epoch-zero timestamps. Claude-Session: https://claude.ai/code/session_01KDvdo7uVQpxKWgk2gLnWpE
The table deduplicates by content-derived message ID across capture clients, so per-observation columns were nondeterministic: whichever row survived the merge won. peer_id_unique_key and reject_reason are dropped (peer attribution and reject reasons live in libp2p_deliver_message and libp2p_reject_message), and outcome moves into the sorting key so a message delivered by one client and rejected by another keeps one row per outcome instead of racing. Claude-Session: https://claude.ai/code/session_01KDvdo7uVQpxKWgk2gLnWpE
The table stores message content keyed by content-derived message ID; validation outcome is a fact about an observation, not the bytes, and lives per-vantage in libp2p_deliver_message and libp2p_reject_message. This also collapses fork-contested messages (delivered by some clients, rejected by others) to a single row. The proto event keeps outcome and reject_reason for stream consumers. Claude-Session: https://claude.ai/code/session_01KDvdo7uVQpxKWgk2gLnWpE
A nil Data slice is a legitimate zero-byte message, and rejecting it meant exactly that class of message was never archived. Also documents that dedup is best-effort for messages received across wallclock slot or partition boundaries. Claude-Session: https://claude.ai/code/session_01KDvdo7uVQpxKWgk2gLnWpE
samcm
added a commit
that referenced
this pull request
Jul 17, 2026
…load event Brings LIBP2P_TRACE_GOSSIPSUB_MESSAGE_PAYLOAD (#877) onto the devnet branch. Proto numbering reconciled: this branch had already assigned the numbers master used, so the event takes 136 (enum), 124 (ClientMeta oneof) and 257 (DecoratedEvent oneof) here, with notes for eventual convergence. Generated protos rebuilt; categorizer union of the ePBS and payload events. Claude-Session: https://claude.ai/code/session_01KDvdo7uVQpxKWgk2gLnWpE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new
LIBP2P_TRACE_GOSSIPSUB_MESSAGE_PAYLOADevent that carries the raw wire bytes of gossipsub messages, wired end-to-end: proto definitions, clmimicry handler (off by default viagossipSubMessagePayloadEnabled), event-ingester registration, alibp2p_gossipsub_message_payloadClickHouse table, and the consumoor route. Message IDs are content-derived, so the table's sorting and sharding keys dedupe identical messages captured by multiple clients. Validation-rejected messages are captured too, with the outcome and reject reason stored alongside the bytes.https://claude.ai/code/session_01KDvdo7uVQpxKWgk2gLnWpE