always increment finishedQueries counter if incrementing allQueries#13786
Open
jkt-signal wants to merge 1 commit into
Open
always increment finishedQueries counter if incrementing allQueries#13786jkt-signal wants to merge 1 commit into
jkt-signal wants to merge 1 commit into
Conversation
Author
|
(Unfortunately, I don't appear to have the rights to apply the RFC label as suggested by the commit process document, so I'm marking this PR ready for review instead.) |
jkt-signal
marked this pull request as ready for review
July 24, 2026 20:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Storage server processes publish a pair of counters
total_queriesandfinished_queries(as well as a derived special counterquery_queue_max) tracking how many read operations have been started and finished (and the high-water-mark for the number of pending queries—those that have been started, but not finished—in a status-fetch interval).Unfortunately, the counter underlying
total_querieswas incremented at the beginning of one of several actors, and the matching increment tofinished_querieswas done at the end, with several interveningwaits. If the actor was cancelled (or experienced another non-returnable error),finished_querieswas not incremented, resulting in a permanent drift betweentotal_queriesandall_queriesand apparently endless growth inquery_queue_max.The easiest way to demonstrate this behavior is to have two different clients alternate between setting watches and updating values for the same keys—for example, run two simultaneous instances of this simple Java program against a single database, and monitor its status to see
query_queue_maxon storage servers go up permanently every time you run the test.To solve this problem, we add a simple class
CountedSectiontoflow/Stats.hto bump two counters in RAII-enforced balance; once a localCountedSectionhas bumped the first counter, it will always eventually bump the second one whether it is destroyed as a result of normal exit or exception. (Of course, if you store one in a heap object and leak it, you will still experience drift, so, uh, don't do that.)CountedSectionis made move-only so you can use it in actor state, sincestatevariables seem to always be default-constructed and then assigned to their initializers even if the initialization comes before the firstwaitin an actor.One of the actors involved actually updates a second counter pair,
getMappedRangeQueriesandfinishedGetMappedRangeQueries, in a way vulnerable to the same problem, so we useCountedSectionto address that as well, though I have not seen any instances of these getting out of sync (entirely due to lack of trying).Some other metrics updated at the ends of the affected actors—mostly latency metrics, but also some tagged size distributions—are not addressed by this change. Arguably they should not be, since including latency and response size of interrupted queries may significantly throw off the statistics in a not-necessarily-useful way.
Leaving this as an RFC because I'm not sufficiently familiar with the FoundationDB testing setup to write a good test; the Java program linked above is a great demonstration but is not trivially adapted to an integration test (and presumably we'd like tests of the fdbserver itself to be C++, not dependent on the Java bindings). Advice for how to test the change greatly appreciated!
This PR is against the 7.3 branch, rather than
main, because all the relevant actors have been rewritten to use C++20 standard coroutines in the interim, and while I strongly suspect the same bug applies equally inmain, I was not able to confirm this becausemaindoes not currently build a usable fdb for me. Porting forward tomainshould be relatively trivial and I'm happy to do so if requested.