refactor(core_text): split BenchDatasetEvaluatorQuestion.run and fix row indexing - #514
Open
MBemera wants to merge 8 commits into
Open
refactor(core_text): split BenchDatasetEvaluatorQuestion.run and fix row indexing#514MBemera wants to merge 8 commits into
MBemera wants to merge 8 commits into
Conversation
re.template was deprecated in 3.11 and removed in 3.13, so importing dataflow.prompts.text2sql raised ImportError and took every text2sql operator and pipeline with it. The name was never used.
_read_audio_base64 called base64.b64decode without the import, so any base64 audio input raised NameError.
Providers send "content": null when a reply carries only tool calls or
only reasoning. .get('content', '') returns None there, so the regex
below raised TypeError, which the caller swallowed into a dropped row
after five pointless retries.
The guard raised NameError instead of the intended ValueError, hiding which method was rejected.
…estion Both names were read but never assigned, so the 'no valid reference answers' branch raised AttributeError then NameError on an ordinary data condition. Mirrors ReasoningAnswerModelJudgeFilter, which this operator was derived from.
Each test fails on the parent commit and passes with its fix.
support_subquestions was assigned only in the semantic branch, but statistic() reads it and match mode calls statistic(), so match mode raised AttributeError. 'from email.policy import strict' was an unused editor auto-import, the same class of mistake as the re.template one. Also documents keep_all_samples in get_desc and tightens the tests to decode real base64 audio rather than assert the module attribute.
…xing run() carried both evaluation modes inline at ~90 lines. Splitting it into per-mode helpers made two latent problems visible: - Match mode indexed the answer Series by position. That is a label lookup on a pandas Series, so any frame with a non-default index -- what an upstream filter produces -- raised KeyError. Now iterates dataframe.index. - Column validation ran after 'answer_match_result' had already been added, so a frame failing validation came back mutated. Also drops the unused numpy and json imports and the dead result_mask.
Collaborator
|
Thanks for your interest in DataFlow and for taking the time to contribute these fixes. I noticed that this PR currently includes most of the commits from #513 in addition to the I’ll leave a separate comment on #513 regarding the shared runtime fixes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #513.
run()held both evaluation modes inline at ~90 lines; this splits it into per-mode helpers.That surfaced two real problems. Match mode indexed the answer Series by position, which is a label lookup on a pandas Series, so any frame with a non-default index (what an upstream filter produces) raised
KeyError. And column validation ran afteranswer_match_resultwas already added, so a frame failing validation came back mutated. Both have new tests; the indexing one is parametrised over both modes on a frame indexed[3, 7].Behaviour is otherwise unchanged. I exercised semantic, match and subquestion modes on gapped indices against the previous implementation.
pytest test/goes 75 to 78 passing, same 8 pre-existing failures.Happy to adjust.