Guidance on adding REFUTE benchmark task (scientific critique & calibration)

[connerlambden]

Hi LightEval team,

BGPT released REFUTE, an Apache-2.0 Hugging Face benchmark for scientific critique and epistemic calibration:

• Dataset: https://huggingface.co/datasets/BGPT-OFFICIAL/refute
• Technical report: https://huggingface.co/datasets/BGPT-OFFICIAL/refute/blob/main/TECHNICAL_REPORT.md

We have prototype integrations:

• Inspect AI: https://huggingface.co/datasets/BGPT-OFFICIAL/refute/tree/main/integrations/inspect_ai
• lm-evaluation-harness: https://huggingface.co/datasets/BGPT-OFFICIAL/refute/tree/main/integrations/lm_eval_harness

Before opening a LightEval contribution, we'd appreciate guidance on:

1. Preferred task format for mixed generative (rubric-scored critique) + objective (forced-choice / soundness) axes
2. Whether REFUTE fits LightEval's benchmark scope
3. Recommended path for Hub eval-results compatibility

Happy to adapt to repo conventions. Thanks!

[connerlambden]

Inspect AI adapter now packaged at https://github.com/connerlambden/refute-inspect (register PR to Inspect Evals pending). lm-eval upstream PR: EleutherAI/lm-evaluation-harness#3821

[connerlambden]

Update: lm-eval upstream PR EleutherAI/lm-evaluation-harness#3821 and packaged Inspect adapter https://github.com/connerlambden/refute-inspect. Happy to align a LightEval task YAML to your preferred HF-dataset task pattern.

[connerlambden]

INTEGRATORS.md now live on the dataset repo: https://huggingface.co/datasets/BGPT-OFFICIAL/refute/blob/main/INTEGRATORS.md — lists Inspect + lm-eval paths; LightEval native task still welcome if you can point to a preferred HF-dataset task template.