feat(qa): targeted test command templates + full test discovery in project index#402
Merged
Conversation
…oject index
Roadmap phase 4 (iteration economics): the coder/QA prompts read
per-service test commands from project_index.json, but
ServiceAnalyzer._detect_testing() only knew Vitest/Jest/pytest and
never emitted test_command at all.
- TestDiscovery: new targeted_command template per framework
({target} = path for pytest/jest/go-style runners, name/filter for
mvn/gradle/dotnet/ctest). Exposed on the result and in to_dict().
- ServiceAnalyzer: _detect_testing now delegates to TestDiscovery,
emitting testing, e2e_testing, test_command, coverage_command, and
targeted_test_command for every supported ecosystem, so QA fixers
can run single tests instead of full suites in compiled stacks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Oleg Miagkov <mrobenner@gmail.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Contributor
📝 WalkthroughWalkthroughTesting analysis now uses ChangesTesting discovery integration
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant ServiceAnalyzer
participant TestDiscovery
participant TestDiscoveryResult
ServiceAnalyzer->>TestDiscovery: discover service path
TestDiscovery->>TestDiscoveryResult: populate frameworks and commands
TestDiscovery-->>ServiceAnalyzer: return discovery result
ServiceAnalyzer->>ServiceAnalyzer: record testing, e2e_testing, and command fields
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ServiceAnalyzer now reports framework names via TestDiscovery, which uses canonical lowercase identifiers (vitest, playwright) instead of display names (Vitest, Playwright). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Oleg Miagkov <mrobenner@gmail.com>
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/backend/analysis/analyzers/service_analyzer.py`:
- Around line 216-235: Broaden the exception handling around
TestDiscovery().discover(self.path) in the analyzer’s testing-detection block to
also handle filesystem-related OSError failures, preserving the existing
fallback behavior when discovery cannot run. Keep unrelated exceptions out of
scope unless required by the surrounding error-handling convention.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 6297f263-0407-4574-9a36-ef3cc8a387a6
📒 Files selected for processing (5)
apps/backend/analysis/analyzers/service_analyzer.pyapps/backend/analysis/test_discovery.pytests/test_debug_assistant.pytests/test_discovery.pytests/test_service_analyzer_testing.py
Comment on lines
+216
to
+235
| try: | ||
| from analysis.test_discovery import TestDiscovery | ||
|
|
||
| discovery = TestDiscovery().discover(self.path) | ||
| unit_frameworks = [f.name for f in discovery.frameworks if f.type != "e2e"] | ||
| e2e_frameworks = [f.name for f in discovery.frameworks if f.type == "e2e"] | ||
|
|
||
| if unit_frameworks: | ||
| self.analysis["testing"] = unit_frameworks[0] | ||
| if e2e_frameworks: | ||
| self.analysis["e2e_testing"] = e2e_frameworks[0] | ||
| if discovery.test_command: | ||
| self.analysis["test_command"] = discovery.test_command | ||
| if discovery.coverage_command: | ||
| self.analysis["coverage_command"] = discovery.coverage_command | ||
| if discovery.targeted_command: | ||
| # Template with {target}: path for pytest/jest/go-style | ||
| # runners, test name/filter for mvn/gradle/dotnet/ctest | ||
| self.analysis["targeted_test_command"] = discovery.targeted_command | ||
| except ImportError: |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
Consider broadening the exception guard for robustness.
Only ImportError is caught, but TestDiscovery().discover(self.path) could raise other exceptions (e.g., OSError from Path.resolve() or glob() on restricted paths). This would crash the entire analyze() call rather than falling back to minimal detection. The discovery code handles most file/JSON errors internally, so the risk is low, but wrapping the discover() call in a broader guard (or adding OSError) would improve resilience.
As per path instructions, check for proper error handling and security considerations.
♻️ Proposed refactor
try:
from analysis.test_discovery import TestDiscovery
discovery = TestDiscovery().discover(self.path)
- except ImportError:
+ except (ImportError, OSError):
# Fall back to the previous minimal detection📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| try: | |
| from analysis.test_discovery import TestDiscovery | |
| discovery = TestDiscovery().discover(self.path) | |
| unit_frameworks = [f.name for f in discovery.frameworks if f.type != "e2e"] | |
| e2e_frameworks = [f.name for f in discovery.frameworks if f.type == "e2e"] | |
| if unit_frameworks: | |
| self.analysis["testing"] = unit_frameworks[0] | |
| if e2e_frameworks: | |
| self.analysis["e2e_testing"] = e2e_frameworks[0] | |
| if discovery.test_command: | |
| self.analysis["test_command"] = discovery.test_command | |
| if discovery.coverage_command: | |
| self.analysis["coverage_command"] = discovery.coverage_command | |
| if discovery.targeted_command: | |
| # Template with {target}: path for pytest/jest/go-style | |
| # runners, test name/filter for mvn/gradle/dotnet/ctest | |
| self.analysis["targeted_test_command"] = discovery.targeted_command | |
| except ImportError: | |
| try: | |
| from analysis.test_discovery import TestDiscovery | |
| discovery = TestDiscovery().discover(self.path) | |
| unit_frameworks = [f.name for f in discovery.frameworks if f.type != "e2e"] | |
| e2e_frameworks = [f.name for f in discovery.frameworks if f.type == "e2e"] | |
| if unit_frameworks: | |
| self.analysis["testing"] = unit_frameworks[0] | |
| if e2e_frameworks: | |
| self.analysis["e2e_testing"] = e2e_frameworks[0] | |
| if discovery.test_command: | |
| self.analysis["test_command"] = discovery.test_command | |
| if discovery.coverage_command: | |
| self.analysis["coverage_command"] = discovery.coverage_command | |
| if discovery.targeted_command: | |
| # Template with {target}: path for pytest/jest/go-style | |
| # runners, test name/filter for mvn/gradle/dotnet/ctest | |
| self.analysis["targeted_test_command"] = discovery.targeted_command | |
| except (ImportError, OSError): |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/backend/analysis/analyzers/service_analyzer.py` around lines 216 - 235,
Broaden the exception handling around TestDiscovery().discover(self.path) in the
analyzer’s testing-detection block to also handle filesystem-related OSError
failures, preserving the existing fallback behavior when discovery cannot run.
Keep unrelated exceptions out of scope unless required by the surrounding
error-handling convention.
Source: Path instructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Roadmap phase 4 — QA iteration economics. Two connected gaps:
project_index.json(what coder/QA prompts actually read) had almost no test info.ServiceAnalyzer._detect_testing()recognized only Vitest/Jest/pytest and never emittedtest_command. Now it delegates toTestDiscovery(all ecosystems from feat(qa): test discovery parity for JVM, .NET, native, and scripting-tail languages #388) and emitstesting,e2e_testing,test_command,coverage_command, andtargeted_test_commandper service.No way to run a subset of tests. For compiled stacks a full-suite run per QA-fix iteration is expensive.
TestDiscoverynow produces atargeted_commandtemplate with a{target}placeholder:pytest {target},npx jest {target},go test {target},mix test {target}mvn test -Dtest={target},./gradlew test --tests {target},dotnet test --filter "{target}",ctest -R {target}The Gradle template keeps the wrapper when the project ships one. Frameworks without a reliable targeted form (zig/stack/cabal/make) yield
Nonerather than a guess.QA agents pick the template up from
project_index.json— the LLM does the file→target mapping, deterministic code stays honest.Testing
tests/test_discovery.py: 72 passed (5 new). Newtests/test_service_analyzer_testing.py: 4 passed. Ruff check + format clean.🤖 Generated with Claude Code
Summary by CodeRabbit