Add localhost embeddings support for scenario automation#3947
Draft
gryan11 wants to merge 6 commits intomicrosoft:mainfrom
Draft
Add localhost embeddings support for scenario automation#3947gryan11 wants to merge 6 commits intomicrosoft:mainfrom
gryan11 wants to merge 6 commits intomicrosoft:mainfrom
Conversation
Member
gryan11
commented
Feb 23, 2026
gryan11
commented
- ScenarioAutomationCAPIClientImpl: intercepts EmbeddingsCodeSearch requests, strips CAPI auth headers, sends clean JSON to localhost:4443
- ScenarioAutomationGithubCodeSearchService: overrides getRemoteIndexState to always return Ready, bypassing production index checks
- Register MockGithubAvailableEmbeddingTypesService in scenario automation to use metis_1024_I16_Binary without hitting production endpoint
- Move all scenario-automation service overrides into the isScenarioAutomation if/else branch in services.ts
rwoll
reviewed
Feb 24, 2026
| import { IFetcherService } from '../../networking/common/fetcherService'; | ||
| import { CAPIClientImpl } from './capiClientImpl'; | ||
|
|
||
| const SCENARIO_AUTOMATION_CODE_SEARCH_URL = 'https://localhost:4443/embeddings/code/search'; |
Member
There was a problem hiding this comment.
@gryan11 - Avoid hard coding this. I suggest adding a formal vscode setting, marked as experimental, that allows you to set the URL (and potentially specify an environment variable for auth).
By doing so, you'll be able to open vscode (outside test scenarios) and then configure it to use your local or (remote) staging endpoint to dogfood and debug.
There's similar settings: see github.copilot.advanced.debug.overrideCapiUrl and github.copilot.advanced.debug.overrideProxyUrl for an example. Essentially, you want a similar feature.
gryan11
force-pushed
the
ryangabriel/semantic_search_evaluation_override
branch
4 times, most recently
from
c7d7d24 to
4a4937e
Compare
- ScenarioAutomationCAPIClientImpl: intercepts EmbeddingsCodeSearch requests, strips CAPI auth headers, sends clean JSON to localhost:4443 - ScenarioAutomationGithubCodeSearchService: overrides getRemoteIndexState to always return Ready, bypassing production index checks - Register MockGithubAvailableEmbeddingTypesService in scenario automation to use metis_1024_I16_Binary without hitting production endpoint - Move all scenario-automation service overrides into the isScenarioAutomation if/else branch in services.ts
Add github.copilot.chat.advanced.debug.overrideEmbeddingsUrl setting (experimental) so the embeddings code search endpoint can be configured in VS Code settings instead of being hardcoded in ScenarioAutomationCAPIClientImpl.
When set, adds Authorization: Bearer header to embeddings requests sent to the override URL in ScenarioAutomationCAPIClientImpl.
gryan11
force-pushed
the
ryangabriel/semantic_search_evaluation_override
branch
2 times, most recently
from
f610ea6 to
4123940
Compare
Member
|
@gryan11 - ping me on teams when you need another review. |
Add ScenarioAutomation overrides so the semantic_search tool works in msbench evaluation runs with Blackbird local server: - ScenarioAutomationEndpointProviderImpl: fall back to copilot-base when a model family (e.g. copilot-fast/gpt-4o-mini) is unavailable via the capi proxy - StaticGitHubAuthenticationService: override copilotToken getter to return a placeholder when the real token is missing or is noAuth, so WorkspaceChunkSearchService can initialize - ScenarioAutomationWorkspaceChunkSearchService: new service that prevents both index states from reporting disabled simultaneously and gracefully handles searchFileChunks errors - services.ts: register the new service in isScenarioAutomation block - searchSubagentPrompt.tsx: encourage semantic_search usage in search subagent prompt - package.json/configurationService.ts: sync overrideEmbeddingsUrl default between package.json and code to fix extension activation - Remove stale copilot-chat-0.37.0.vsix that caused DEPENDENCY errors Tested: msbench run 22411601902 — 0 infra errors, semantic_search called in 3/11 tasks returning valid Blackbird code snippets. https://msbenchapp.azurewebsites.net/run-analysis/22411601902 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
gryan11
force-pushed
the
ryangabriel/semantic_search_evaluation_override
branch
from
4123940 to
bc6b15b
Compare
Override copilotToken getter in StaticGitHubAuthenticationService to return a placeholder token when overrideEmbeddingsUrl is configured and the real token is missing or noAuth. This allows WorkspaceChunkSearchService.tryInit() to pass the copilotToken check and initialize, enabling semantic_search to route requests through the Blackbird local server. The override is conditioned on DebugOverrideEmbeddingsUrl being set, so normal extension test behaviour is preserved. Also remove the silent try/catch from ScenarioAutomationWorkspaceChunkSearchService.searchFileChunks so errors propagate to logs instead of being swallowed. Test run: https://msbenchapp.azurewebsites.net/run-analysis/22452275228 (90.91% resolved, 0 errors, semantic_search returning valid Blackbird code snippets in 3/11 tasks) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The default value for DebugOverrideEmbeddingsUrl was set to 'http://localhost:4443/api/' which caused the copilotToken placeholder to always activate, breaking the ConversationFeature extension test. Reset the default to undefined so the placeholder only activates when the setting is explicitly configured (e.g. via vscodeSettings in the msbench config YAML). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.