fix: add system view fallback chain for Redshift schema discovery#38
Merged
Merged
Conversation
On some Redshift clusters (data sharing, Lake Formation, Serverless), svv_tables returns empty results. Add auto-detection at init time with fallback chain: svv_tables -> svv_all_tables -> information_schema. - Probe each source with COUNT query, pick first with rows > 0 - Handle column name differences (table_schema vs schema_name) - Filter svv_all_tables by current_database() to avoid cross-DB noise - Add db_filter to _get_table_columns for cross-DB isolation - Extract _process_bulk_rows and _get_tables_bulk_information_schema to avoid state mutation in fallback (concurrency safe) - Skip caching empty results to prevent 24h dead cache - Normalize schema name to lowercase - Add 8 unit tests for detection chain and row processing Closes #37 Bump version to 0.2.0rc11.
luiscarbonel1991
added
bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
svv_tables→svv_all_tables→information_schema_get_table_columnsmissing database filter when usingsvv_all_columns(could return cross-DB columns)_process_bulk_rowsand_get_tables_bulk_information_schemato avoid state mutation in fallback (concurrency safe)Problem
On some Redshift clusters,
svv_tablesreturns 0 rows for users that DO have access to tables. Validated with direct probes:svv_tablessvv_all_tablesinformation_schema.tablesTest plan
nlp2sql inspectagainst affected cluster — detectssvv_all_tables, finds 95 tablesnlp2sql querygenerates valid SQL and executes against RedshiftCloses #37