-
Notifications
You must be signed in to change notification settings - Fork 150
Added Golden Checks for EIA Imports [EIA_Electricity, EIA_Coal, EIA_Petroleum, EIA_NaturalGas, EIA_International, EIA_TotalEnergy, EIA_SEDS, EIA_NuclearOutages] and CensusCountyBusinessPatterns #2101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
kartik-s21
wants to merge
37
commits into
datacommonsorg:master
Choose a base branch
from
kartik-s21:addedgoldens
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
ea4cdd2
Fixed Code for 3 files
kartik-s21 91d22a7
NameError Resolved
kartik-s21 c3084c4
Merge branch 'master' into code_fix_unenergy
kartik-s21 297374c
Merge branch 'master' into code_fix_unenergy
kartik-s21 34bb357
Merge pull request #1 from kartik-s21/code_fix_unenergy
kartik-s21 ee1e0b2
Merge branch 'datacommonsorg:master' into master
kartik-s21 8707bba
Merge branch 'datacommonsorg:master' into master
kartik-s21 5527dcd
Merge branch 'datacommonsorg:master' into master
kartik-s21 e41a8a7
Merge branch 'datacommonsorg:master' into master
kartik-s21 7f17aa3
Resolve merge conflict in energy process script
kartik-s21 59bcad2
Merge branch 'datacommonsorg:master' into master
kartik-s21 4fc439a
Merge branch 'datacommonsorg:master' into master
kartik-s21 9e5417c
Merge branch 'datacommonsorg:master' into master
kartik-s21 cb72442
Merge branch 'datacommonsorg:master' into master
kartik-s21 d9bb0a6
Merge branch 'datacommonsorg:master' into master
kartik-s21 2515f96
Merge branch 'datacommonsorg:master' into master
kartik-s21 c099907
Merge branch 'datacommonsorg:master' into master
kartik-s21 1359560
Merge branch 'datacommonsorg:master' into master
kartik-s21 0dfeecc
Merge branch 'datacommonsorg:master' into master
kartik-s21 b09267f
Merge branch 'datacommonsorg:master' into master
kartik-s21 776feff
Merge branch 'datacommonsorg:master' into master
kartik-s21 4b06212
Merge branch 'datacommonsorg:master' into master
kartik-s21 8bb164a
Merge branch 'datacommonsorg:master' into master
kartik-s21 cf8f281
Added goldens for EIA imports, NewYork_Education and CensusCountyBusi…
kartik-s21 fd7d02d
Merge branch 'master' into addedgoldens
kartik-s21 3ec4d2c
added goldens
kartik-s21 6a59bb6
Revert changes for new_york_education import
kartik-s21 750401b
Update validation_config.json
kartik-s21 2a0b12c
Update validation_config_EIA_Coal.json
kartik-s21 73e067a
Update validation_config_EIA_Electricity.json
kartik-s21 9493d45
Update validation_config_EIA_International.json
kartik-s21 cd5105b
Update validation_config_EIA_NaturalGas.json
kartik-s21 36e093b
Update validation_config_EIA_NuclearOutages.json
kartik-s21 9cdfb10
Update validation_config_EIA_Petroleum.json
kartik-s21 975c316
Update validation_config_EIA_SEDS.json
kartik-s21 17b850f
Update validation_config_EIA_TotalEnergy.json
kartik-s21 9bb5507
Merge branch 'master' into addedgoldens
balit-raibot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
10,392 changes: 10,392 additions & 0 deletions
10,392
scripts/census_county_business_patterns/golden_data/golden_observations.csv
Large diffs are not rendered by default.
Oops, something went wrong.
6,139 changes: 6,139 additions & 0 deletions
6,139
scripts/census_county_business_patterns/golden_data/golden_summary_report.csv
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
29 changes: 29 additions & 0 deletions
29
scripts/census_county_business_patterns/validation_config.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| { | ||
| "schema_version": "1.0", | ||
| "rules": [ | ||
| { | ||
| "rule_id": "check_deleted_records_percent", | ||
| "description": "Checks that the percentage of deleted points is within the threshold.", | ||
| "validator": "DELETED_RECORDS_PERCENT", | ||
| "params": { | ||
| "threshold": 0.1 | ||
| } | ||
| }, | ||
| { | ||
| "rule_id": "check_goldens_summary_report", | ||
| "validator": "GOLDENS_CHECK", | ||
| "params": { | ||
| "golden_files": "../../../../golden_data/golden_summary_report.csv" | ||
| } | ||
| }, | ||
| { | ||
| "rule_id": "check_goldens_output_csv", | ||
| "description": "Verifies the generated output CSV data matches established critical golden records", | ||
| "validator": "GOLDENS_CHECK", | ||
| "params": { | ||
| "golden_files": "../../../../golden_data/golden_observations.csv", | ||
| "input_files": "../../../../gcs_output/output/*.csv" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
52 changes: 52 additions & 0 deletions
52
scripts/us_eia/opendata/golden_data/golden_observations_EIA_Coal.csv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| "place" | ||
| "geoId/12" | ||
| "geoId/19" | ||
| "geoId/06" | ||
| "geoId/13" | ||
| "geoId/08" | ||
| "geoId/15" | ||
| "geoId/09" | ||
| "geoId/10" | ||
| "geoId/11" | ||
| "geoId/02" | ||
| "geoId/05" | ||
| "geoId/04" | ||
| "geoId/01" | ||
| "geoId/56" | ||
| "geoId/54" | ||
| "geoId/55" | ||
| "geoId/47" | ||
| "geoId/53" | ||
| "geoId/51" | ||
| "country/USA" | ||
| "geoId/48" | ||
| "geoId/49" | ||
| "geoId/42" | ||
| "geoId/45" | ||
| "geoId/36" | ||
| "geoId/46" | ||
| "geoId/39" | ||
| "geoId/40" | ||
| "geoId/41" | ||
| "geoId/28" | ||
| "geoId/38" | ||
| "geoId/37" | ||
| "geoId/32" | ||
| "geoId/33" | ||
| "geoId/29" | ||
| "geoId/35" | ||
| "geoId/34" | ||
| "geoId/30" | ||
| "geoId/44" | ||
| "geoId/31" | ||
| "geoId/26" | ||
| "geoId/24" | ||
| "geoId/23" | ||
| "geoId/27" | ||
| "geoId/16" | ||
| "geoId/20" | ||
| "geoId/17" | ||
| "geoId/18" | ||
| "geoId/21" | ||
| "geoId/22" | ||
| "geoId/25" |
53 changes: 53 additions & 0 deletions
53
scripts/us_eia/opendata/golden_data/golden_observations_EIA_Electricity.csv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| "place" | ||
| "geoId/30" | ||
| "geoId/55" | ||
| "geoId/46" | ||
| "geoId/44" | ||
| "geoId/56" | ||
| "geoId/40" | ||
| "geoId/53" | ||
| "geoId/29" | ||
| "geoId/28" | ||
| "geoId/36" | ||
| "geoId/39" | ||
| "geoId/50" | ||
| "geoId/54" | ||
| "geoId/45" | ||
| "geoId/38" | ||
| "geoId/31" | ||
| "geoId/37" | ||
| "geoId/42" | ||
| "geoId/01" | ||
| "geoId/21" | ||
| "geoId/27" | ||
| "geoId/05" | ||
| "geoId/41" | ||
| "geoId/17" | ||
| "geoId/18" | ||
| "geoId/09" | ||
| "geoId/32" | ||
| "geoId/51" | ||
| "geoId/11" | ||
| "geoId/24" | ||
| "geoId/04" | ||
| "geoId/20" | ||
| "geoId/02" | ||
| "geoId/25" | ||
| "geoId/06" | ||
| "geoId/08" | ||
| "geoId/19" | ||
| "geoId/16" | ||
| "geoId/22" | ||
| "geoId/13" | ||
| "geoId/15" | ||
| "geoId/34" | ||
| "geoId/35" | ||
| "geoId/49" | ||
| "geoId/33" | ||
| "geoId/48" | ||
| "country/USA" | ||
| "geoId/12" | ||
| "geoId/47" | ||
| "geoId/23" | ||
| "geoId/10" | ||
| "geoId/26" |
215 changes: 215 additions & 0 deletions
215
scripts/us_eia/opendata/golden_data/golden_observations_EIA_International.csv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,215 @@ | ||
| "place" | ||
| "country/ARM" | ||
| "country/CUB" | ||
| "country/HUN" | ||
| "southamerica" | ||
| "country/NPL" | ||
| "country/NOR" | ||
| "country/PER" | ||
| "country/TUR" | ||
| "country/BWA" | ||
| "country/ECU" | ||
| "country/DOM" | ||
| "country/DMA" | ||
| "country/LBY" | ||
| "country/LBR" | ||
| "country/CRI" | ||
| "country/HTI" | ||
| "country/AGO" | ||
| "country/AFG" | ||
| "country/GIN" | ||
| "country/GNB" | ||
| "country/IND" | ||
| "country/CAN" | ||
| "country/LCA" | ||
| "country/GNQ" | ||
| "country/IRQ" | ||
| "country/DNK" | ||
| "country/ZMB" | ||
| "country/ZAF" | ||
| "country/TJK" | ||
| "country/TCD" | ||
| "country/TCA" | ||
| "country/PYF" | ||
| "country/CYM" | ||
| "country/CHE" | ||
| "country/LKA" | ||
| "country/USA" | ||
| "country/PNG" | ||
| "country/GUY" | ||
| "country/HKG" | ||
| "country/PRK" | ||
| "country/TWN" | ||
| "country/COM" | ||
| "country/DZA" | ||
| "country/UKR" | ||
| "country/SVK" | ||
| "country/ZWE" | ||
| "country/WSM" | ||
| "country/SYC" | ||
| "country/SVN" | ||
| "country/BMU" | ||
| "country/BRN" | ||
| "country/RWA" | ||
| "country/EGY" | ||
| "country/VUT" | ||
| "country/ARE" | ||
| "country/COD" | ||
| "country/NER" | ||
| "country/HND" | ||
| "country/MUS" | ||
| "country/COG" | ||
| "country/NZL" | ||
| "country/VGB" | ||
| "country/PAK" | ||
| "country/VEN" | ||
| "country/ALB" | ||
| "country/VIR" | ||
| "country/BGD" | ||
| "country/BFA" | ||
| "country/IDN" | ||
| "country/JAM" | ||
| "country/GAB" | ||
| "country/JOR" | ||
| "country/MNG" | ||
| "country/SAU" | ||
| "country/KAZ" | ||
| "country/CMR" | ||
| "country/THA" | ||
| "country/RUS" | ||
| "country/IRN" | ||
| "country/STP" | ||
| "country/MEX" | ||
| "country/ISR" | ||
| "country/SWE" | ||
| "country/BDI" | ||
| "country/MDV" | ||
| "country/ASM" | ||
| "country/MLI" | ||
| "country/MLT" | ||
| "country/MMR" | ||
| "country/TUN" | ||
| "country/CPV" | ||
| "country/MYS" | ||
| "country/NAM" | ||
| "country/NCL" | ||
| "country/SDN" | ||
| "country/SEN" | ||
| "country/UGA" | ||
| "country/GUF" | ||
| "country/GRL" | ||
| "country/SUR" | ||
| "country/LSO" | ||
| "country/NIC" | ||
| "country/LAO" | ||
| "country/BTN" | ||
| "northamerica" | ||
| "country/FRA" | ||
| "country/KNA" | ||
| "country/DJI" | ||
| "country/YEM" | ||
| "country/HRV" | ||
| "country/GTM" | ||
| "country/FJI" | ||
| "country/BOL" | ||
| "country/CIV" | ||
| "Earth" | ||
| "country/CHN" | ||
| "country/KGZ" | ||
| "africa" | ||
| "country/ABW" | ||
| "country/AUT" | ||
| "country/ESH" | ||
| "country/BGR" | ||
| "country/TLS" | ||
| "country/BHS" | ||
| "country/EST" | ||
| "country/QAT" | ||
| "country/PAN" | ||
| "country/PHL" | ||
| "country/AZE" | ||
| "country/BLR" | ||
| "country/IRL" | ||
| "country/MDG" | ||
| "country/PRY" | ||
| "country/AUS" | ||
| "country/BRA" | ||
| "country/BEL" | ||
| "country/LVA" | ||
| "country/MOZ" | ||
| "country/NLD" | ||
| "country/CHL" | ||
| "country/TZA" | ||
| "country/BRB" | ||
| "country/LBN" | ||
| "country/MRT" | ||
| "country/REU" | ||
| "country/ITA" | ||
| "country/FIN" | ||
| "country/VCT" | ||
| "country/DEU" | ||
| "country/KWT" | ||
| "country/GHA" | ||
| "country/GIB" | ||
| "country/GRD" | ||
| "country/SRB" | ||
| "country/SOM" | ||
| "country/MAR" | ||
| "europe" | ||
| "country/ATG" | ||
| "country/LUX" | ||
| "country/ESP" | ||
| "country/KHM" | ||
| "country/MKD" | ||
| "country/MTQ" | ||
| "country/CAF" | ||
| "country/NGA" | ||
| "country/GRC" | ||
| "country/KIR" | ||
| "country/OMN" | ||
| "country/SLV" | ||
| "country/ERI" | ||
| "country/SLE" | ||
| "country/SGP" | ||
| "country/GUM" | ||
| "country/COL" | ||
| "country/ETH" | ||
| "country/PRT" | ||
| "country/BLZ" | ||
| "country/VNM" | ||
| "country/BEN" | ||
| "country/ARG" | ||
| "country/GEO" | ||
| "country/SYR" | ||
| "country/CYP" | ||
| "country/UZB" | ||
| "country/LTU" | ||
| "country/PSE" | ||
| "country/TGO" | ||
| "country/TTO" | ||
| "country/URY" | ||
| "country/MDA" | ||
| "country/TON" | ||
| "country/POL" | ||
| "country/COK" | ||
| "country/KOR" | ||
| "country/JPN" | ||
| "country/GLP" | ||
| "country/SLB" | ||
| "country/MNE" | ||
| "country/ISL" | ||
| "country/FRO" | ||
| "country/GBR" | ||
| "country/SWZ" | ||
| "country/BHR" | ||
| "country/ROU" | ||
| "country/BIH" | ||
| "country/MAC" | ||
| "country/MWI" | ||
| "country/PRI" | ||
| "country/TKM" | ||
| "country/KEN" | ||
| "country/GMB" | ||
| "country/CZE" | ||
| "country/FSM" | ||
| "country/MNP" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.