Skip to content

test: fix failing kitchen suites by pinning salt to the image version#591

Merged
dafyddj merged 2 commits into
saltstack-formulas:masterfrom
propertyguru:fix-kitchen-salt-version-pin
Jun 26, 2026
Merged

test: fix failing kitchen suites by pinning salt to the image version#591
dafyddj merged 2 commits into
saltstack-formulas:masterfrom
propertyguru:fix-kitchen-salt-version-pin

Conversation

@pault-pg

Copy link
Copy Markdown
Contributor

I opened #590 to fix a one-line typo, but its GitLab CI jobs failed — and
other PRs (e.g. #589) fail with the same error. So the kitchen suites seem to
be failing independently of any change; this PR is an attempt to fix that.

The *-3006* / *-3007* suites die on every platform while applying the states:

'salt' changed from '3006.17-0' to '3008.1-0'
[ERROR ] ModuleNotFoundError: spec not found for the module 'site'
>>>>>> Converge failed on <default-oraclelinux-9-3006-17>

Those images ship salt 3006.17 / 3007.9, but salt.pkgrepo adds the
salt-repo-latest repo (currently 3008.1) and the unpinned pkg.installed
upgrades salt in place — swapping the onedir under the running salt-call,
which breaks the post-install module refresh.

The per-version pins in test/salt/pillar/ only cover 3002/3003/3004 and set
salt:release, which no state reads anymore — so the current images get no
pin. This pins salt:version to the image's running salt (grains.saltversion)
instead, so the formula keeps it rather than upgrading.

I couldn't run kitchen locally, so this leans on the PR's CI matrix to validate.
(The stale v3002/3/4-py3.sls release pins are dead now and could be dropped
in a follow-up.)

Disclosure: prepared with the assistance of Claude (an LLM).

On the *-3006* / *-3007* suites, kitchen fails on every platform while
applying the salt states:

    'salt' changed from '3006.17-0' to '3008.1-0'
    [ERROR ] ModuleNotFoundError: spec not found for the module 'site'
    >>>>>> Converge failed on <default-oraclelinux-9-3006-17>

The images ship salt 3006.17/3007.9, but salt.pkgrepo adds the
salt-repo-latest repo (saltproject-rpm/, currently 3008.1) and the unpinned
pkg.installed upgrades salt in place; swapping the running onedir under
salt-call breaks the post-install module refresh, so the run dies.

The per-version pins in test/salt/pillar/ only cover 3002/3003/3004 and set
salt:release, which is no longer read by any state. Pin salt:version to the
image's running version instead, so the formula keeps the installed salt.

Co-authored-by: Claude <noreply@anthropic.com>
@pault-pg pault-pg force-pushed the fix-kitchen-salt-version-pin branch from 8b50270 to 5c894b9 Compare June 25, 2026 15:16
@pault-pg

Copy link
Copy Markdown
Contributor Author

Update after watching the matrix: this pin does get past the crash above — converge completes and salt stays at the image's 3006.17 (no in-place onedir upgrade). But it trades that for one inspec failure:

×  salt._mapdata: `map.jinja` should match the reference file
   -    version: ''
   +    version: '3006.17'

Pinning salt:version writes the running version into the computed map, but test/integration/default/files/_mapdata/<os>.yaml has version: '', and that one fixture per OS is shared by the -3006 / -3007 / -master suites — so it can't hold a per-version value. Pinning salt:version therefore can't satisfy _mapdata here, whatever value it takes.

Rather than force a dead end, I'd rather ask how you'd prefer to handle it. Two directions I can see:

  1. keep the pin and have the _mapdata control ignore version, or
  2. don't pin salt:version — instead keep the suites from layering the "latest" repo over the image's pinned one, so pkg.installed stays on the image version and the map stays version: ''.

I'll open a separate PR trying option 2 (it's a state_top change, not a test-pillar one). Whatever you'd recommend works — thanks for maintaining this.

Pinning salt:version (previous commit) writes the running version into
salt_settings.version, but the _mapdata reference files are static and one
per OS — shared across the -3006/-3007/-master suites — so they can't carry a
per-image version. Drop salt_settings.version from both sides of the compare:
it now tracks the kitchen image's salt rather than a fixed default, so it's
not a value the static reference should assert.

Co-authored-by: Claude <noreply@anthropic.com>
@pault-pg pault-pg requested a review from a team as a code owner June 25, 2026 16:40
@pault-pg

Copy link
Copy Markdown
Contributor Author

Update — I went with the first option above. Two commits:

  1. pin salt:version to the kitchen image's salt ({{ grains.saltversion }}), so pkg.installed stops upgrading salt to "latest" in place — which was swapping the running onedir and crashing the post-install module refresh;
  2. have the salt._mapdata control drop values.salt_settings.version before its compare — the pin makes that field track the image rather than the fixed '' default, so it's no longer something the static per-OS reference should assert.

On the latest pipeline all 34 blocking -3006/-3007 suites pass (0 blocking failures). The remaining red checks are all allow_failure and unrelated to this change:

  • the -master suites fail in salt/utils/requisite.py _get_chunk_order with TypeError: '<' not supported between instances of 'str' and 'float' — a salt-master regression around order: last, not specific to this formula;
  • amazonlinux-2-3006/-3007 fail at kitchen create with SSH session could not be established — container flakiness, the converge never runs.

I also closed #592 (the "skip salt.pkgrepo" idea): it's disproven, since the image's own repo serves "latest" and dropping salt.pkgrepo didn't stop the upgrade.

Thanks for maintaining this!

@dafyddj dafyddj merged commit 4aa332d into saltstack-formulas:master Jun 26, 2026
26 checks passed
@dafyddj

dafyddj commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Thank you for taking a look at this - it was on the to-do list to get to eventually.

Great first contibution!

@saltstack-formulas-github

Copy link
Copy Markdown

🎉 This PR is included in version 1.13.3 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@tobiaspal

tobiaspal commented Jun 27, 2026

Copy link
Copy Markdown

Correction to my note above on the -master suites — looking closer, they don't all fail the same way:

  • The order: last TypeError is saltstack/salt#67120, fixed upstream in June 2025. Only ubuntu-2404-master still hits it — its salt-master-py3:ubuntu-24.04 image is from Nov 2024, before that fix.
  • The other -master suites fail on something else: the image runs a master build (e.g. 3007.9+636.ge3c222b77), but the formula installs from the "Salt LATEST release" repo, which has no package at that version, so salt-master and salt-minion pkg.installed both get No match. (My test: fix failing kitchen suites by pinning salt to the image version #591 pin makes it that exact build version; unpinned, it would instead install the latest release over the master build — the in-place upgrade the pin avoids — so it fails either way.)

Separately worth flagging: even the freshest -master images were built back in Nov–Dec 2025, ~6 months ago — so these suites are not close to master.

So as long as the -master images run a master build with no matching package in a release repo, those suites can't install salt — for them to pass, they'd need to install from a nightly/master repo. Both that and the staleness are image/CI-template matters, not the formula. Flagging in case it's useful — happy to help if you'd like to pursue it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants