JIT: regression in compare simplification rewriting `~x op k` without inverting the comparison

[AndyAyersMS]

Note

This issue was authored by GitHub Copilot CLI on @AndyAyersMS's machine,
based on a bug surfaced by an experimental in-development fuzzer.
The C# repro and disassembly are verified; the hypothesis about the
offending phase is informed speculation.

Repro

using System;
using System.Runtime.CompilerServices;

public static class Program
{
    private static volatile int Input_p0 = unchecked((int)0x800335C5);
    private static volatile int Input_p1 = 0;
    private static volatile int Sink;

    [MethodImpl(MethodImplOptions.NoInlining)]
    public static int Func(int p0, int p1)
    {
        unchecked
        {
            int v1 = unchecked((int)0x000335C5) & p1;
            int v3 = (v1 != p0) ? 1 : 0;
            if (v3 == 0) return 99;
            int v4 = ~v3;
            return (v4 >= -1) ? 1 : 0;
        }
    }
    public static int Main()
    {
        Console.WriteLine($"{(uint)Func(Input_p0, Input_p1):X8}");
        return 0;
    }
}

Manual trace: p0 = 0x800335C5, p1 = 0 ⇒ v1 = 0, v3 = (0 != p0) ? 1 : 0 = 1, branch not taken, v4 = ~1 = -2, (-2 >= -1) ? 1 : 0 = 0. Expected: 0.

Observed

Configuration	Output	Correct?
DOTNET_JITMinOpts=1	00000000	✅
DOTNET_TieredCompilation=1 DOTNET_TC_QuickJitForLoops=1 (Tier1 via tiering)	00000000	✅
DOTNET_TieredCompilation=0 (FullOpts direct)	00000001	❌
Shipping dotnet 11.0.100-preview.5.26227.104 (any config)	00000000	✅

Repros on dotnet/runtime at HEAD a8b2c92ce21 (2026-06-05) on osx-arm64 Checked. Does not repro on preview-5 — appears to be a regression introduced since then.

Disassembly (osx-arm64 Checked, DOTNET_TieredCompilation=0)

; Assembly listing for method Program:Func(int,int):int (FullOpts)
G_M26094_IG02:
    movz    w2, #0x35C5
    movk    w2, #3 LSL #16     ;; w2 = 0x000335C5
    and     w1, w1, w2         ;; v1 = 0x335C5 & p1
    cmp     w1, w0             ;; (v1 == p0)?
    cset    x0, ne             ;; v3 = (v1 != p0) ? 1 : 0
    mov     w1, #1
    mov     w2, #99
    cmp     w0, #0             ;; v3 == 0?
    csel    w0, w1, w2, ne     ;; return (v3 != 0) ? 1 : 99      <-- WRONG

The whole post-branch return has been folded to (v3 != 0) ? 1 : 99. The correct fold would be (v3 != 0) ? 0 : 99 — i.e., the THEN constant should be 0, not 1.

For comparison, MinOpts emits the actual mvn/cmn/cset ge sequence and returns the correct value.

Hypothesis

After if (v3 == 0) return 99, the JIT knows v3 ∈ {1} (since v3 is a compare result with range {0, 1} and the branch narrowed it). It then evaluates (~v3 >= -1):

• Correct: (~v3 >= -1) ⇔ (v3 <= 0) (complementing both sides reverses the comparison). With v3 ∈ {1} this is false, so the THEN constant is 0.
• What the JIT seems to do: (~v3 >= -1) ⇔ (v3 >= 0) (forgot to invert the comparison direction). With v3 ∈ {1} (or any non-negative range) this is true, so the THEN constant is 1.

Looks like a peephole / fgOptimizeRelationalComparisonWithCasts-style rewrite of ~x op k to x op ~k that missed reversing the comparison sign for ordered (non-EQ/NE) compares.

Candidate offending PRs

All merged between preview-5 and HEAD a8b2c92ce21:

• JIT: Enable fgOptimizeRelationalComparisonWithCasts for EQ/NE #128091 — JIT: Enable fgOptimizeRelationalComparisonWithCasts for EQ/NE (2026-06-01)
• JIT: Unify post-order cmp transforms into fgOptimizeRelationalComparison #127883 — JIT: Unify post-order cmp transforms into fgOptimizeRelationalCompari… (2026-05-21)
• Update fgMorphSmpOp to canonicalize commutative and compare ops at the start of POST-ORDER #127661 — Update fgMorphSmpOp to canonicalize commutative and compare ops… (2026-05-02)
• JIT: relop simplification during redundant branch opts #127181 — JIT: relop simplification during redundant branch opts (2026-04-24)

How it was found

ReifyCs (a from-scratch implementation of the PLDI 2026 semantic-reification
technique adapted to C#) ran a 500-trial, 4-type, 7-config differential
campaign and reported exactly one mismatch. Manual reduction took the
generated method down to the 5-line body above.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.