UCT/ROCM: fix dmabuf support check for coreos by hahahannes · Pull Request #11541 · openucx/ucx

hahahannes · 2026-06-09T15:04:32Z

What?

Instead of only checking /boot/config-%s, it will check also other locations that are used by other OS.

Why?

On Fedora CoreOS and possibly other OS, /boot/config-%s does not exist. Instead they use other locations. See #11540 for more.

How?

Instead of checking a single path, loop over a list of locations.
The current implementation is missing "/proc/config.gz". For now I left out the gz handling to keep it simple.

This is based on the help from @edgargabriel who helped us to identify the root cause and pointed us to the implementation of ROCM (https://github.com/ROCm/rocm-systems/blob/develop/projects/rccl/src/misc/rocmwrap.cc#L282)

hahahannes · 2026-06-10T07:16:45Z

We are patching this as well in the linux-rdma package as this happens as well with the ib_write_bw tests. See linux-rdma/perftest#392

hahahannes · 2026-06-10T09:18:30Z

We noticed that uname -r inside a container reports the host kernel and not the container kernel version. Therefor it won't match the file in e.g. /usr/lib/modules .

hahahannes · 2026-06-10T10:49:25Z

We noticed that uname -r inside a container reports the host kernel and not the container kernel version. Therefor it won't match the file in e.g. /usr/lib/modules .

We fixed by checking /proc/kallsyms as done in ROCM (https://github.com/ROCm/rocm-systems/blob/develop/projects/rccl/src/misc/rocmwrap.cc#L367)

hahahannes · 2026-06-10T10:56:36Z

We validated the fix on our testbed based on a kubernetes cluster with ROCM 7.1, AMD Radeon W7900 GPUs and a Broadcom RoCEv2 NIC.

../configure \
        --prefix=/usr/local/ucx \
        --with-rocm=/opt/rocm \
        --with-go=no \
        --with-verbs \
        --without-java \
        --disable-doxygen-doc \
        --enable-optimizations \
        --enable-mt \
        --disable-debug \
        --disable-logging \
        --disable-assertions \
        --disable-params-check \
        --disable-dependency-tracking \
        --enable-cma \
        --with-rc \
        --with-ud \
        --with-dc \
        --with-mlx5-dv \
        --with-ib-hw-tm \
        --with-dm \
        --with-avx \
        --without-cm \
        --with-rdmacm

and

HSA_FORCE_FINE_GRAIN_PCIE=1 UCX_ROCM_COPY_D2H_THRESH=0 UCX_ROCM_COPY_DMABUF=yes  HIP_VISIBLE_DEVICES=0 UCX_IB_GPU_DIRECT_RDMA=yes UCX_TLS=rc,rocm,cuda ucx_perftest $ROCE_HOST -t ucp_am_bw -m rocm -s 2,4096,1048576,8388608

Resulting in nearly the expected bandwidth performance:

+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[thread 0]              2593      0.780   385.606   385.606    23350.03   23350.03        2593        2593
[thread 0]              5154      0.770   391.233   388.402    23014.17   23181.93        2556        2575
[thread 0]              7746      0.780   386.703   387.833    23283.79   23215.91        2586        2578
[thread 0]             10338      0.750   386.702   387.550    23283.83   23232.91        2586        2580

hahahannes · 2026-06-10T11:36:20Z

The gz handling was added as well. Highly inspired by the ROCM implementation. So kudos to the team!

hahahannes · 2026-06-10T12:35:35Z

Maybe it's worth to add Fedora CoreOS and RHEL bootc to the ci jobs as well.

guy-ealey-morag · 2026-06-12T13:41:48Z

The PR description says that it doesn't include gz handling, but the code is included in the PR.
Is it intentional?

guy-ealey-morag · 2026-06-12T13:38:30Z

+            fp = popen("zcat /proc/config.gz 2>/dev/null", "r");
+          } else {
+            fp = fopen(kernel_conf_file, "r");


Notice that here fp is opened either by popen or fopen, but always closed with fclose.
When using popen the closing function must be pclose.

Thanks! Should be better handled in the latest commit.

hahahannes · 2026-06-12T14:45:02Z

The PR description says that it doesn't include gz handling, but the code is included in the PR. Is it intentional?

Hi @guy-ealey-morag, thanks a lot for having a look! I implemented it after opening the PR. So yes intentional!
Will implement your requested changes.

guy-ealey-morag · 2026-06-12T15:48:20Z

Please add prefix UCT/ROCM: to the last commit message (it fails in CI)

guy-ealey-morag · 2026-06-12T15:50:44Z

+    /* Skip if zcat is unavailable or /proc/config.gz does not exist. */
+    /* popen() succeeds even when the file is missing, producing an empty */
+    /* stream that falsely triggers the "not found" error path. */
+    /* Check if zcat is available in the system */


Suggested change

/* Skip if zcat is unavailable or /proc/config.gz does not exist. */

/* popen() succeeds even when the file is missing, producing an empty */

/* stream that falsely triggers the "not found" error path. */

/* Check if zcat is available in the system */

/* Skip if zcat is unavailable or /proc/config.gz does not exist.

* popen() succeeds even when the file is missing, producing an empty

* stream that falsely triggers the "not found" error path.

* Check if zcat is available in the system */

guy-ealey-morag · 2026-06-12T15:52:33Z


    while (fgets(buf, sizeof(buf), fp) != NULL) {
-        if (strstr(buf, kernel_opt1) != NULL) {
+        if (!found_opt1 && (strstr(buf, kernel_opt1) != NULL || strstr(buf, kernel_sym1) != NULL)) {


Suggested change

if (!found_opt1 && (strstr(buf, kernel_opt1) != NULL || strstr(buf, kernel_sym1) != NULL)) {

if (!found_opt1 && ((strstr(buf, kernel_opt1) != NULL) || (strstr(buf, kernel_sym1) != NULL))) {

guy-ealey-morag · 2026-06-12T15:52:50Z

            found_opt1 = 1;
        }
-        if (strstr(buf, kernel_opt2) != NULL) {
+        if (!found_opt2 && (strstr(buf, kernel_opt2) != NULL || strstr(buf, kernel_sym2) != NULL)) {


Suggested change

if (!found_opt2 && (strstr(buf, kernel_opt2) != NULL || strstr(buf, kernel_sym2) != NULL)) {

if (!found_opt2 && ((strstr(buf, kernel_opt2) != NULL) || (strstr(buf, kernel_sym2) != NULL))) {

guy-ealey-morag · 2026-06-12T15:55:41Z

Build errors:

/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c: In function 'uct_rocm_base_load_prod_config_file':
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:306:23: error: 'kernel_conf_file' undeclared (first use in this function)
  306 |     ucs_snprintf_safe(kernel_conf_file, sizeof(kernel_conf_file), path, utsname.release);
      |                       ^~~~~~~~~~~~~~~~
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:306:23: note: each undeclared identifier is reported only once for each function it appears in
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:306:67: error: 'path' undeclared (first use in this function)
  306 |     ucs_snprintf_safe(kernel_conf_file, sizeof(kernel_conf_file), path, utsname.release);
      |                                                                   ^~~~
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:306:73: error: 'utsname' undeclared (first use in this function)
  306 |     ucs_snprintf_safe(kernel_conf_file, sizeof(kernel_conf_file), path, utsname.release);
      |                                                                         ^~~~~~~
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:307:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
  307 |     int has_zcat = (system("which zcat > /dev/null 2>&1") == 0);
      |     ^~~
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c: In function 'uct_rocm_base_load_kernel_info_file':
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:332:5: error: 'fp' undeclared (first use in this function)
  332 |     fp = uct_rocm_base_load_prod_config_file();
      |     ^~
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:339:27: error: 'kernel_conf_file' undeclared (first use in this function)
  339 |         ucs_snprintf_safe(kernel_conf_file, sizeof(kernel_conf_file), path, utsname.release);
      |                           ^~~~~~~~~~~~~~~~
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:339:77: error: 'utsname' undeclared (first use in this function)
  339 |         ucs_snprintf_safe(kernel_conf_file, sizeof(kernel_conf_file), path, utsname.release);
      |                                                                             ^~~~~~~
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c: In function 'uct_rocm_base_is_dmabuf_supported':
/__w/1/s/contrib/../src/uct/rocm/base/rocm_base.c:365:10: error: unused variable 'kernel_conf_file' [-Werror=unused-variable]
  365 |     char kernel_conf_file[128];
      |          ^~~~~~~~~~~~~~~~

hahahannes · 2026-06-12T16:07:42Z

@guy-ealey-morag thanks for the amazing help! You are reviewing faster than I can push my new changes. I refactored a bit more, and also fixed the fopen/popen issue.

guy-ealey-morag · 2026-06-12T16:09:37Z

Please add prefix UCT/ROCM: to the last commit message (it fails in CI)

@hahahannes To make CI not fail you need to rebase and change the commit message (all of the commits need a valid prefix), then force push.

hahahannes · 2026-06-12T18:19:35Z

@guy-ealey-morag Builds should be fixed. I also ran the test described in the issue on our testbed and I got the expected performance. The testbed is basically a kubernetes cluster with Fedora CoreOS on the hosts and AlmaLinux in the container + W7900 AMD GPU + Broadcom RoCEv2 NIC. I compiled UCX from my branch and ran with the following command:

HSA_FORCE_FINE_GRAIN_PCIE=1 UCX_ROCM_COPY_D2H_THRESH=0 UCX_ROCM_COPY_DMABUF=yes  HIP_VISIBLE_DEVICES=0 UCX_IB_GPU_DIRECT_RDMA=no UCX_TLS=rc,rocm ucx_perftest $ROCE_HOST -t ucp_am_bw -m rocm -s 2,4096,1048576,8388608

guy-ealey-morag · 2026-06-15T08:08:06Z

@hahahannes I ran the auto-formatter on the code and made some minor cleanups.
Let me know if you're okay with these changes.

hahahannes · 2026-06-15T08:13:39Z

@hahahannes I ran the auto-formatter on the code and made some minor cleanups. Let me know if you're okay with these changes.

@guy-ealey-morag Yes of course! Thank you very much!

hahahannes · 2026-06-15T08:24:49Z

I am trying to send the CLA to admin at ucfconsortium.org but my mail provider is failing to resolve the domain name. It will retry for a while. Just fyi, in case there might be an issue.

edgargabriel · 2026-06-16T15:41:10Z

I will test this branch on some of our internal systems, please hold merging the PR until I confirm that everything is good. Based on reading the code, it looks ok, but I want to ensure based on actual testing.

hahahannes · 2026-06-16T15:45:58Z

Would it be an option to include more OS in the CI test suite? It looks quite comprehensive!

guy-ealey-morag · 2026-06-16T16:08:27Z

Would it be an option to include more OS in the CI test suite? It looks quite comprehensive!

@dpressle wdyt?

edgargabriel · 2026-06-16T19:19:22Z

Would it be an option to include more OS in the CI test suite? It looks quite comprehensive!

Note that this particular code path would not be tested on a generic OS image. The issue only arises when running code with AMD GPUs + activating dmabuf. There are currently no systems with AMD GPUs in the UCX CI, we do a compilation only testing on the ROCm stack at the moment.

edgargabriel

Looks good, was able to run on two separate system, one where the old solution worked, and one where the old solution didn't work and the new one now does.

Thank you!

hahahannes · 2026-06-18T21:26:33Z

Awesome, thanks for the additional testing @edgargabriel!

edgargabriel · 2026-06-18T21:27:19Z

@guy-ealey-morag I let you merge the PR once you confirm that the contributors agreement has been received (or whatever paperwork is precisely required at this point).

brminich

waiting for CLA

hahahannes · 2026-06-22T12:46:54Z

waiting for CLA

Hi @brminich, thanks for checking! I have been trying to send it to the email specified in the readme but my mail providers (gmail and outlook) seem to fail to send due to resolution errors.

edgargabriel · 2026-06-25T12:57:33Z

Is there any news on this front? It would be good to have the PR merged.

Would it help if I am officially listed as a co-authored by , since I provided the pointer to the fix? Could the AMD CLA be used as a basis to speed things up?

hahahannes · 2026-06-25T13:02:32Z

I didn't manage to send it by email unfortunately. If we can use the AMD CLA, that would be great. The fix is basically the same as the upstream rocm implementation.

edgargabriel · 2026-06-25T13:03:46Z

I didn't manage to send it by email unfortunately. If we can use the AMD CLA, that would be great. The fix is basically the same as the upstream rocm implementation.

Can you maybe squash your commits down to a single commit and add me as a co-authored by line in the commit message?

hahahannes · 2026-06-25T13:04:11Z

Out of curiosity, would the fix make it to the next minor release and is there a timeline for this ?
Thanks again to all the maintainers!

edgargabriel · 2026-06-25T13:05:04Z

Out of curiosity, would the fix make it to the next minor release and is there a timeline for this ? Thanks again to all the maintainers!

That's one of the reason why I am pushing for it. Once its merged in the master branch, we can file a cherry-pick PR to backport it to the v1.21.x branch. In case there is a 1.21.1 release, it would be ready and available.

Co-authored-by: Edgar Gabriel <Edgar.Gabriel@amd.com>

edgargabriel · 2026-06-25T17:44:40Z

@brminich @yosefe @roiedanino given that the code is based on a solution by AMD used in RCCL, an AMD person is listed as co-author on the PR, and AMD is vouching for it, can we use the AMD CLA to get this PR merged?

hahahannes force-pushed the fix-core-os-dmabuf branch 3 times, most recently from 062a2ca to 93b2cbc Compare June 10, 2026 07:06

hahahannes force-pushed the fix-core-os-dmabuf branch 4 times, most recently from 50ca093 to e567b83 Compare June 10, 2026 08:38

hahahannes force-pushed the fix-core-os-dmabuf branch 2 times, most recently from 6ccdcec to bc979af Compare June 10, 2026 09:48

hahahannes force-pushed the fix-core-os-dmabuf branch 3 times, most recently from b4cfdf3 to 49fa660 Compare June 10, 2026 11:29

hahahannes force-pushed the fix-core-os-dmabuf branch from 49fa660 to 46b4cbe Compare June 10, 2026 11:55

hahahannes force-pushed the fix-core-os-dmabuf branch from 46b4cbe to 7596521 Compare June 12, 2026 12:55

guy-ealey-morag changed the title ~~fix dmabuf support check for coreos~~ UCT/ROCM: fix dmabuf support check for coreos Jun 12, 2026

guy-ealey-morag reviewed Jun 12, 2026

View reviewed changes

Comment thread src/uct/rocm/base/rocm_base.c Outdated

guy-ealey-morag reviewed Jun 12, 2026

View reviewed changes

guy-ealey-morag previously approved these changes Jun 15, 2026

View reviewed changes

guy-ealey-morag dismissed their stale review via 2ba16f6 June 15, 2026 08:11

guy-ealey-morag previously approved these changes Jun 15, 2026

View reviewed changes

guy-ealey-morag dismissed their stale review via 05e0009 June 15, 2026 08:22

guy-ealey-morag previously approved these changes Jun 15, 2026

View reviewed changes

edgargabriel previously approved these changes Jun 18, 2026

View reviewed changes

brminich requested changes Jun 22, 2026

View reviewed changes

hahahannes dismissed stale reviews from edgargabriel and guy-ealey-morag via 18cb298 June 25, 2026 13:11

hahahannes force-pushed the fix-core-os-dmabuf branch 2 times, most recently from 18cb298 to 2ab3109 Compare June 25, 2026 13:13

UCT/ROCM: Fix dmabuf support check for more os.

814cf1b

Co-authored-by: Edgar Gabriel <Edgar.Gabriel@amd.com>

hahahannes force-pushed the fix-core-os-dmabuf branch from 2ab3109 to 814cf1b Compare June 25, 2026 13:15

	if (!found_opt1 && (strstr(buf, kernel_opt1) != NULL \|\| strstr(buf, kernel_sym1) != NULL)) {
	if (!found_opt1 && ((strstr(buf, kernel_opt1) != NULL) \|\| (strstr(buf, kernel_sym1) != NULL))) {

	if (!found_opt2 && (strstr(buf, kernel_opt2) != NULL \|\| strstr(buf, kernel_sym2) != NULL)) {
	if (!found_opt2 && ((strstr(buf, kernel_opt2) != NULL) \|\| (strstr(buf, kernel_sym2) != NULL))) {

Uh oh!

Conversation

hahahannes commented Jun 9, 2026

What?

Why?

How?

Uh oh!

hahahannes commented Jun 10, 2026

Uh oh!

hahahannes commented Jun 10, 2026

Uh oh!

hahahannes commented Jun 10, 2026

Uh oh!

hahahannes commented Jun 10, 2026

Uh oh!

hahahannes commented Jun 10, 2026

Uh oh!

hahahannes commented Jun 10, 2026

Uh oh!

guy-ealey-morag commented Jun 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guy-ealey-morag Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

hahahannes Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hahahannes commented Jun 12, 2026

Uh oh!

guy-ealey-morag commented Jun 12, 2026

Uh oh!

guy-ealey-morag Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

guy-ealey-morag Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

guy-ealey-morag Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

guy-ealey-morag commented Jun 12, 2026

Uh oh!

hahahannes commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guy-ealey-morag commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hahahannes commented Jun 12, 2026

Uh oh!

guy-ealey-morag commented Jun 15, 2026

Uh oh!

hahahannes commented Jun 15, 2026

Uh oh!

hahahannes commented Jun 15, 2026

Uh oh!

edgargabriel commented Jun 16, 2026

Uh oh!

hahahannes commented Jun 16, 2026

Uh oh!

guy-ealey-morag commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edgargabriel commented Jun 16, 2026

Uh oh!

edgargabriel left a comment

Choose a reason for hiding this comment

Uh oh!

hahahannes commented Jun 18, 2026

Uh oh!

edgargabriel commented Jun 18, 2026

Uh oh!

brminich left a comment

Choose a reason for hiding this comment

hahahannes commented Jun 12, 2026 •

edited

Loading

guy-ealey-morag commented Jun 12, 2026 •

edited

Loading

guy-ealey-morag commented Jun 16, 2026 •

edited

Loading

edgargabriel commented Jun 25, 2026 •

edited

Loading