CANOpenNode slows down after quitting debugging session

[zaaynn123]

The issue appears when two different devices are running CANopenNode on the same CAN bus.
I have observed that the problem occurs most frequently during:

• LSS-based scanning and node assignment
• SDO communication

Problem behavior

Initially, it seemed that the device not being debugged was responding late. However, after further investigation, I have confirmed that the issue actually originates from the device being debugged

After debugging session is stopped while the CANopen stack is running, the following behavior is observed:

• The debugged device starts responding very slowly or incorrectly
• SDO communication fails
• LSS scanning and node assignment become unreliable
• SDO abort errors are observed (e.g., 0x06040043 – parameter incompatibility)

Once this condition occurs:

• Retrying the request does not recover communication
• The system remains stuck in this faulty state

---

Important observation

• The issue is triggered specifically when:

  • Debugging during CANopen execution
  • Using step-over / line-by-line debugging

• The issue does NOT occur if:

  • Debugging is performed before the CANopen stack starts

• This problem do not occur currently when debugging session is not done for now.

---

Recovery methods

The only ways to recover are:

1. Performing a full device reset, after which communication works again
2. Performing a CANopen communication reset, which also restores operation
3. Sometimes increasing the timeout of the SDO (time_difference_us and SDOtimeoutTime_ms) resolves the issue but the system and SDO become very slow and eventually fail as well. I am following the SDO code given here : https://canopennode.github.io/CANopenNode/group__CO__SDOclient.html

However, this issue should ideally not occur in the first place.

---

How to reproduce

1. Connect two CANopenNode devices on the same bus
2. Start normal CANopen communication
3. Begin debugging one device
4. Step through the firmware (step-over) while CANopen is running
5. Stop the debugging session
6. Now the SDO communication or LSS communication must start to let the problem occur. Also try to call them consecutively as well in a loop.

After this, the communication problem appears consistently once it starts occurring.

---

Similar issue

My issue seems very similar to the one reported here:
#89

---

Screenshots

SDO

Overall :

SDO request:

SDO response:

Abort SDO:

LSS

LSS Timed out after this:

Request

I would appreciate any guidance on how to resolve this issue, as I am currently blocked by it.
I would be happy to provide any additional information if required.

[HamedJafarzadeh]

Hi @zaaynn123, Thanks for reporting this issue. However, I wasn't able to reproduce the issue.

• I have a STM32F0 and STM32G0 boards, I programmed both to the latest commits.
• Once everything is up and running.
• Both nodes are very responsive to SDO calls to read Device Type
• I added a breakpoint in one of the "canopen_app_process" lines on STM32F0 board.
• I step over few lines and then "Terminate" the debugging session.
• I checked SDO calls and they were both very responsive. I did over 20 SDO Read on the STM32F0 node and everything was smooth and responsive.
• I also tried to "Continue" the debug session without breakpoints and still the same result. Didn't see any issue.

If you can provide any other evidence, please do. I would be interested in fixing this issue.

[Frab012]

I can confirm this issue.

I have 6 nodes, all with STM32G0. I can trigger the slow response time on any device by doing a line-by-line debugging session over ST-LINK.

The issue does not disappear when I reset the canopen stack (CO_RESET_COMM) or the node (CO_RESET_APP), regardless of whether the ST-LINK connection was terminated or not.

Only power cycling solves the problem. That's why I think it's a hardware problem.

[MaJerle]

I try to understand your point. If I go line by line

I have 6 nodes, all with STM32G0. I can trigger the slow response time on any device by doing a line-by-line debugging session over ST-LINK.

Did you try and if not, could you please, the latest repository where we fixed one low-level bug with FDCAN buffer management?

The issue does not disappear when I reset the canopen stack (CO_RESET_COMM) or the node (CO_RESET_APP), regardless of whether the ST-LINK connection was terminated or not.

Only power cycling solves the problem. That's why I think it's a hardware problem.

If you are able to fix everything after resetting the canopen, I wouldn't be surprised the issue is fixed with the low-level driver fixed. This is the error: 3dce94f

[Frab012]

Did you try and if not, could you please, the latest repository where we fixed one low-level bug with FDCAN buffer management?

Yes, I did.

If you are able to fix everything after resetting the canopen, I wouldn't be surprised the issue is fixed with the low-level driver fixed.

No, unfortunately not. Only power cycling helps.

[Frab012]

I did some logging. Baudrate 50kBit.

Logging with delta timestamp:

candump -c can0 -td | tee dump.log

With debugging:

cocomm  "13 r 0x2000 1"
[0] 03
cocomm  "13 r 0x2000 1"
[0] ERROR:0x05040000 #SDO protocol timed out.
cocomm  "13 r 0x2000 1"
[0] 03
cocomm  "13 r 0x2000 1"
[0] ERROR:0x05040000 #SDO protocol timed out.

 (006.377457)  can0  60D   [8]  40 00 20 01 00 00 00 00
 (000.003351)  can0  58D   [8]  4F 00 20 01 03 00 00 00
 (001.621051)  can0  60D   [8]  40 00 20 01 00 00 00 00
 (000.499845)  can0  60D   [8]  80 00 20 01 00 00 04 05
 (000.002675)  can0  58D   [8]  4F 00 20 01 03 00 00 00
 (006.201282)  can0  60D   [8]  40 00 20 01 00 00 00 00
 (000.003051)  can0  58D   [8]  4F 00 20 01 03 00 00 00
 (001.412688)  can0  60D   [8]  40 00 20 01 00 00 00 00
 (000.499408)  can0  60D   [8]  80 00 20 01 00 00 04 05
 (000.003786)  can0  58D   [8]  4F 00 20 01 03 00 00 00

Without debugging:

cocomm  "13 r 0x2000 1"
[0] 03
cocomm  "13 r 0x2000 1"
[0] 03
cocomm  "13 r 0x2000 1"
[0] 03
cocomm  "13 r 0x2000 1"
[0] 03

 (000.697916)  can0  60D   [8]  2B 17 10 00 00 00 00 00
 (000.003026)  can0  58D   [8]  60 17 10 00 00 00 00 00
 (001.303042)  can0  081   [8]  30 81 10 1B 01 00 00 00
 (006.198678)  can0  60D   [8]  40 00 20 01 00 00 00 00
 (000.003565)  can0  58D   [8]  4F 00 20 01 03 00 00 00
 (000.900102)  can0  60D   [8]  40 00 20 01 00 00 00 00
 (000.003718)  can0  58D   [8]  4F 00 20 01 03 00 00 00
 (000.804904)  can0  60D   [8]  40 00 20 01 00 00 00 00
 (000.003436)  can0  58D   [8]  4F 00 20 01 03 00 00 00
 (000.795506)  can0  60D   [8]  40 00 20 01 00 00 00 00
 (000.003349)  can0  58D   [8]  4F 00 20 01 03 00 00 00

[MaJerle]

Performing a CANopen communication reset, which also restores operation

I would be interesting if just resetting FDCAN low-level (nothing else, just low-level peripheral) solves the issue.

[CANopenNode]

I also suggest exploring with CANopen communication reset.

[zaaynn123]

@CANopenNode @MaJerle @HamedJafarzadeh

A CANopen communication reset does fix the problem, but it is not a good solution due to re-initialization of the whole stack, making it very hard to monitor states when the CANopen stack starts hanging until the reset occurs.

How to reproduce:

Two devices are required where one will be debugged and the other sending packets targeting the debugged device for example the debugged device will be a heartbeat consumer and will getting heartbeat packets continuously while being in debugged mode, this is likely why @HamedJafarzadeh was not able to reproduce it if he did not tried it like that way. One device needs to be in a debug/halted state while the other continuously sends packets including heartbeat messages to the halted device.
What I debugged:
The workaround that solved the problem was flushing the RX FIFO and clearing the stuck interrupt flags:

void canopen_flush_rxfifo(void) {
    uint8_t fill = FDCAN1->RXF0S & 0x7F;
    if (fill > 0) {
        printf("Flushing %d stuck frames from RX FIFO\n", fill);
        while ((FDCAN1->RXF0S & 0x7F) > 0) {
            uint8_t get_idx = (FDCAN1->RXF0S >> 8) & 0x3F;
            FDCAN1->RXF0A = get_idx;
        }
    }
    /* Clear ALL RX-related IR flags — RF0F(bit2), RF0L(bit3), RF0N(bit0) */
    FDCAN1->IR = FDCAN_IR_RF0F | FDCAN_IR_RF0L | FDCAN_IR_RF0N;
    printf("IR flags cleared. IR now: 0x%08lX\n", FDCAN1->IR);
}

/* In main loop — detects and clears sticky IR flags */
if (FDCAN1->IR & (FDCAN_IR_RF0F | FDCAN_IR_RF0L)) {
    printf("WARNING: Sticky IR flag detected: 0x%08lX — clearing\n",
        FDCAN1->IR);
    canopen_flush_rxfifo();
}

When one device is halted in a debug session and the other device continuously sends packets including heartbeat messages, at a certain point the whole CANopen stack on the halted device slows down and only recovers when the RX interrupt flags are cleared and the RX FIFO is flushed as shown above. This is consistent with issue #89 where the same solution fixed the same problem.
This was only a workaround while waiting for a real fix, which is why I am raising it here.

[MaJerle]

Do you see any bus-off or any other errors if you check the HAL Protocol Status check?

Overall, I am surprised that the interrupt would not fire if the FIFO is not empty. Unless if there was a RX Error counter which may have put device in the bus-off state.

I don't have device to test so it is just an assumption.

[zaaynn123]

@MaJerle

I am not fully sure but it might be because there are two flags missing in HAL_FDCAN_ActivateNotification that are causing my errors.

In the stack I am not seeing FDCAN_IT_RX_FIFO0_FULL and FDCAN_IT_RX_FIFO0_MESSAGE_LOST as seen below:

I made a debug dump function while testing:

void canopen_debug_dump(void) {
    printf("========= Debug Dump =========\n");
    printf("NMT state     : %d\n", CO->NMT->operatingState);
    printf("CANnormal     : %d\n", CO->CANmodule->CANnormal);
    printf("CANerrorStatus: 0x%04X\n", CO->CANmodule->CANerrorStatus);
    printf("RXF0 fill     : %lu\n", FDCAN1->RXF0S & 0x7F);
    printf("RXF0 lost     : %lu\n", (FDCAN1->RXF0S >> 25) & 1);
    printf("RXF0 get_idx  : %lu\n", (FDCAN1->RXF0S >> 8)  & 0x3F);
    printf("RXF0 put_idx  : %lu\n", (FDCAN1->RXF0S >> 16) & 0x3F);
    printf("FDCAN IE      : 0x%08lX\n", FDCAN1->IE);
    printf("FDCAN ILE     : 0x%08lX\n", FDCAN1->ILE);
    printf("FDCAN IR      : 0x%08lX\n", FDCAN1->IR);
    printf("FDCAN PSR     : 0x%08lX\n", FDCAN1->PSR);
    printf("FDCAN ECR     : 0x%08lX\n", FDCAN1->ECR);
    printf("==============================\n");
}

What was interesting is that FDCAN_IR changed from 0x00000000 to 0x0000000C or 0x00000004 after a debug halt. I believe the two bits being set correspond to FDCAN_IT_RX_FIFO0_FULL and FDCAN_IT_RX_FIFO0_MESSAGE_LOST. Since these are missing from HAL_FDCAN_ActivateNotification, HAL_FDCAN_IRQHandler never automatically clears them. When they remain set, the entire CANopen system slows down significantly and almost every request fails due to timeouts.

Manually flushing the RX FIFO and clearing these two bits resolved the issue and restored normal CANopen speed. However I think there should be a proper way to handle a full RX FIFO and message loss at the driver level, since some data may be critical and silently discarding frames is not ideal.

FYI: @CANopenNode @HamedJafarzadeh

[Frab012]

@MaJerle

I would be interesting if just resetting FDCAN low-level (nothing else, just low-level peripheral) solves the issue.

You are right. My mistake was resetting the device using cocomm "0 reset comm" instead of addressing the device directly, so only my ascii gateway node was reset.