Thunderbolt controller fails after suspend on 6.12.10

[thomas-zimmerman]

Customer reports that Thunderbolt devices stop working after a suspend once 6.12.10 arrived.

Positive cases found on serw13 and oryp12 systems. Has been found on both Pop!OS and Ubuntu installs that have the 6.12.10 kernel.

[jacobktm]

I've been able to replicate this behavior on our serw13 and oryp12.

When I suspend either system with no thunderbolt device connected I see this:

May 05 11:56:05 pop-os kernel: ------------[ cut here ]------------
May 05 11:56:05 pop-os kernel: xhci_hcd 0000:0e:00.0: disabling already-disabled device
May 05 11:56:05 pop-os kernel: WARNING: CPU: 0 PID: 240 at drivers/pci/pci.c:2269 pci_disable_device+0xac/0xc0
May 05 11:56:05 pop-os kernel: Modules linked in: ccm rfcomm snd_seq_dummy snd_hrtimer
May 05 11:56:05 pop-os kernel: pcieport 0000:03:00.0: Unable to change power state from D3hot to D0, device inaccessible
May 05 11:56:05 pop-os kernel:  cmac
May 05 11:56:05 pop-os kernel:  algif_hash algif_skcipher af_alg nvidia_uvm(OE) bnep zram 842_decompress 842_compress lz4hc_compress lz4_compress snd_sof_pci_intel_tgl snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel soundwire_cadence snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda_mlink snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_hda_codec_realtek snd_sof snd_hda_codec_generic snd_sof_utils snd_soc_acpi_intel_match snd_hda_scodec_component soundwire_generic_allocation snd_soc_acpi soundwire_bus snd_soc_avs snd_soc_hda_codec snd_hda_ext_core intel_rapl_msr snd_hda_codec_hdmi snd_soc_core snd_compress ac97_bus intel_rapl_common snd_pcm_dmaengine iwlmvm snd_hda_intel intel_uncore_frequency intel_uncore_frequency_common snd_intel_dspcfg intel_tcc_cooling snd_intel_sdw_acpi snd_hda_codec x86_pkg_temp_thermal nvidia_drm(OE) snd_hda_core intel_powerclamp snd_hwdep mac80211 nvidia_modeset(OE) coretemp snd_pcm uvcvideo videobuf2_vmalloc uvc kvm_intel snd_seq_midi snd_seq_midi_event videobuf2_memops
May 05 11:56:05 pop-os kernel:  snd_rawmidi btusb dm_crypt binfmt_misc libarc4 cmdlinepart videobuf2_v4l2 kvm snd_seq btrtl nls_iso8859_1 btintel videobuf2_common iTCO_wdt joydev btbcm iwlwifi input_leds videodev spi_nor btmtk rapl snd_seq_device intel_pmc_bxt nvidia(OE)
May 05 11:56:05 pop-os kernel: pcieport 0000:03:01.0: Unable to change power state from D3hot to D0, device inaccessible
May 05 11:56:05 pop-os kernel:  bluetooth
May 05 11:56:05 pop-os kernel: pcieport 0000:03:02.0: Unable to change power state from D3hot to D0, device inaccessible
May 05 11:56:05 pop-os kernel:  mc
May 05 11:56:05 pop-os kernel:  mtd snd_timer iTCO_vendor_support serio_raw spd5118 intel_cstate cfg80211 snd mei_me hid_multitouch mei soundcore intel_pmc_core intel_vsec pmt_telemetry pmt_class intel_hid intel_scu_pltdrv mac_hid sparse_keymap sch_fq_codel kyber_iosched msr parport_pc ppdev lp parport efi_pstore ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 system76_io(OE) xe drm_gpuvm drm_exec gpu_sched drm_suballoc_helper drm_ttm_helper i915 crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic hid_generic ghash_clmulni_intel drm_buddy nvme sha256_ssse3 i2c_algo_bit ttm sha1_ssse3 sdhci_pci psmouse r8169 nvme_core i2c_i801 cqhci drm_display_helper i2c_mux thunderbolt spi_intel_pci realtek sdhci i2c_smbus spi_intel nvme_auth intel_lpss_pci cec intel_lpss i2c_hid_acpi idma64 rc_core i2c_hid hid video pinctrl_alderlake wmi system76_acpi(OE) aesni_intel crypto_simd cryptd
May 05 11:56:05 pop-os kernel: CPU: 0 UID: 0 PID: 240 Comm: irq/123-pciehp Tainted: G           OE      6.12.10-76061203-generic #202412060638~1743109366~22.04~1fce33b
May 05 11:56:05 pop-os kernel: Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
May 05 11:56:05 pop-os kernel: Hardware name: System76 Serval WS/Serval WS, BIOS 2024-07-08_926f73d 06/28/2024
May 05 11:56:05 pop-os kernel: RIP: 0010:pci_disable_device+0xac/0xc0
May 05 11:56:05 pop-os kernel: Code: 4d 85 e4 75 07 4c 8b a3 c8 00 00 00 48 8d bb c8 00 00 00 e8 c6 23 29 00 4c 89 e2 48 c7 c7 00 d7 8a bd 48 89 c6 e8 84 47 75 ff <0f> 0b e9 66 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90
May 05 11:56:05 pop-os kernel: pci 0000:03:03.0: Unable to change power state from D3hot to D0, device inaccessible
May 05 11:56:05 pop-os kernel: RSP: 0018:ffffa55501267bc8 EFLAGS: 00010246
May 05 11:56:05 pop-os kernel: thunderbolt 0000:04:00.0: Unable to change power state from D3hot to D0, device inaccessible
May 05 11:56:05 pop-os kernel: 
May 05 11:56:05 pop-os kernel: RAX: 0000000000000000 RBX: ffff92b3c3e37000 RCX: 0000000000000000
May 05 11:56:05 pop-os kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 05 11:56:05 pop-os kernel: RBP: ffffa55501267bd8 R08: 0000000000000000 R09: 0000000000000000
May 05 11:56:05 pop-os kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff92b3c24d6c00
May 05 11:56:05 pop-os kernel: R13: 0000000000000040 R14: ffff92b3c3e37148 R15: 0000000000000080
May 05 11:56:05 pop-os kernel: FS:  0000000000000000(0000) GS:ffff92c2fd400000(0000) knlGS:0000000000000000
May 05 11:56:05 pop-os kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 05 11:56:05 pop-os kernel: CR2: 00007ec227d90130 CR3: 000000011c55e000 CR4: 0000000000f50ef0
May 05 11:56:05 pop-os kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 05 11:56:05 pop-os kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 00000000000f0400
May 05 11:56:05 pop-os kernel: PKRU: 55555554
May 05 11:56:05 pop-os kernel: Call Trace:
May 05 11:56:05 pop-os kernel:  <TASK>
May 05 11:56:05 pop-os kernel:  ? show_regs+0x6c/0x80
May 05 11:56:05 pop-os kernel:  ? __warn+0x8d/0x150
May 05 11:56:05 pop-os kernel:  ? pci_disable_device+0xac/0xc0
May 05 11:56:05 pop-os kernel:  ? report_bug+0x182/0x1b0
May 05 11:56:05 pop-os kernel:  ? handle_bug+0x6e/0xb0
May 05 11:56:05 pop-os kernel:  ? exc_invalid_op+0x18/0x80
May 05 11:56:05 pop-os kernel:  ? asm_exc_invalid_op+0x1b/0x20
May 05 11:56:05 pop-os kernel:  ? pci_disable_device+0xac/0xc0
May 05 11:56:05 pop-os kernel:  ? pci_disable_device+0xac/0xc0
May 05 11:56:05 pop-os kernel:  usb_hcd_pci_remove+0xaa/0x110
May 05 11:56:05 pop-os kernel:  xhci_pci_remove+0x8a/0xf0
May 05 11:56:05 pop-os kernel:  pci_device_remove+0x3e/0xb0
May 05 11:56:05 pop-os kernel:  device_remove+0x40/0x80
May 05 11:56:05 pop-os kernel:  device_release_driver_internal+0x20b/0x270
May 05 11:56:05 pop-os kernel:  device_release_driver+0x12/0x20
May 05 11:56:05 pop-os kernel:  pci_stop_bus_device+0x92/0xc0
May 05 11:56:05 pop-os kernel:  pci_stop_bus_device+0x30/0xc0
May 05 11:56:05 pop-os kernel:  pci_stop_bus_device+0x41/0xc0
May 05 11:56:05 pop-os kernel:  pci_stop_and_remove_bus_device+0x12/0x30
May 05 11:56:05 pop-os kernel:  pciehp_unconfigure_device+0x98/0x170
May 05 11:56:05 pop-os kernel:  pciehp_disable_slot+0x6c/0x140
May 05 11:56:05 pop-os kernel:  pciehp_handle_presence_or_link_change+0x14d/0x220
May 05 11:56:05 pop-os kernel:  pciehp_ist+0x227/0x250
May 05 11:56:05 pop-os kernel:  irq_thread_fn+0x21/0x70
May 05 11:56:05 pop-os kernel:  irq_thread+0xee/0x1c0
May 05 11:56:05 pop-os kernel:  ? __pfx_irq_thread_fn+0x10/0x10
May 05 11:56:05 pop-os kernel: xhci_hcd 0000:0e:00.0: Unable to change power state from D3hot to D0, device inaccessible
May 05 11:56:05 pop-os kernel:  ? __pfx_irq_thread_dtor+0x10/0x10
May 05 11:56:05 pop-os kernel:  ? __pfx_irq_thread+0x10/0x10
May 05 11:56:05 pop-os kernel:  kthread+0xe1/0x110
May 05 11:56:05 pop-os kernel:  ? __pfx_kthread+0x10/0x10
May 05 11:56:05 pop-os kernel:  ret_from_fork+0x44/0x70
May 05 11:56:05 pop-os kernel:  ? __pfx_kthread+0x10/0x10
May 05 11:56:05 pop-os kernel:  ret_from_fork_asm+0x1a/0x30
May 05 11:56:05 pop-os kernel:  </TASK>
May 05 11:56:05 pop-os kernel: ---[ end trace 0000000000000000 ]---
May 05 11:56:05 pop-os kernel: ------------[ cut here ]------------
May 05 11:56:05 pop-os kernel: thunderbolt 0000:04:00.0: TX ring 0 is still active
May 05 11:56:05 pop-os kernel: WARNING: CPU: 0 PID: 240 at drivers/thunderbolt/nhi.c:1138 nhi_shutdown+0x6c/0x160 [thunderbolt]
May 05 11:56:05 pop-os kernel: Modules linked in: ccm rfcomm snd_seq_dummy snd_hrtimer cmac algif_hash algif_skcipher af_alg nvidia_uvm(OE) bnep zram 842_decompress 842_compress lz4hc_compress lz4_compress snd_sof_pci_intel_tgl snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel soundwire_cadence snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda_mlink snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_hda_codec_realtek snd_sof snd_hda_codec_generic snd_sof_utils snd_soc_acpi_intel_match snd_hda_scodec_component soundwire_generic_allocation snd_soc_acpi soundwire_bus snd_soc_avs snd_soc_hda_codec snd_hda_ext_core intel_rapl_msr snd_hda_codec_hdmi snd_soc_core snd_compress ac97_bus intel_rapl_common snd_pcm_dmaengine iwlmvm snd_hda_intel intel_uncore_frequency intel_uncore_frequency_common snd_intel_dspcfg intel_tcc_cooling snd_intel_sdw_acpi snd_hda_codec x86_pkg_temp_thermal nvidia_drm(OE) snd_hda_core intel_powerclamp snd_hwdep mac80211 nvidia_modeset(OE) coretemp snd_pcm uvcvideo videobuf2_vmalloc uvc kvm_intel
May 05 11:56:05 pop-os kernel:  snd_seq_midi snd_seq_midi_event videobuf2_memops snd_rawmidi btusb dm_crypt binfmt_misc libarc4 cmdlinepart videobuf2_v4l2 kvm snd_seq btrtl nls_iso8859_1 btintel videobuf2_common iTCO_wdt joydev btbcm iwlwifi input_leds videodev spi_nor btmtk rapl snd_seq_device intel_pmc_bxt nvidia(OE) bluetooth mc mtd snd_timer iTCO_vendor_support serio_raw spd5118 intel_cstate cfg80211 snd mei_me hid_multitouch mei soundcore intel_pmc_core intel_vsec pmt_telemetry pmt_class intel_hid intel_scu_pltdrv mac_hid sparse_keymap sch_fq_codel kyber_iosched msr parport_pc ppdev lp parport efi_pstore ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 system76_io(OE) xe drm_gpuvm drm_exec gpu_sched drm_suballoc_helper drm_ttm_helper i915 crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic hid_generic ghash_clmulni_intel drm_buddy nvme sha256_ssse3 i2c_algo_bit ttm sha1_ssse3 sdhci_pci psmouse r8169 nvme_core i2c_i801 cqhci
May 05 11:56:05 pop-os kernel:  drm_display_helper i2c_mux thunderbolt spi_intel_pci realtek sdhci i2c_smbus spi_intel nvme_auth intel_lpss_pci cec intel_lpss i2c_hid_acpi idma64 rc_core i2c_hid hid video pinctrl_alderlake wmi system76_acpi(OE) aesni_intel crypto_simd cryptd
May 05 11:56:05 pop-os kernel: CPU: 0 UID: 0 PID: 240 Comm: irq/123-pciehp Tainted: G        W  OE      6.12.10-76061203-generic #202412060638~1743109366~22.04~1fce33b
May 05 11:56:05 pop-os kernel: Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
May 05 11:56:05 pop-os kernel: Hardware name: System76 Serval WS/Serval WS, BIOS 2024-07-08_926f73d 06/28/2024
May 05 11:56:05 pop-os kernel: RIP: 0010:nhi_shutdown+0x6c/0x160 [thunderbolt]
May 05 11:56:05 pop-os kernel: Code: 75 07 4c 8b b7 c8 00 00 00 48 81 c7 c8 00 00 00 e8 d9 3b 3f fc 44 89 e1 4c 89 f2 48 c7 c7 f8 a7 4c c0 48 89 c6 e8 94 5f 8b fb <0f> 0b 48 8b 43 28 4a 83 3c e8 00 74 3a 48 8b 7b 08 4c 8b af 18 01
May 05 11:56:05 pop-os kernel: RSP: 0018:ffffa55501267be0 EFLAGS: 00010246
May 05 11:56:05 pop-os kernel: RAX: 0000000000000000 RBX: ffff92b3c1ce8028 RCX: 0000000000000000
May 05 11:56:05 pop-os kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
May 05 11:56:05 pop-os kernel: RBP: ffffa55501267c00 R08: 0000000000000000 R09: 0000000000000000
May 05 11:56:05 pop-os kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
May 05 11:56:05 pop-os kernel: R13: 0000000000000000 R14: ffff92b3c24d6760 R15: 0000000000000080
May 05 11:56:05 pop-os kernel: FS:  0000000000000000(0000) GS:ffff92c2fd400000(0000) knlGS:0000000000000000
May 05 11:56:05 pop-os kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 05 11:56:05 pop-os kernel: CR2: 00007ec227d90130 CR3: 000000011c55e000 CR4: 0000000000f50ef0
May 05 11:56:05 pop-os kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May 05 11:56:05 pop-os kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 00000000000f0400
May 05 11:56:05 pop-os kernel: PKRU: 55555554
May 05 11:56:05 pop-os kernel: Call Trace:
May 05 11:56:05 pop-os kernel:  <TASK>
May 05 11:56:05 pop-os kernel:  ? show_regs+0x6c/0x80
May 05 11:56:05 pop-os kernel:  ? __warn+0x8d/0x150
May 05 11:56:05 pop-os kernel:  ? nhi_shutdown+0x6c/0x160 [thunderbolt]
May 05 11:56:05 pop-os kernel:  ? report_bug+0x182/0x1b0
May 05 11:56:05 pop-os kernel:  ? handle_bug+0x6e/0xb0
May 05 11:56:05 pop-os kernel:  ? exc_invalid_op+0x18/0x80
May 05 11:56:05 pop-os kernel:  ? asm_exc_invalid_op+0x1b/0x20
May 05 11:56:05 pop-os kernel:  ? nhi_shutdown+0x6c/0x160 [thunderbolt]
May 05 11:56:05 pop-os kernel:  nhi_remove+0x53/0x70 [thunderbolt]
May 05 11:56:05 pop-os kernel:  pci_device_remove+0x3e/0xb0
May 05 11:56:05 pop-os kernel:  device_remove+0x40/0x80
May 05 11:56:05 pop-os kernel:  device_release_driver_internal+0x20b/0x270
May 05 11:56:05 pop-os kernel:  device_release_driver+0x12/0x20
May 05 11:56:05 pop-os kernel:  pci_stop_bus_device+0x92/0xc0
May 05 11:56:05 pop-os kernel:  pci_stop_bus_device+0x30/0xc0
May 05 11:56:05 pop-os kernel:  pci_stop_bus_device+0x41/0xc0
May 05 11:56:05 pop-os kernel:  pci_stop_and_remove_bus_device+0x12/0x30
May 05 11:56:05 pop-os kernel:  pciehp_unconfigure_device+0x98/0x170
May 05 11:56:05 pop-os kernel:  pciehp_disable_slot+0x6c/0x140
May 05 11:56:05 pop-os kernel:  pciehp_handle_presence_or_link_change+0x14d/0x220
May 05 11:56:05 pop-os kernel:  pciehp_ist+0x227/0x250
May 05 11:56:05 pop-os kernel:  irq_thread_fn+0x21/0x70
May 05 11:56:05 pop-os kernel:  irq_thread+0xee/0x1c0
May 05 11:56:05 pop-os kernel:  ? __pfx_irq_thread_fn+0x10/0x10
May 05 11:56:05 pop-os kernel:  ? __pfx_irq_thread_dtor+0x10/0x10
May 05 11:56:05 pop-os kernel:  ? __pfx_irq_thread+0x10/0x10
May 05 11:56:05 pop-os kernel:  kthread+0xe1/0x110
May 05 11:56:05 pop-os kernel:  ? __pfx_kthread+0x10/0x10
May 05 11:56:05 pop-os kernel:  ret_from_fork+0x44/0x70
May 05 11:56:05 pop-os kernel:  ? __pfx_kthread+0x10/0x10
May 05 11:56:05 pop-os kernel:  ret_from_fork_asm+0x1a/0x30
May 05 11:56:05 pop-os kernel:  </TASK>
May 05 11:56:05 pop-os kernel: ---[ end trace 0000000000000000 ]---

thunderbolt ports still work on resume, though. If I connect a thunderbolt device and suspend the system will fail to resume from suspend. Nothing gets logged when the system freezes on resume.

oryp12-tbt-suspend-logs.txt
serw13-tbt-suspend-logs.txt

After bisecting the kernel I found this commit, which seems to have introduced the issue:

9d573d1

So far I have only tested reverting this on the serw13, I am working on testing on orypt12 currently. After reverting this commit, the above error no longer occurs, thunderbolt works as expected, and the system no longer freezes upon resuming from suspend.

I will update this once I have results from testing on oryp12

[jacobktm]

With the oryp12, the crash has gone away if the system is suspended and resumed with no thunderbolt device connected, and the system no longer freezes when resuming from suspend with a thunderbolt device connected after reverting the above commit.