CWWK x86-P6-N305 power unable to change power state of NVME drives - D3Cold to D0

ISSUE:

Whenever I try to perform a large copy operation from one ZFS folder to another like:

cp -r /zfs/folder1/* /zfs/folder2/

the copy fails part way through and I get the dmesg error seen below. I currently have PCI Express clock gating and PCI Express power gating disabled on Port 1 and Port 3 on the bios. This was done by going to pci section in the bios (I do not remember the exact location of this setting but I can find it if that helps). I think it was configuration -> top option -> pci option -> then port 1 and port 3 . Whenever I tried disabling these gating options on port 2 it would then only show the first nvme drive on the system until I re-enabled the gating options and I had to take the other nvme drives out and re-insert them to make them show up again. 

HARDWARE:

CWWK X86 P5 development board 4*M.2 NVMe expansion board 12th generation Intel N100 i3-N305 DDR5 4800MHz firewall PC 2x i226-V 2.5G network card

SETUP:

I am using the daughter board with 3 4TB Sabrent Rocket NVME Gen 3 populating the slots 1, 2, and 4 (The 3rd slot has a stripped m.2 screw). I am using the power supply that was provided with the machine. 

BIOS INFORMATION:

CW-AL-2L_V2-V3(X86 N100-N200-I3-N305_V2-V3 4M.2)2024.05.15.iso

https://drive.google.com/file/d/1ZO1-ZUJhSJuE9dsrC6DdkTyBQ-g3LxfE/view?usp=drive_link

OS INFORMATION:

user@localhost:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.2 LTS
Release: 24.04
Codename: noble

ZFS VERSION:

user@localhost:~$ sudo dmesg | grep ZFS
[ 6.242181] ZFS: Loaded module v2.2.2-0ubuntu9.2, ZFS pool version 5000, ZFS filesystem version 5

DMESG Error Report:

user@localhost:~$ sudo dmesg | grep nvme
[ 0.875196] nvme 0000:01:00.0: platform quirk: setting simple suspend
[ 0.875272] nvme 0000:02:00.0: platform quirk: setting simple suspend
[ 0.875959] nvme 0000:03:00.0: platform quirk: setting simple suspend
[ 0.877158] nvme nvme0: pci function 0000:01:00.0
[ 0.878930] nvme nvme1: pci function 0000:03:00.0
[ 0.880332] nvme nvme2: pci function 0000:02:00.0
[ 0.885747] nvme nvme1: missing or invalid SUBNQN field.
[ 0.887192] nvme nvme1: Shutdown timeout set to 10 seconds
[ 0.889403] nvme nvme2: missing or invalid SUBNQN field.
[ 0.891173] nvme nvme2: Shutdown timeout set to 10 seconds
[ 0.891670] nvme nvme0: missing or invalid SUBNQN field.
[ 0.894895] nvme nvme0: Shutdown timeout set to 10 seconds
[ 0.896748] nvme nvme2: 8/0/0 default/read/poll queues
[ 0.898708] nvme nvme2: Ignoring bogus Namespace Identifiers
[ 0.902407] nvme2n1: p1
[ 0.904948] nvme nvme1: 8/0/0 default/read/poll queues
[ 0.906102] nvme nvme0: 8/0/0 default/read/poll queues
[ 0.909625] nvme nvme0: Ignoring bogus Namespace Identifiers
[ 0.910757] nvme nvme1: Ignoring bogus Namespace Identifiers
[ 0.912930] nvme1n1: p1
[ 0.917076] nvme0n1: p1
[ 261.506801] nvme nvme2: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
[ 261.506821] nvme nvme2: Does your device have a faulty power saving mode enabled?
[ 261.506825] nvme nvme2: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
[ 261.506862] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
[ 261.506867] nvme nvme0: Does your device have a faulty power saving mode enabled?
[ 261.506871] nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
[ 261.524827] nvme 0000:02:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 261.525301] nvme nvme2: Disabling device after reset failure: -19
[ 261.528847] nvme 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible
[ 261.529504] nvme nvme0: Disabling device after reset failure: -19
[ 577.156066] Buffer I/O error on dev nvme0n1p1, logical block 0, async page read
[ 577.157599] Buffer I/O error on dev nvme0n1p1, logical block 0, async page read
[ 577.157662] Buffer I/O error on dev nvme0n1p1, logical block 1, async page read
[ 577.165133] nvme nvme0: Identify namespace failed (-5)
[ 577.169745] nvme nvme2: Identify namespace failed (-5)
[ 577.456954] Buffer I/O error on dev nvme2n1p1, logical block 0, async page read
[ 577.458525] Buffer I/O error on dev nvme2n1p1, logical block 0, async page read
[ 577.458573] Buffer I/O error on dev nvme2n1p1, logical block 1, async page read  [ 587.096462] nvme nvme0: Identify namespace failed (-5)