[2025-10-31 01:45:20.524] [info] Running: "basecaller" "hac" "-r" "/gscratch/srlab/kdurkin1/SIFP-nanopore/A-Group1/data/06.01-G1-Library4-MinION-Dorado-recall-GPU//" "--kit-name" "SQK-NBD114-96" "--trim" "all" "--reference" "/gscratch/srlab/kdurkin1/SIFP-nanopore/data/GCA_965233905.1_jaEunKnig1.1/GCA_965233905.1_jaEunKnig1.1_genomic.fna" "--modified-bases" "5mCG_5hmCG" "6mA" "--device" "auto"
[2025-10-31 01:46:01.773] [info]  - downloading dna_r10.4.1_e8.2_400bps_hac@v5.2.0 with httplib
[2025-10-31 01:46:02.547] [info]  - downloading dna_r10.4.1_e8.2_400bps_hac@v5.2.0_5mCG_5hmCG@v2 with httplib
[2025-10-31 01:46:03.371] [info]  - downloading dna_r10.4.1_e8.2_400bps_hac@v5.2.0_6mA@v1 with httplib
[2025-10-31 01:46:04.316] [info] > Creating basecall pipeline
[2025-10-31 01:46:07.545] [info] Using CUDA devices:
[2025-10-31 01:46:07.545] [info] cuda:0 - Tesla P100-PCIE-16GB
[2025-10-31 01:46:09.059] [error] CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /builds/machine-learning/torch-builds/pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x14db3b9dd5e8 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xe0 (0x14db3b9724a2 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x14db3baafa42 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libc10_cuda.so)
frame #3: void at::native::gpu_kernel_impl_nocast<at::native::FillFunctor<c10::Half> >(at::TensorIteratorBase&, at::native::FillFunctor<c10::Half> const&) + 0x5eb (0x14db3d26328b in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x16a5afb (0x14db3d24dafb in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #5: at::native::fill_kernel_cuda(at::TensorIterator&, c10::Scalar const&) + 0x1e (0x14db3d24f96e in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #6: <unknown function> + 0x1747baf (0x14dba1262baf in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x324dfe1 (0x14db3edf5fe1 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #8: at::_ops::fill__Scalar::call(at::Tensor&, c10::Scalar const&) + 0x172 (0x14dba19bcee2 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #9: at::native::zero_(at::Tensor&) + 0xa3 (0x14dba1262f63 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #10: <unknown function> + 0x324c8b9 (0x14db3edf48b9 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #11: at::_ops::zero_::call(at::Tensor&) + 0xaa (0x14dba1e6531a in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #12: at::native::zeros_symint(c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) + 0x23b (0x14dba150c42b in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #13: <unknown function> + 0x2790ae7 (0x14dba22abae7 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #14: at::_ops::zeros::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) + 0xe5 (0x14dba196a115 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x2535750 (0x14dba2050750 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #16: at::_ops::zeros::call(c10::ArrayRef<c10::SymInt>, std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::optional<bool>) + 0x1fa (0x14dba19d50ca in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #17: at::native::cudnn_rnn::copy_weights_to_flat_buf_views(c10::ArrayRef<at::Tensor>, long, long, long, long, long, long, bool, bool, cudnnDataType_t, c10::TensorOptions const&, bool, bool, bool) + 0x29d (0x14db3c7dadbd in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #18: at::native::_cudnn_rnn_flatten_weight(c10::ArrayRef<at::Tensor>, long, long, long, long, long, long, bool, bool) + 0x98 (0x14db3c7db858 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #19: <unknown function> + 0x32705bf (0x14db3ee185bf in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #20: <unknown function> + 0x32a12a3 (0x14db3ee492a3 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cuda.so)
frame #21: at::_ops::_cudnn_rnn_flatten_weight::call(c10::ArrayRef<at::Tensor>, long, c10::SymInt, long, c10::SymInt, c10::SymInt, long, bool, bool) + 0x2c4 (0x14dba1b3ce54 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #22: <unknown function> + 0x57ce376 (0x14dba52e9376 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #23: torch::nn::detail::RNNImplBase<torch::nn::LSTMImpl>::flatten_parameters() + 0x342 (0x14dba52f0a22 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #24: void torch::nn::Module::to_impl<c10::Device&, bool&>(c10::Device&, bool&) + 0xa5 (0x14dba5244ec5 in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #25: void torch::nn::Module::to_impl<c10::Device&, bool&>(c10::Device&, bool&) + 0x7f (0x14dba5244e9f in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #26: torch::nn::Module::to(c10::Device, bool) + 0x1c (0x14dba5241b8c in /srlab/programs/dorado-1.2.0-linux-x64/bin/../lib/libtorch_cpu.so)
frame #27: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0x108c4f1]
frame #28: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0x107e02f]
frame #29: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0x107eb52]
frame #30: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0xeb66cb]
frame #31: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0xeb2928]
frame #32: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0x570430]
frame #33: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0x575719]
frame #34: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0x526c9d]
frame #35: <unknown function> + 0x2a1ca (0x14db3412f1ca in /lib/x86_64-linux-gnu/libc.so.6)
frame #36: __libc_start_main + 0x8b (0x14db3412f28b in /lib/x86_64-linux-gnu/libc.so.6)
frame #37: /srlab/programs/dorado-1.2.0-linux-x64/bin/dorado() [0x5499de]