RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

[Shaw1919]

When I run the command python src/track.py with reid, I encounter the error:
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling 'cublasSgemm(handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)'.
How can I resolve this issue?

(trackformer) xiaoxiao@xiaoxiao-Precision-3680:~/code/trackformer-main$ python src/track.py with reid
WARNING - track - No observers have been added to this run
INFO - track - Running command 'main'
INFO - track - Started
Configuration (modified, added, typechanged, doc):
data_root_dir = 'data'
dataset_name = 'MOT17-ALL-ALL'
generate_attention_maps = False
interpolate = False
load_results_dir = None
obj_detect_checkpoint_file = 'models/mot17_crowdhuman_deformable_multi_frame/checkpoint_epoch_40.pth'
output_dir = None
seed = 666
verbose = False
write_images = False
frame_range:
end = 1.0
start = 0.0
tracker_cfg:
detection_nms_thresh = 0.9
detection_obj_score_thresh = 0.4
inactive_patience = 5
prev_frame_dist = 1
public_detections = False
reid_greedy_matching = False
reid_score_thresh = 0.4
reid_sim_only = False
reid_sim_threshold = 0.0
steps_termination = 1
track_nms_thresh = 0.9
track_obj_score_thresh = 0.4
INFO - main - INIT object detector [EPOCH: 40]
INFO - main - ------------------
INFO - main - TRACK SEQ: MOT17-01-DPM
0%| | 0/450 [00:00<?, ?it/s]
ERROR - track - Failed after 0:00:03!
Traceback (most recent calls WITHOUT Sacred internals):
File "src/track.py", line 135, in main
tracker.step(frame_data)
File "/home/xiaoxiao/code/trackformer-main/src/trackformer/models/tracker.py", line 306, in step
outputs, _, features, _, _ = self.obj_detector(img, target, self._prev_features[0])
File "/home/xiaoxiao/anaconda3/envs/trackformer/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/xiaoxiao/code/trackformer-main/src/trackformer/models/detr_tracking.py", line 275, in forward
out, targets, features, memory, hs = super().forward(samples, targets, prev_features)
File "/home/xiaoxiao/code/trackformer-main/src/trackformer/models/deformable_detr.py", line 227, in forward
self.transformer(src_list, mask_list, pos_list, query_embeds, targets)
File "/home/xiaoxiao/anaconda3/envs/trackformer/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/xiaoxiao/code/trackformer-main/src/trackformer/models/deformable_transformer.py", line 166, in forward
mask_flatten[:, :src_flatten.shape[1] // 2])
File "/home/xiaoxiao/anaconda3/envs/trackformer/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/xiaoxiao/code/trackformer-main/src/trackformer/models/deformable_transformer.py", line 325, in forward
output = layer(output, pos, reference_points, spatial_shapes, padding_mask)
File "/home/xiaoxiao/anaconda3/envs/trackformer/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/xiaoxiao/code/trackformer-main/src/trackformer/models/deformable_transformer.py", line 290, in forward
src2 = self.self_attn(self.with_pos_embed(src, pos), reference_points, src, spatial_shapes, padding_mask)
File "/home/xiaoxiao/anaconda3/envs/trackformer/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/xiaoxiao/code/trackformer-main/src/trackformer/models/ops/modules/ms_deform_attn.py", line 65, in forward
value = self.value_proj(input_flatten.to(dtype=self.value_proj.weight.dtype, device=self.value_proj.weight.device))
File "/home/xiaoxiao/anaconda3/envs/trackformer/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/xiaoxiao/anaconda3/envs/trackformer/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/home/xiaoxiao/anaconda3/envs/trackformer/lib/python3.7/site-packages/torch/nn/functional.py", line 1612, in linear
output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)