Test erros on PyTorch Release 25.08 on Series 50

[garciadias]

Describe the bug

Running ./runtests.sh -u -q --net --coverage withing the (b5bc69d, dev, release 1.5.1) version of the repo.

I am sharing the results of b5bc69d, I will be sending the results of the dev branch soon, if there are any differences.

To Reproduce

Steps to reproduce the behavior:

0. Clone monai: git clone https://github.com/Project-MONAI/MONAI.git
1. Check out the relevant branch: git checkout b5bc69d or git checkout dev
2. Run pytorch:25.08 image: docker run -v ./:/workspace/ --name torch2508 --gpus=all -it nvcr.io/nvidia/pytorch:25.08-py3 bash
3. Run the tests via ./runtests.sh -u -q --net --coverage

Expected behavior

Returns no errors.

The following errors are returned from b5bc69d:

======================================================================
ERROR: test_dataloading_img_label_0 (tests.data.test_arraydataset.TestArrayDataset.test_dataloading_img_label_0)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1286, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 113, in get
    if not self._poll(timeout):
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 440, in _poll
    r = wait([self], timeout)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 1136, in wait
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 3473) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/data/test_arraydataset.py", line 184, in test_dataloading_img_label
    new_data = next(iter(loader))  # test batching
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 735, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1493, in _next_data
    idx, data = self._get_data()
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1455, in _get_data
    success, data = self._try_get_data()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1299, in _try_get_data
    raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 3473) exited unexpectedly

======================================================================
ERROR: test_multiprocess_mapping_file (tests.data.test_mapping_file.TestWriteFileMapping.test_multiprocess_mapping_file)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1286, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 113, in get
    if not self._poll(timeout):
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 440, in _poll
    r = wait([self], timeout)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 1136, in wait
    ready = selector.select(timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 4575) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/tests/data/test_mapping_file.py", line 102, in test_multiprocess_mapping_file
    for _ in multi_loader:
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 735, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1493, in _next_data
    idx, data = self._get_data()
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1455, in _get_data
    success, data = self._try_get_data()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1299, in _try_get_data
    raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 4575) exited unexpectedly

======================================================================
ERROR: test_get_history (tests.integration.test_auto3dseg_hpo.TestHPO.test_get_history)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/tests/integration/test_auto3dseg_hpo.py", line 174, in test_get_history
    NNIGen().run_algo(obj_filename, self.work_dir)
  File "/workspace/monai/apps/auto3dseg/hpo_gen.py", line 233, in run_algo
    self.algo.train(self.params)
  File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 300, in train
    return self._run_cmd(cmd)
           ^^^^^^^^^^^^^^^^^^
  File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 277, in _run_cmd
    return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/monai/utils/misc.py", line 889, in run_cmd
    return subprocess.run(cmd_list, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/scripts/train.py', 'run', "--config_file='/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/hyper_parameters.yaml,/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/network.yaml,/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/transforms_infer.yaml,/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/transforms_train.yaml,/tmp/tmpflaic_ks/workdir/swinunetr_0_override_None/configs/transforms_validate.yaml'"]' returned non-zero exit status 1.

======================================================================
ERROR: test_run_algo (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_algo)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/tests/integration/test_auto3dseg_hpo.py", line 142, in test_run_algo
    NNIGen().run_algo(obj_filename, self.work_dir)
  File "/workspace/monai/apps/auto3dseg/hpo_gen.py", line 233, in run_algo
    self.algo.train(self.params)
  File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 300, in train
    return self._run_cmd(cmd)
           ^^^^^^^^^^^^^^^^^^
  File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 277, in _run_cmd
    return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/monai/utils/misc.py", line 889, in run_cmd
    return subprocess.run(cmd_list, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/scripts/train.py', 'run', "--config_file='/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/hyper_parameters.yaml,/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/network.yaml,/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/transforms_infer.yaml,/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/transforms_train.yaml,/tmp/tmpqaxb1d91/workdir/swinunetr_0_override_None/configs/transforms_validate.yaml'"]' returned non-zero exit status 1.

======================================================================
ERROR: test_run_optuna (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_optuna)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/tests/integration/test_auto3dseg_hpo.py", line 157, in test_run_optuna
    study.optimize(
  File "/usr/local/lib/python3.12/dist-packages/optuna/study/study.py", line 490, in optimize
    _optimize(
  File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 63, in _optimize
    _optimize_sequential(
  File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 160, in _optimize_sequential
    frozen_trial_id = _run_trial(study, func, catch)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 258, in _run_trial
    raise func_err
  File "/usr/local/lib/python3.12/dist-packages/optuna/study/_optimize.py", line 201, in _run_trial
    value_or_values = func(trial)
                      ^^^^^^^^^^^
  File "/workspace/monai/apps/auto3dseg/hpo_gen.py", line 338, in __call__
    self.run_algo(obj_filename, output_folder, template_path)
  File "/workspace/monai/apps/auto3dseg/hpo_gen.py", line 396, in run_algo
    self.algo.train(self.params)
  File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 300, in train
    return self._run_cmd(cmd)
           ^^^^^^^^^^^^^^^^^^
  File "/workspace/monai/apps/auto3dseg/bundle_gen.py", line 277, in _run_cmd
    return run_cmd(cmd.split(), run_cmd_verbose=True, env=ps_environ, check=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/monai/utils/misc.py", line 889, in run_cmd
    return subprocess.run(cmd_list, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/scripts/train.py', 'run', "--config_file='/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/hyper_parameters.yaml,/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/network.yaml,/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/transforms_infer.yaml,/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/transforms_train.yaml,/tmp/tmpez132vik/optuna_test/swinunetr_0_override_learning_rate_0.01/configs/transforms_validate.yaml'", '--learning_rate=0.01']' returned non-zero exit status 1.

======================================================================
FAIL: test_affine_028 (tests.transforms.test_affine.TestAffine.test_affine_028)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.      , 0.041667, 0.208333, 0.125   ],
         [0.083333, 0.25    , 0.583333, 0.333333],
         [0.416667, 0.916667, 1.25    , 0.666667],...
 y: array([[[[0.      , 0.041728, 0.208516, 0.125183],
         [0.083455, 0.250244, 0.583659, 0.3337  ],
         [0.417033, 0.917074, 1.250488, 0.667277],...

======================================================================
FAIL: test_affine_029 (tests.transforms.test_affine.TestAffine.test_affine_029)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.25    , 0.416667, 0.083333, 0.      ],
         [0.541667, 0.916667, 0.25    , 0.041667],
         [0.708333, 1.25    , 0.583333, 0.208333],...
 y: array([[[[0.250366, 0.417033, 0.083455, 0.      ],
         [0.542216, 0.917074, 0.250244, 0.041728],
         [0.709005, 1.250488, 0.583659, 0.208516],...

======================================================================
FAIL: test_affine_058 (tests.transforms.test_affine.TestAffine.test_affine_058)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.      , 0.041667, 0.208333, 0.125   ],
         [0.083333, 0.25    , 0.583333, 0.333333],
         [0.416667, 0.916667, 1.25    , 0.666667],...
 y: array([[[[0.      , 0.041728, 0.208516, 0.125183],
         [0.083455, 0.250244, 0.583659, 0.3337  ],
         [0.417033, 0.917074, 1.250488, 0.667277],...

======================================================================
FAIL: test_affine_059 (tests.transforms.test_affine.TestAffine.test_affine_059)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.25    , 0.416667, 0.083333, 0.      ],
         [0.541667, 0.916667, 0.25    , 0.041667],
         [0.708333, 1.25    , 0.583333, 0.208333],...
 y: array([[[[0.250366, 0.417033, 0.083455, 0.      ],
         [0.542216, 0.917074, 0.250244, 0.041728],
         [0.709005, 1.250488, 0.583659, 0.208516],...

======================================================================
FAIL: test_affine_088 (tests.transforms.test_affine.TestAffine.test_affine_088)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.      , 0.041667, 0.208333, 0.125   ],
         [0.083333, 0.25    , 0.583333, 0.333333],
         [0.416667, 0.916667, 1.25    , 0.666667],...
 y: array([[[[0.      , 0.041728, 0.208516, 0.125183],
         [0.083455, 0.250244, 0.583659, 0.3337  ],
         [0.417033, 0.917074, 1.250488, 0.667277],...

======================================================================
FAIL: test_affine_089 (tests.transforms.test_affine.TestAffine.test_affine_089)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.25    , 0.416667, 0.083333, 0.      ],
         [0.541667, 0.916667, 0.25    , 0.041667],
         [0.708333, 1.25    , 0.583333, 0.208333],...
 y: array([[[[0.250366, 0.417033, 0.083455, 0.      ],
         [0.542216, 0.917074, 0.250244, 0.041728],
         [0.709005, 1.250488, 0.583659, 0.208516],...

======================================================================
FAIL: test_affine_118 (tests.transforms.test_affine.TestAffine.test_affine_118)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.      , 0.041667, 0.208333, 0.125   ],
         [0.083333, 0.25    , 0.583333, 0.333333],
         [0.416667, 0.916667, 1.25    , 0.666667],...
 y: array([[[[0.      , 0.041728, 0.208516, 0.125183],
         [0.083455, 0.250244, 0.583659, 0.3337  ],
         [0.417033, 0.917074, 1.250488, 0.667277],...

======================================================================
FAIL: test_affine_119 (tests.transforms.test_affine.TestAffine.test_affine_119)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, input_data, output_idx=output_idx)
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.25    , 0.416667, 0.083333, 0.      ],
         [0.541667, 0.916667, 0.25    , 0.041667],
         [0.708333, 1.25    , 0.583333, 0.208333],...
 y: array([[[[0.250366, 0.417033, 0.083455, 0.      ],
         [0.542216, 0.917074, 0.250244, 0.041728],
         [0.709005, 1.250488, 0.583659, 0.208516],...

======================================================================
FAIL: test_affine_028 (tests.transforms.test_affined.TestAffined.test_affine_028)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.      , 0.041667, 0.208333, 0.125   ],
         [0.083333, 0.25    , 0.583333, 0.333333],
         [0.416667, 0.916667, 1.25    , 0.666667],...
 y: array([[[[0.      , 0.041728, 0.208516, 0.125183],
         [0.083455, 0.250244, 0.583659, 0.3337  ],
         [0.417033, 0.917074, 1.250488, 0.667277],...

======================================================================
FAIL: test_affine_029 (tests.transforms.test_affined.TestAffined.test_affine_029)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.25    , 0.416667, 0.083333, 0.      ],
         [0.541667, 0.916667, 0.25    , 0.041667],
         [0.708333, 1.25    , 0.583333, 0.208333],...
 y: array([[[[0.250366, 0.417033, 0.083455, 0.      ],
         [0.542216, 0.917074, 0.250244, 0.041728],
         [0.709005, 1.250488, 0.583659, 0.208516],...

======================================================================
FAIL: test_affine_058 (tests.transforms.test_affined.TestAffined.test_affine_058)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.      , 0.041667, 0.208333, 0.125   ],
         [0.083333, 0.25    , 0.583333, 0.333333],
         [0.416667, 0.916667, 1.25    , 0.666667],...
 y: array([[[[0.      , 0.041728, 0.208516, 0.125183],
         [0.083455, 0.250244, 0.583659, 0.3337  ],
         [0.417033, 0.917074, 1.250488, 0.667277],...

======================================================================
FAIL: test_affine_059 (tests.transforms.test_affined.TestAffined.test_affine_059)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.25    , 0.416667, 0.083333, 0.      ],
         [0.541667, 0.916667, 0.25    , 0.041667],
         [0.708333, 1.25    , 0.583333, 0.208333],...
 y: array([[[[0.250366, 0.417033, 0.083455, 0.      ],
         [0.542216, 0.917074, 0.250244, 0.041728],
         [0.709005, 1.250488, 0.583659, 0.208516],...

======================================================================
FAIL: test_affine_088 (tests.transforms.test_affined.TestAffined.test_affine_088)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.      , 0.041667, 0.208333, 0.125   ],
         [0.083333, 0.25    , 0.583333, 0.333333],
         [0.416667, 0.916667, 1.25    , 0.666667],...
 y: array([[[[0.      , 0.041728, 0.208516, 0.125183],
         [0.083455, 0.250244, 0.583659, 0.3337  ],
         [0.417033, 0.917074, 1.250488, 0.667277],...

======================================================================
FAIL: test_affine_089 (tests.transforms.test_affined.TestAffined.test_affine_089)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.25    , 0.416667, 0.083333, 0.      ],
         [0.541667, 0.916667, 0.25    , 0.041667],
         [0.708333, 1.25    , 0.583333, 0.208333],...
 y: array([[[[0.250366, 0.417033, 0.083455, 0.      ],
         [0.542216, 0.917074, 0.250244, 0.041728],
         [0.709005, 1.250488, 0.583659, 0.208516],...

======================================================================
FAIL: test_affine_118 (tests.transforms.test_affined.TestAffined.test_affine_118)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.      , 0.041667, 0.208333, 0.125   ],
         [0.083333, 0.25    , 0.583333, 0.333333],
         [0.416667, 0.916667, 1.25    , 0.666667],...
 y: array([[[[0.      , 0.041728, 0.208516, 0.125183],
         [0.083455, 0.250244, 0.583659, 0.3337  ],
         [0.417033, 0.917074, 1.250488, 0.667277],...

======================================================================
FAIL: test_affine_119 (tests.transforms.test_affined.TestAffined.test_affine_119)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/transforms/test_affined.py", line 186, in test_affine
    test_resampler_lazy(resampler, non_lazy_result, lazy_input_param, call_param, output_key="img")
  File "/workspace/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

Mismatched elements: 63 / 64 (98.4%)
Max absolute difference: 0.002
Max relative difference: 0.001
 x: array([[[[0.25    , 0.416667, 0.083333, 0.      ],
         [0.541667, 0.916667, 0.25    , 0.041667],
         [0.708333, 1.25    , 0.583333, 0.208333],...
 y: array([[[[0.250366, 0.417033, 0.083455, 0.      ],
         [0.542216, 0.917074, 0.250244, 0.041728],
         [0.709005, 1.250488, 0.583659, 0.208516],...

======================================================================
FAIL: test_flash_attention_0 (tests.networks.blocks.test_crossattention.TestResBlock.test_flash_attention_0)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/networks/blocks/test_crossattention.py", line 174, in test_flash_attention
    assert_allclose(out_1, out_2, atol=1e-4)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0.0001

Mismatched elements: 219 / 2048 (10.7%)
Max absolute difference: 0.
Max relative difference: 0.835
 x: array([[[ 0.0093  , -0.26159 , -0.350302, ...,  0.028707,  0.324385,
          0.086146],
        [-0.059419, -0.61913 ,  0.034504, ..., -0.658143, -0.110547,...
 y: array([[[ 0.0093  , -0.26159 , -0.350302, ...,  0.028707,  0.324385,
          0.086146],
        [-0.059226, -0.61921 ,  0.034631, ..., -0.658143, -0.110504,...

======================================================================
FAIL: test_flash_attention_1 (tests.networks.blocks.test_crossattention.TestResBlock.test_flash_attention_1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/networks/blocks/test_crossattention.py", line 174, in test_flash_attention
    assert_allclose(out_1, out_2, atol=1e-4)
  File "/workspace/tests/test_utils.py", line 140, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0.0001

Mismatched elements: 16 / 2048 (0.781%)
Max absolute difference: 0.
Max relative difference: 0.511
 x: array([[[-0.029829,  0.030035, -0.015213, ...,  0.130288,  0.059629,
          0.06318 ],
        [-0.062274, -0.03078 , -0.053948, ...,  0.15801 ,  0.027424,...
 y: array([[[-0.029799,  0.030049, -0.015215, ...,  0.130326,  0.059602,
          0.063214],
        [-0.06229 , -0.030772, -0.053921, ...,  0.15801 ,  0.027432,...

----------------------------------------------------------------------
Ran 17268 tests in 1100.158s

FAILED (failures=18, errors=5, skipped=991)


tests finished, printing completed times >10.0s in ascending order...

test_gen_patches_0 (tests.data.test_masked_patch_wsi_dataset.TestSlidingPatchWSIDatasetOpenSlide.test_gen_patches_0) (10.1s)
test_content_1 (tests.handlers.test_handler_garbage_collector.TestHandlerGarbageCollector.test_content_1) (13.5s)
test_script (tests.networks.nets.test_highresnet.TestHighResNet.test_script) (14.9s)
test_against_official_code_using_cg (tests.transforms.test_ultrasound_confidence_map_transform.TestUltrasoundConfidenceMapTransform.test_against_official_code_using_cg) (16.1s)
test_start (tests.integration.test_loader_semaphore.TestImportLock.test_start) (16.1s)
test_pdb (tests.bundle.test_config_parser.TestConfigParser.test_pdb) (16.3s)
test_verify_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_metadata.TestVerifyMetaData.test_verify_0__workspace_tests_testing_data_metadata_json) (21.5s)
test_verify_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_net.TestVerifyNetwork.test_verify_0__workspace_tests_testing_data_metadata_json) (21.6s)
test_bundle (tests.networks.nets.test_bundle_init_bundle.TestBundleInit.test_bundle) (21.7s)
test_default_value_1_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_1_model) (21.8s)
test_export_1_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_1_model) (21.9s)
test_default_value_0_ (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_0_) (21.9s)
test_verify_fp16_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_net.TestVerifyNetwork.test_verify_fp16_0__workspace_tests_testing_data_metadata_json) (22.1s)
test_export_0_ (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_0_) (22.2s)
test_export_2_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_2_model) (22.5s)
test_default_value_2_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_2_model) (22.5s)
test_multiprocess_mapping_file (tests.data.test_mapping_file.TestWriteFileMapping.test_multiprocess_mapping_file) (35.7s)
test_run_algo (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_algo) (41.5s)
test_run_optuna (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_optuna) (41.7s)
test_get_history (tests.integration.test_auto3dseg_hpo.TestHPO.test_get_history) (42.1s)
test discovery time: 5.56s
total testing time: 4.68e+02s
Remember to check above times for any errors!

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c "import monai; monai.config.print_debug_info()"

TensorRT-LLM is not installed. Please install TensorRT-LLM or set TRTLLM_PLUGINS_PATH to the directory containing libnvinfer_plugin_tensorrt_llm.so to use converters for torch.distributed ops
================================
Printing MONAI config...
================================
MONAI version: 0+unknown
Numpy version: 1.26.4
Pytorch version: 2.8.0a0+34c6371d24.nv25.08
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: None
MONAI __file__: /workspace/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.4
Nibabel version: 5.3.2
scikit-image version: 0.25.2
scipy version: 1.15.3
Pillow version: 11.3.0
Tensorboard version: 2.16.2
gdown version: 5.2.0
TorchVision version: 0.23.0a0+428a54c9
tqdm version: 4.67.1
lmdb version: 1.7.3
psutil version: 7.0.0
pandas version: 2.2.3
einops version: 0.8.1
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 3.4.0
pynrrd version: 1.1.3
clearml version: 2.0.3rc0

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 24.04.2 LTS
Platform: Linux-6.8.0-83-generic-x86_64-with-glibc2.39
Processor: x86_64
Machine: x86_64
Python version: 3.12.3
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 16
Num logical CPUs: 32
Num usable CPUs: 32
CPU usage (%): [2.1, 41.4, 1.6, 2.1, 1.6, 1.6, 1.6, 1.6, 1.7, 1.9, 1.6, 1.6, 1.6, 1.6, 1.6, 28.0, 1.6, 15.8, 1.9, 1.6, 1.6, 1.6, 1.6, 2.1, 3.7, 1.6, 1.6, 1.4, 1.6, 1.6, 1.6, 17.5]
CPU freq. (MHz): 2373
Load avg. in last 1, 5, 15 mins (%): [3.0, 9.8, 8.8]
Disk usage (%): 13.7
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 125.7
Available memory (GB): 115.2
Used memory (GB): 9.2

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 13.0
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: 1
cuDNN version: 91200
Current device: 0
Library compiled for CUDA architectures: ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
GPU 0 Name: NVIDIA GeForce RTX 5090
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 170
GPU 0 Total memory (GB): 31.4
GPU 0 CUDA capability (maj.min): 12.0

Additional context

Related to issue #8507 and #8487

[garciadias]

From dev, there are two more errors, and 38 more tests are skipped.

======================================================================
ERROR: test_state_cacher_1 (tests.utils.test_state_cacher.TestStateCacher.test_state_cacher_1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/tests/utils/test_state_cacher.py", line 51, in test_state_cacher
    data_obj_restored = state_cacher.retrieve(key)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/monai/utils/state_cacher.py", line 127, in retrieve
    data_obj = torch.load(fn, map_location=lambda storage, location: storage)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/clearml/binding/frameworks/__init__.py", line 36, in _inner_patch
    raise ex
  File "/usr/local/lib/python3.12/dist-packages/clearml/binding/frameworks/__init__.py", line 34, in _inner_patch
    ret = patched_fn(original_fn, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/clearml/binding/frameworks/pytorch_bind.py", line 270, in _load
    return original_fn(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/serialization.py", line 1529, in load
    raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
Please file an issue with the following so that we can make `weights_only=True` compatible with your use case: WeightsUnpickler error: Unsupported operand 149

Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.

======================================================================
ERROR: test_pickling (tests.data.meta_tensor.test_meta_tensor.TestMetaTensor.test_pickling)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/tests/data/meta_tensor/test_meta_tensor.py", line 248, in test_pickling
    m2 = torch.load(fname)
         ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/clearml/binding/frameworks/__init__.py", line 36, in _inner_patch
    raise ex
  File "/usr/local/lib/python3.12/dist-packages/clearml/binding/frameworks/__init__.py", line 34, in _inner_patch
    ret = patched_fn(original_fn, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/clearml/binding/frameworks/pytorch_bind.py", line 270, in _load
    return original_fn(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/serialization.py", line 1529, in load
    raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, do those steps only if you trust the source of the checkpoint. 
	(1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
	(2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message.
	WeightsUnpickler error: Unsupported global: GLOBAL monai.utils.enums.MetaKeys was not an allowed global by default. Please use `torch.serialization.add_safe_globals([monai.utils.enums.MetaKeys])` or the `torch.serialization.safe_globals([monai.utils.enums.MetaKeys])` context manager to allowlist this global if you trust this class/function.

Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.

----------------------------------------------------------------------
Ran 16856 tests in 1079.372s

FAILED (failures=18, errors=7, skipped=1029)


tests finished, printing completed times >10.0s in ascending order...

test_content_1 (tests.handlers.test_handler_garbage_collector.TestHandlerGarbageCollector.test_content_1) (13.5s)
test_script (tests.networks.nets.test_highresnet.TestHighResNet.test_script) (14.9s)
test_against_official_code_using_cg (tests.transforms.test_ultrasound_confidence_map_transform.TestUltrasoundConfidenceMapTransform.test_against_official_code_using_cg) (16.2s)
test_pdb (tests.bundle.test_config_parser.TestConfigParser.test_pdb) (16.4s)
test_start (tests.integration.test_loader_semaphore.TestImportLock.test_start) (16.6s)
test_verify_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_metadata.TestVerifyMetaData.test_verify_0__workspace_tests_testing_data_metadata_json) (21.3s)
test_verify_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_net.TestVerifyNetwork.test_verify_0__workspace_tests_testing_data_metadata_json) (21.3s)
test_bundle (tests.networks.nets.test_bundle_init_bundle.TestBundleInit.test_bundle) (21.3s)
test_default_value_1_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_1_model) (21.7s)
test_default_value_0_ (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_0_) (21.9s)
test_export_0_ (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_0_) (21.9s)
test_verify_fp16_0__workspace_tests_testing_data_metadata_json (tests.bundle.test_bundle_verify_net.TestVerifyNetwork.test_verify_fp16_0__workspace_tests_testing_data_metadata_json) (22.0s)
test_export_1_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_1_model) (22.0s)
test_export_2_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_export_2_model) (22.3s)
test_default_value_2_model (tests.bundle.test_bundle_ckpt_export.TestCKPTExport.test_default_value_2_model) (22.4s)
test_multiprocess_mapping_file (tests.data.test_mapping_file.TestWriteFileMapping.test_multiprocess_mapping_file) (37.6s)
test_run_optuna (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_optuna) (41.7s)
test_run_algo (tests.integration.test_auto3dseg_hpo.TestHPO.test_run_algo) (41.8s)
test_get_history (tests.integration.test_auto3dseg_hpo.TestHPO.test_get_history) (42.1s)
test discovery time: 7.78s
total testing time: 4.59e+02s
Remember to check above times for any errors!

Environment:

python -c "import monai; monai.config.print_debug_info()"

TensorRT-LLM is not installed. Please install TensorRT-LLM or set TRTLLM_PLUGINS_PATH to the directory containing libnvinfer_plugin_tensorrt_llm.so to use converters for torch.distributed ops
================================
Printing MONAI config...
================================
MONAI version: 0+unknown
Numpy version: 1.26.4
Pytorch version: 2.8.0a0+34c6371d24.nv25.08
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: None
MONAI __file__: /workspace/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.4
Nibabel version: 5.3.2
scikit-image version: 0.25.2
scipy version: 1.15.3
Pillow version: 11.3.0
Tensorboard version: 2.16.2
gdown version: 5.2.0
TorchVision version: 0.23.0a0+428a54c9
tqdm version: 4.67.1
lmdb version: 1.7.3
psutil version: 7.0.0
pandas version: 2.2.3
einops version: 0.8.1
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 3.4.0
pynrrd version: 1.1.3
clearml version: 2.0.3rc0

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 24.04.2 LTS
Platform: Linux-6.8.0-83-generic-x86_64-with-glibc2.39
Processor: x86_64
Machine: x86_64
Python version: 3.12.3
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 16
Num logical CPUs: 32
Num usable CPUs: 32
CPU usage (%): [1.4, 2.1, 1.6, 1.6, 1.6, 1.6, 1.6, 1.9, 3.3, 1.9, 1.6, 1.9, 1.4, 1.6, 1.4, 1.6, 1.9, 55.6, 1.9, 1.6, 1.8, 1.6, 1.6, 1.6, 1.6, 1.6, 1.6, 1.6, 1.6, 1.8, 1.6, 43.6]
CPU freq. (MHz): 2379
Load avg. in last 1, 5, 15 mins (%): [0.6, 1.6, 5.4]
Disk usage (%): 13.9
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 125.7
Available memory (GB): 115.2
Used memory (GB): 9.3

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 13.0
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: 1
cuDNN version: 91200
Current device: 0
Library compiled for CUDA architectures: ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120', 'compute_120']
GPU 0 Name: NVIDIA GeForce RTX 5090
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 170
GPU 0 Total memory (GB): 31.4
GPU 0 CUDA capability (maj.min): 12.0