[Feature] SAC Trainer

[vmoens]

Stack from ghstack (oldest at bottom):

• [Quality] Fix flaky test #3211
• [Feature] Documentation #3192
• -> [Feature] SAC Trainer #3191
• [Feature] PPO Trainer Updates #3190
• [Feature] Trainer Algorithms - Configuration System #3189
• [Feature] Trainer Infrastructure - Timing and Utilities #3188
• [BugFix] Fix GRPO tests and runs #3213

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3191

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm][CI] Machines under the label linux.rocm.gpu.2 are undergoing maintenance.

❌ 6 New Failures, 10 Unrelated Failures

As of commit 70d35e9 with merge base e7ec9c3 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 139.
• Generate documentation / build-docs (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 317b96c94f68985fce7d22ed8e0afb3b88362fc9c194bee01d97bf19751350ba /exec failed with exit code 2
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t f54a32620870d4c9557cc544fe36275ec2dd143d6b3646bba1cf80eae7a7b204 /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 2522ae95083f1d7c9568c9b13f0762cc41f0f9c02a21809e3fa6fc92178e9a82 /exec failed with exit code 1
• Libs Tests on Linux / unittests-sklearn (3.10, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t a1176c92597aa53f4733a1c64c98cba70f2639d97af294abbb030dd94d8ccf23 /exec failed with exit code 2
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 54d1df45eb1f11bca96c35547e6e1fc6a8935731750eb559d0790da25d20d45d /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh) (trunk failure)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh) (trunk failure)
  Process completed with exit code 2.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}26$. Worsened: $\large\color{#d91a1a}4$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_tensor_to_bytestream_speed[pickle]	80.6714μs	79.1547μs	12.6335 KOps/s	12.5415 KOps/s	$\color{#35bf28}+0.73\%$
test_tensor_to_bytestream_speed[torch.save]	0.1357ms	0.1343ms	7.4487 KOps/s	7.3524 KOps/s	$\color{#35bf28}+1.31\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1198s	0.1195s	8.3663 Ops/s	8.3298 Ops/s	$\color{#35bf28}+0.44\%$
test_tensor_to_bytestream_speed[numpy]	2.8442μs	2.8380μs	352.3577 KOps/s	352.0813 KOps/s	$\color{#35bf28}+0.08\%$
test_tensor_to_bytestream_speed[safetensors]	43.1625μs	42.7261μs	23.4049 KOps/s	23.5025 KOps/s	$\color{#d91a1a}-0.42\%$
test_simple	0.5264s	0.5256s	1.9024 Ops/s	1.7542 Ops/s	$\textbf{\color{#35bf28}+8.45\%}$
test_transformed	1.1727s	1.0814s	0.9247 Ops/s	0.8979 Ops/s	$\color{#35bf28}+2.99\%$
test_serial	1.5979s	1.5947s	0.6271 Ops/s	0.6064 Ops/s	$\color{#35bf28}+3.40\%$
test_parallel	1.1803s	1.0616s	0.9420 Ops/s	0.9237 Ops/s	$\color{#35bf28}+1.98\%$
test_step_mdp_speed[True-True-True-True-True]	0.2615ms	41.9271μs	23.8509 KOps/s	23.0585 KOps/s	$\color{#35bf28}+3.44\%$
test_step_mdp_speed[True-True-True-True-False]	62.9920μs	24.0001μs	41.6665 KOps/s	41.4993 KOps/s	$\color{#35bf28}+0.40\%$
test_step_mdp_speed[True-True-True-False-True]	80.6920μs	24.2354μs	41.2619 KOps/s	41.6427 KOps/s	$\color{#d91a1a}-0.91\%$
test_step_mdp_speed[True-True-True-False-False]	38.9410μs	13.3450μs	74.9346 KOps/s	75.0579 KOps/s	$\color{#d91a1a}-0.16\%$
test_step_mdp_speed[True-True-False-True-True]	85.7820μs	45.7489μs	21.8585 KOps/s	21.8385 KOps/s	$\color{#35bf28}+0.09\%$
test_step_mdp_speed[True-True-False-True-False]	58.7410μs	26.5692μs	37.6375 KOps/s	37.6296 KOps/s	$\color{#35bf28}+0.02\%$
test_step_mdp_speed[True-True-False-False-True]	62.1820μs	26.8044μs	37.3073 KOps/s	37.6168 KOps/s	$\color{#d91a1a}-0.82\%$
test_step_mdp_speed[True-True-False-False-False]	51.6810μs	16.0588μs	62.2713 KOps/s	62.8407 KOps/s	$\color{#d91a1a}-0.91\%$
test_step_mdp_speed[True-False-True-True-True]	0.1057ms	49.4537μs	20.2209 KOps/s	20.4370 KOps/s	$\color{#d91a1a}-1.06\%$
test_step_mdp_speed[True-False-True-True-False]	63.1310μs	29.2380μs	34.2021 KOps/s	34.1886 KOps/s	$\color{#35bf28}+0.04\%$
test_step_mdp_speed[True-False-True-False-True]	64.8920μs	26.8886μs	37.1905 KOps/s	37.4942 KOps/s	$\color{#d91a1a}-0.81\%$
test_step_mdp_speed[True-False-True-False-False]	47.3310μs	15.9201μs	62.8137 KOps/s	63.7036 KOps/s	$\color{#d91a1a}-1.40\%$
test_step_mdp_speed[True-False-False-True-True]	88.7220μs	50.5647μs	19.7766 KOps/s	19.6719 KOps/s	$\color{#35bf28}+0.53\%$
test_step_mdp_speed[True-False-False-True-False]	63.5720μs	32.0598μs	31.1917 KOps/s	31.7280 KOps/s	$\color{#d91a1a}-1.69\%$
test_step_mdp_speed[True-False-False-False-True]	46.7210μs	29.3916μs	34.0234 KOps/s	34.1455 KOps/s	$\color{#d91a1a}-0.36\%$
test_step_mdp_speed[True-False-False-False-False]	57.3710μs	18.5345μs	53.9534 KOps/s	54.8338 KOps/s	$\color{#d91a1a}-1.61\%$
test_step_mdp_speed[False-True-True-True-True]	83.2320μs	47.5772μs	21.0184 KOps/s	20.6104 KOps/s	$\color{#35bf28}+1.98\%$
test_step_mdp_speed[False-True-True-True-False]	67.5610μs	29.1937μs	34.2540 KOps/s	33.9019 KOps/s	$\color{#35bf28}+1.04\%$
test_step_mdp_speed[False-True-True-False-True]	2.6859ms	30.3961μs	32.8989 KOps/s	32.0950 KOps/s	$\color{#35bf28}+2.50\%$
test_step_mdp_speed[False-True-True-False-False]	49.6310μs	17.5125μs	57.1022 KOps/s	56.8441 KOps/s	$\color{#35bf28}+0.45\%$
test_step_mdp_speed[False-True-False-True-True]	91.2230μs	50.5487μs	19.7829 KOps/s	19.6961 KOps/s	$\color{#35bf28}+0.44\%$
test_step_mdp_speed[False-True-False-True-False]	70.0510μs	32.0620μs	31.1896 KOps/s	31.5958 KOps/s	$\color{#d91a1a}-1.29\%$
test_step_mdp_speed[False-True-False-False-True]	76.3520μs	32.6796μs	30.6001 KOps/s	30.4062 KOps/s	$\color{#35bf28}+0.64\%$
test_step_mdp_speed[False-True-False-False-False]	54.3020μs	20.4608μs	48.8739 KOps/s	49.9814 KOps/s	$\color{#d91a1a}-2.22\%$
test_step_mdp_speed[False-False-True-True-True]	94.4030μs	53.7922μs	18.5901 KOps/s	18.6702 KOps/s	$\color{#d91a1a}-0.43\%$
test_step_mdp_speed[False-False-True-True-False]	69.2720μs	34.7886μs	28.7450 KOps/s	29.1518 KOps/s	$\color{#d91a1a}-1.40\%$
test_step_mdp_speed[False-False-True-False-True]	77.2020μs	32.7984μs	30.4893 KOps/s	30.4173 KOps/s	$\color{#35bf28}+0.24\%$
test_step_mdp_speed[False-False-True-False-False]	59.2820μs	20.0504μs	49.8743 KOps/s	50.1998 KOps/s	$\color{#d91a1a}-0.65\%$
test_step_mdp_speed[False-False-False-True-True]	0.1060ms	55.2416μs	18.1023 KOps/s	17.8961 KOps/s	$\color{#35bf28}+1.15\%$
test_step_mdp_speed[False-False-False-True-False]	72.9420μs	36.7992μs	27.1745 KOps/s	27.3451 KOps/s	$\color{#d91a1a}-0.62\%$
test_step_mdp_speed[False-False-False-False-True]	70.4720μs	34.6634μs	28.8489 KOps/s	28.7647 KOps/s	$\color{#35bf28}+0.29\%$
test_step_mdp_speed[False-False-False-False-False]	56.9110μs	22.4425μs	44.5584 KOps/s	44.9623 KOps/s	$\color{#d91a1a}-0.90\%$
test_values[generalized_advantage_estimate-True-True]	11.0664ms	9.9592ms	100.4099 Ops/s	100.4000 Ops/s	$+0.01\%$
test_values[vec_generalized_advantage_estimate-True-True]	13.3970ms	10.9627ms	91.2181 Ops/s	57.0516 Ops/s	$\textbf{\color{#35bf28}+59.89\%}$
test_values[td0_return_estimate-False-False]	0.2579ms	0.1311ms	7.6299 KOps/s	7.8930 KOps/s	$\color{#d91a1a}-3.33\%$
test_values[td1_return_estimate-False-False]	27.7744ms	26.7646ms	37.3628 Ops/s	36.9688 Ops/s	$\color{#35bf28}+1.07\%$
test_values[vec_td1_return_estimate-False-False]	11.2669ms	11.0957ms	90.1248 Ops/s	56.8070 Ops/s	$\textbf{\color{#35bf28}+58.65\%}$
test_values[td_lambda_return_estimate-True-False]	41.0581ms	40.1197ms	24.9254 Ops/s	25.3273 Ops/s	$\color{#d91a1a}-1.59\%$
test_values[vec_td_lambda_return_estimate-True-False]	11.4212ms	10.9850ms	91.0332 Ops/s	56.6284 Ops/s	$\textbf{\color{#35bf28}+60.76\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.6403ms	8.3659ms	119.5330 Ops/s	118.6605 Ops/s	$\color{#35bf28}+0.74\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.6430ms	1.4241ms	702.1967 Ops/s	675.3809 Ops/s	$\color{#35bf28}+3.97\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.4926ms	0.4102ms	2.4379 KOps/s	2.4078 KOps/s	$\color{#35bf28}+1.25\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	18.6652ms	17.9240ms	55.7910 Ops/s	33.9468 Ops/s	$\textbf{\color{#35bf28}+64.35\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	1.8754ms	1.7091ms	585.0864 Ops/s	587.0173 Ops/s	$\color{#d91a1a}-0.33\%$
test_dqn_speed[False-None]	1.6086ms	1.3855ms	721.7414 Ops/s	718.3807 Ops/s	$\color{#35bf28}+0.47\%$
test_dqn_speed[False-backward]	1.9723ms	1.9094ms	523.7344 Ops/s	516.6339 Ops/s	$\color{#35bf28}+1.37\%$
test_dqn_speed[True-None]	0.6231ms	0.5224ms	1.9141 KOps/s	1.9338 KOps/s	$\color{#d91a1a}-1.02\%$
test_dqn_speed[True-backward]	1.0257ms	0.9833ms	1.0169 KOps/s	864.2553 Ops/s	$\textbf{\color{#35bf28}+17.67\%}$
test_dqn_speed[reduce-overhead-None]	0.6496ms	0.5226ms	1.9134 KOps/s	1.8937 KOps/s	$\color{#35bf28}+1.04\%$
test_dqn_speed[reduce-overhead-backward]	0.9751ms	0.9401ms	1.0637 KOps/s	872.0748 Ops/s	$\textbf{\color{#35bf28}+21.98\%}$
test_ddpg_speed[False-None]	3.1368ms	2.8097ms	355.9057 Ops/s	351.9829 Ops/s	$\color{#35bf28}+1.11\%$
test_ddpg_speed[False-backward]	4.4197ms	4.0813ms	245.0206 Ops/s	247.5133 Ops/s	$\color{#d91a1a}-1.01\%$
test_ddpg_speed[True-None]	1.4443ms	1.3798ms	724.7169 Ops/s	686.0367 Ops/s	$\textbf{\color{#35bf28}+5.64\%}$
test_ddpg_speed[True-backward]	2.4086ms	2.3491ms	425.7035 Ops/s	353.4522 Ops/s	$\textbf{\color{#35bf28}+20.44\%}$
test_ddpg_speed[reduce-overhead-None]	1.5841ms	1.3976ms	715.5094 Ops/s	711.7057 Ops/s	$\color{#35bf28}+0.53\%$
test_ddpg_speed[reduce-overhead-backward]	2.4006ms	2.3442ms	426.5906 Ops/s	412.4856 Ops/s	$\color{#35bf28}+3.42\%$
test_sac_speed[False-None]	8.4140ms	7.7903ms	128.3649 Ops/s	126.0842 Ops/s	$\color{#35bf28}+1.81\%$
test_sac_speed[False-backward]	11.8674ms	11.0958ms	90.1241 Ops/s	90.0392 Ops/s	$\color{#35bf28}+0.09\%$
test_sac_speed[True-None]	2.2322ms	2.0907ms	478.3029 Ops/s	472.0298 Ops/s	$\color{#35bf28}+1.33\%$
test_sac_speed[True-backward]	4.1851ms	3.9986ms	250.0866 Ops/s	245.2250 Ops/s	$\color{#35bf28}+1.98\%$
test_sac_speed[reduce-overhead-None]	2.2006ms	2.0627ms	484.7932 Ops/s	451.8625 Ops/s	$\textbf{\color{#35bf28}+7.29\%}$
test_sac_speed[reduce-overhead-backward]	4.1448ms	3.9970ms	250.1867 Ops/s	232.2573 Ops/s	$\textbf{\color{#35bf28}+7.72\%}$
test_redq_speed[False-None]	10.6238ms	10.1174ms	98.8393 Ops/s	96.9835 Ops/s	$\color{#35bf28}+1.91\%$
test_redq_speed[False-backward]	18.3954ms	17.5507ms	56.9776 Ops/s	53.8780 Ops/s	$\textbf{\color{#35bf28}+5.75\%}$
test_redq_speed[True-None]	5.6580ms	4.4952ms	222.4582 Ops/s	219.5937 Ops/s	$\color{#35bf28}+1.30\%$
test_redq_speed[True-backward]	10.0427ms	9.7083ms	103.0045 Ops/s	105.9980 Ops/s	$\color{#d91a1a}-2.82\%$
test_redq_speed[reduce-overhead-None]	4.5451ms	4.3092ms	232.0601 Ops/s	231.5308 Ops/s	$\color{#35bf28}+0.23\%$
test_redq_speed[reduce-overhead-backward]	10.1608ms	9.8956ms	101.0551 Ops/s	99.5781 Ops/s	$\color{#35bf28}+1.48\%$
test_redq_deprec_speed[False-None]	11.3553ms	10.8423ms	92.2311 Ops/s	93.0959 Ops/s	$\color{#d91a1a}-0.93\%$
test_redq_deprec_speed[False-backward]	16.3232ms	15.7207ms	63.6103 Ops/s	65.1193 Ops/s	$\color{#d91a1a}-2.32\%$
test_redq_deprec_speed[True-None]	4.2830ms	3.5899ms	278.5615 Ops/s	270.5684 Ops/s	$\color{#35bf28}+2.95\%$
test_redq_deprec_speed[True-backward]	7.8131ms	7.5787ms	131.9492 Ops/s	122.7433 Ops/s	$\textbf{\color{#35bf28}+7.50\%}$
test_redq_deprec_speed[reduce-overhead-None]	3.8149ms	3.5140ms	284.5750 Ops/s	271.1116 Ops/s	$\color{#35bf28}+4.97\%$
test_redq_deprec_speed[reduce-overhead-backward]	7.6538ms	7.4276ms	134.6325 Ops/s	127.4573 Ops/s	$\textbf{\color{#35bf28}+5.63\%}$
test_td3_speed[False-None]	8.2070ms	7.8247ms	127.8006 Ops/s	125.7729 Ops/s	$\color{#35bf28}+1.61\%$
test_td3_speed[False-backward]	11.1808ms	10.7716ms	92.8365 Ops/s	91.4930 Ops/s	$\color{#35bf28}+1.47\%$
test_td3_speed[True-None]	1.8327ms	1.7958ms	556.8469 Ops/s	541.3927 Ops/s	$\color{#35bf28}+2.85\%$
test_td3_speed[True-backward]	4.2259ms	3.6554ms	273.5654 Ops/s	260.6514 Ops/s	$\color{#35bf28}+4.95\%$
test_td3_speed[reduce-overhead-None]	2.1638ms	1.7653ms	566.4641 Ops/s	554.8151 Ops/s	$\color{#35bf28}+2.10\%$
test_td3_speed[reduce-overhead-backward]	3.7955ms	3.6561ms	273.5163 Ops/s	268.3389 Ops/s	$\color{#35bf28}+1.93\%$
test_cql_speed[False-None]	29.3997ms	25.9035ms	38.6049 Ops/s	39.6927 Ops/s	$\color{#d91a1a}-2.74\%$
test_cql_speed[False-backward]	39.2831ms	35.5589ms	28.1223 Ops/s	29.1317 Ops/s	$\color{#d91a1a}-3.46\%$
test_cql_speed[True-None]	12.6806ms	12.3208ms	81.1637 Ops/s	81.4083 Ops/s	$\color{#d91a1a}-0.30\%$
test_cql_speed[True-backward]	18.6856ms	18.0671ms	55.3493 Ops/s	54.1960 Ops/s	$\color{#35bf28}+2.13\%$
test_cql_speed[reduce-overhead-None]	12.7489ms	12.3473ms	80.9896 Ops/s	80.4461 Ops/s	$\color{#35bf28}+0.68\%$
test_cql_speed[reduce-overhead-backward]	18.2968ms	17.7222ms	56.4265 Ops/s	56.1930 Ops/s	$\color{#35bf28}+0.42\%$
test_a2c_speed[False-None]	6.1419ms	5.2809ms	189.3617 Ops/s	182.6623 Ops/s	$\color{#35bf28}+3.67\%$
test_a2c_speed[False-backward]	12.0545ms	11.6879ms	85.5589 Ops/s	84.3936 Ops/s	$\color{#35bf28}+1.38\%$
test_a2c_speed[True-None]	3.9153ms	3.6735ms	272.2198 Ops/s	248.8522 Ops/s	$\textbf{\color{#35bf28}+9.39\%}$
test_a2c_speed[True-backward]	8.7313ms	8.3696ms	119.4805 Ops/s	114.4853 Ops/s	$\color{#35bf28}+4.36\%$
test_a2c_speed[reduce-overhead-None]	4.0206ms	3.6513ms	273.8720 Ops/s	275.7694 Ops/s	$\color{#d91a1a}-0.69\%$
test_a2c_speed[reduce-overhead-backward]	8.9612ms	8.6599ms	115.4748 Ops/s	111.5953 Ops/s	$\color{#35bf28}+3.48\%$
test_ppo_speed[False-None]	6.2649ms	5.8079ms	172.1778 Ops/s	171.8123 Ops/s	$\color{#35bf28}+0.21\%$
test_ppo_speed[False-backward]	12.6728ms	12.2893ms	81.3714 Ops/s	80.7714 Ops/s	$\color{#35bf28}+0.74\%$
test_ppo_speed[True-None]	3.8867ms	3.5706ms	280.0667 Ops/s	260.2639 Ops/s	$\textbf{\color{#35bf28}+7.61\%}$
test_ppo_speed[True-backward]	8.6240ms	8.3329ms	120.0061 Ops/s	110.7808 Ops/s	$\textbf{\color{#35bf28}+8.33\%}$
test_ppo_speed[reduce-overhead-None]	4.0017ms	3.5879ms	278.7150 Ops/s	275.5451 Ops/s	$\color{#35bf28}+1.15\%$
test_ppo_speed[reduce-overhead-backward]	8.8607ms	8.5274ms	117.2688 Ops/s	110.3347 Ops/s	$\textbf{\color{#35bf28}+6.28\%}$
test_reinforce_speed[False-None]	4.8827ms	4.5001ms	222.2165 Ops/s	216.8863 Ops/s	$\color{#35bf28}+2.46\%$
test_reinforce_speed[False-backward]	7.6573ms	7.3193ms	136.6243 Ops/s	134.0167 Ops/s	$\color{#35bf28}+1.95\%$
test_reinforce_speed[True-None]	3.1420ms	2.8209ms	354.4930 Ops/s	338.9339 Ops/s	$\color{#35bf28}+4.59\%$
test_reinforce_speed[True-backward]	7.7142ms	7.5127ms	133.1076 Ops/s	130.4493 Ops/s	$\color{#35bf28}+2.04\%$
test_reinforce_speed[reduce-overhead-None]	3.1527ms	2.8063ms	356.3400 Ops/s	342.7930 Ops/s	$\color{#35bf28}+3.95\%$
test_reinforce_speed[reduce-overhead-backward]	8.0178ms	7.7476ms	129.0715 Ops/s	126.1401 Ops/s	$\color{#35bf28}+2.32\%$
test_iql_speed[False-None]	20.7151ms	19.6853ms	50.7993 Ops/s	50.4185 Ops/s	$\color{#35bf28}+0.76\%$
test_iql_speed[False-backward]	31.0613ms	30.1136ms	33.2076 Ops/s	33.1830 Ops/s	$\color{#35bf28}+0.07\%$
test_iql_speed[True-None]	8.7696ms	8.3735ms	119.4237 Ops/s	117.4421 Ops/s	$\color{#35bf28}+1.69\%$
test_iql_speed[True-backward]	16.8566ms	16.3734ms	61.0745 Ops/s	60.8276 Ops/s	$\color{#35bf28}+0.41\%$
test_iql_speed[reduce-overhead-None]	8.7893ms	8.3920ms	119.1616 Ops/s	112.9002 Ops/s	$\textbf{\color{#35bf28}+5.55\%}$
test_iql_speed[reduce-overhead-backward]	17.2781ms	16.6791ms	59.9552 Ops/s	58.5895 Ops/s	$\color{#35bf28}+2.33\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.3011ms	5.8712ms	170.3220 Ops/s	171.8836 Ops/s	$\color{#d91a1a}-0.91\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6533ms	0.3223ms	3.1031 KOps/s	3.4265 KOps/s	$\textbf{\color{#d91a1a}-9.44\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6184ms	0.3083ms	3.2438 KOps/s	3.9009 KOps/s	$\textbf{\color{#d91a1a}-16.84\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.9171ms	5.5926ms	178.8091 Ops/s	181.7843 Ops/s	$\color{#d91a1a}-1.64\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.2453ms	0.3347ms	2.9880 KOps/s	2.8235 KOps/s	$\textbf{\color{#35bf28}+5.83\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6004ms	0.2923ms	3.4208 KOps/s	3.0031 KOps/s	$\textbf{\color{#35bf28}+13.91\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.5954ms	1.3440ms	744.0290 Ops/s	719.8923 Ops/s	$\color{#35bf28}+3.35\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.5669ms	1.2654ms	790.2767 Ops/s	841.0894 Ops/s	$\textbf{\color{#d91a1a}-6.04\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	9.6417ms	5.8937ms	169.6716 Ops/s	176.6170 Ops/s	$\color{#d91a1a}-3.93\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.7568ms	0.4578ms	2.1843 KOps/s	2.0970 KOps/s	$\color{#35bf28}+4.16\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.7486ms	0.4509ms	2.2180 KOps/s	2.4727 KOps/s	$\textbf{\color{#d91a1a}-10.30\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.7210ms	5.5684ms	179.5845 Ops/s	178.0683 Ops/s	$\color{#35bf28}+0.85\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.7375ms	0.2723ms	3.6726 KOps/s	2.9377 KOps/s	$\textbf{\color{#35bf28}+25.01\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.4900ms	0.2538ms	3.9403 KOps/s	2.9292 KOps/s	$\textbf{\color{#35bf28}+34.52\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.8286ms	5.4874ms	182.2350 Ops/s	178.2802 Ops/s	$\color{#35bf28}+2.22\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7306ms	0.2674ms	3.7402 KOps/s	3.6605 KOps/s	$\color{#35bf28}+2.18\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.4515ms	0.2486ms	4.0232 KOps/s	3.9536 KOps/s	$\color{#35bf28}+1.76\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.8575ms	5.6771ms	176.1449 Ops/s	172.8919 Ops/s	$\color{#35bf28}+1.88\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.1149ms	0.4189ms	2.3871 KOps/s	2.2141 KOps/s	$\textbf{\color{#35bf28}+7.82\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.5915ms	0.3981ms	2.5119 KOps/s	2.1996 KOps/s	$\textbf{\color{#35bf28}+14.20\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	6.7412ms	5.0161ms	199.3565 Ops/s	194.7759 Ops/s	$\color{#35bf28}+2.35\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	6.2290ms	2.2193ms	450.5972 Ops/s	429.8271 Ops/s	$\color{#35bf28}+4.83\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	8.7064ms	1.1629ms	859.9011 Ops/s	846.3895 Ops/s	$\color{#35bf28}+1.60\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.5183s	15.4990ms	64.5201 Ops/s	53.6788 Ops/s	$\textbf{\color{#35bf28}+20.20\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	9.0271ms	1.8923ms	528.4624 Ops/s	524.9186 Ops/s	$\color{#35bf28}+0.68\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	7.8013ms	1.1505ms	869.1586 Ops/s	873.0179 Ops/s	$\color{#d91a1a}-0.44\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	7.0259ms	5.2504ms	190.4610 Ops/s	184.1029 Ops/s	$\color{#35bf28}+3.45\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	8.7896ms	2.1633ms	462.2476 Ops/s	452.3338 Ops/s	$\color{#35bf28}+2.19\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	7.3526ms	1.3209ms	757.0475 Ops/s	752.7150 Ops/s	$\color{#35bf28}+0.58\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	34.1230ms	31.9031ms	31.3449 Ops/s	31.0392 Ops/s	$\color{#35bf28}+0.98\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	18.1743ms	16.7420ms	59.7299 Ops/s	57.8253 Ops/s	$\color{#35bf28}+3.29\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	35.2132ms	32.6550ms	30.6231 Ops/s	29.9079 Ops/s	$\color{#35bf28}+2.39\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.3486ms	17.3169ms	57.7470 Ops/s	56.2532 Ops/s	$\color{#35bf28}+2.66\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	36.8354ms	34.2904ms	29.1627 Ops/s	28.7273 Ops/s	$\color{#35bf28}+1.52\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	19.2885ms	18.1712ms	55.0321 Ops/s	52.8890 Ops/s	$\color{#35bf28}+4.05\%$