[Feature] Weight Synchronization Schemes - Core Infrastructure

[vmoens]

Stack from ghstack (oldest at bottom):

• [Quality] Fix flaky test #3211
• [Feature] Documentation #3192
• [Feature] SAC Trainer #3191
• [Feature] PPO Trainer Updates #3190
• [Feature] Trainer Algorithms - Configuration System #3189
• [Feature] Trainer Infrastructure - Timing and Utilities #3188
• [Feature] Collectors - Weight Sync Scheme Integration #3187
• [Feature] vLLM Weight Synchronization Schemes #3186
• -> [Feature] Weight Synchronization Schemes - Core Infrastructure #3185

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3185

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [ROCm][CI] Machines under the label linux.rocm.gpu.2 are undergoing maintenance.

❌ 13 New Failures, 1 Cancelled Job, 3 Unrelated Failures

As of commit 64ab805 with merge base 4f013a8 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t a67dd4d5998e126ed60a743bee80243098bb6f8e0418b9f6b9a28fc71982f91c /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 92814ab54207f7904313c00f30b3167f7682a78daaa1fe1af108603dc807b5c5 /exec failed with exit code 1
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 78f2d817638884b50ed0950a4474eca5aa910314fb40de0ba55005777e168126 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh)
  test/test_weightsync.py::TestMultiModelUpdates::test_multi_model_tensordict_updates

CANCELLED JOB - The following job was cancelled. Please retry:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build Linux Wheels / pytorch/rl (pytorch/rl, test/smoke_test.py, torchrl, .github/scripts/pre-build-script.sh) / build-manywheel-py3_10-cuda12_9 (gh) (trunk failure)
  Process completed with exit code 1.
• Build Linux Wheels / pytorch/rl (pytorch/rl, test/smoke_test.py, torchrl, .github/scripts/pre-build-script.sh) / upload / upload-manywheel-py3_10-cuda12_9 (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.10_cu129_x86_64
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.