Skip to content

Dispose implicitly converted TorchSharp.Scalar and torch.Tensor #1496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 101 commits into
base: main
Choose a base branch
from
Draft

Dispose implicitly converted TorchSharp.Scalar and torch.Tensor #1496

wants to merge 101 commits into from

Conversation

hiyuh
Copy link
Contributor

@hiyuh hiyuh commented Sep 10, 2025 •
edited
Loading

  • SSIA.
    • included commits from Dispose Scalars implicitly created in Tensor operators #1434.
    • newly introduced TorchSharp.{Scalar,Tensor}LeakDetector can throw exceptions on implicit conversion.
      • if enabled, it would show stack trace for possible missing {TorchSharp.Scalar,torch.Tensor}.Dispose.
      • i'm not sure whether these leak detectors need retain even after all fixes in TorchSharp have been made though.
    • real fix for missing {TorchSharp.Scalar,torch.Tensor}.Dispose in TorchSharp is on going.
  • CC: @ds5678

masaru-kimura-hacarus and others added 16 commits September 10, 2025 15:18
@masaru-kimura-hacarus
SixLabors.ImageSharp version bump.
2ff8ec4 
Otherwise, dotnet build will fail.
@ds5678 @masaru-kimura-hacarus
Dispose Scalars implicitly created in Tensor operators
5b821cb
@ds5678 @masaru-kimura-hacarus
Use Scalar operators in the primitive overloads for clarity and maint…
069bb0d 
…ainability
@masaru-kimura-hacarus
Introduce TorchSharp.ScalarLeakDetector.
5c79729 
In the following case, at least 266 exceptions are observed.
* allowImplicitConversionOperator = false
* dotnet test /p:SkipCuda=true /p:SkipNetFxBuild=true --blame test\TorchSharpTest\TorchSharpTest.csproj -c Release

* Update src/TorchSharp/Scalar.cs.
  + Introduce ScalarLeakDetector.
  + Update Scalar.
    - Use ScalarLeakDetector.ThrowIfImplicitConversionNotAllowed.
@masaru-kimura-hacarus
Introduce TorchSharp.TensorLeakDetector.
fb0ff43 
In the following case, at least 45 exceptions are observed.
* allowImplicitConversionOperator = false
* dotnet test /p:SkipCuda=true /p:SkipNetFxBuild=true --blame test\TorchSharpTest\TorchSharpTest.csproj -c Release

* Update src/TorchSharp/Tensor/Tensor.cs
  + Introduce TensorLeakDetector.
  + Update Tensor.
    - Use TensorLeakDetector.ThrowIfImplicitConversionNotAllowed.
@masaru-kimura-hacarus
Declare TorchSharp.Scalar more explicitly.
512a2ba 
* Update src/TorchSharp/Tensor/Tensor.Operators.cs.
  + Declare TorchSharp.Scalar more explicitly.
    - Use prefix for left or right.
    - Call ToScalar explicitly.
@masaru-kimura-hacarus
Declare TorchSharp.Scalar explicitly.
06b9e45 
* Update src/TorchSharp/Tensor/Tensor.Math.cs.
  + Declare TorchSharp.Scalar explicitly.
  + Add FIXMEs.
@masaru-kimura-hacarus
Update Adadelta.step.
0fb172d 
* Update src/TorchSharp/Optimizers/Adadelta.cs.
  + Update Adadelta.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0 explicitly.
@masaru-kimura-hacarus
Update Adagrad.step.
645523c 
* Update src/TorchSharp/Optimizers/Adagrad.cs.
  + Update Adagrad.step.
    + Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    + Add FIXME for possible unsued initial_accumulator_value.
    + Cache weight_decay != 0.
@masaru-kimura-hacarus
Update Adam.step.
3f0d806 
* Update src/TorchSharp/Optimizers/Adam.cs.
  + Update Adam.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0.
    - Add FIXME for possible no denom disposing.
@masaru-kimura-hacarus
Update Adamax.step.
19d87cc 
* Update src/TorchSharp/Optimizers/Adamax.cs.
  + Update Adamax.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0.
    - Add FIXME for CA1806.
@masaru-kimura-hacarus
Update ASGD.step.
68f00b2 
* Update src/TorchSharp/Optimizers/ASGD.cs.
  + Update ASGD.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unsed weight_decay_scalar.
    - Cache weight_decay != 0.
@masaru-kimura-hacarus
Update NAdam.step.
4cc71c6 
* Update src/TorchSharp/Optimizers/NAdam.cs.
  + Update NAdam.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0.
    - Add FIXME for possible no denom disposing.
@masaru-kimura-hacarus
Update RAdam.step.
997c679 
* Update src/TorchSharp/Optimizers/RAdam.cs.
  + Update RAdam.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache weight_decay != 0.
    - Add FIXME for possible torch.Tensor.sub_ use.
    - Add FIXME for possible no dispose for torch.Tensor.
      - bias_corrected_exp_avg
      - t6
      - adaptive_lr and its intermediates and derives
    - Add FIXME for possible no dispose on param.add_ if rho_t > 5.
@masaru-kimura-hacarus
Update RMSProp.step.
bc130ff 
* Update src/TorchSharp/Optimizers/RMSprop.cs.
  + Update RMSProp.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for possible unused momentum_scalar.
      - Add FIXME for possible unused weight_decay_scalar.
    - Cache momentum > 0.
    - Cache weight_decay != 0.
    - Add FIXME for possible no avg dispose.
@masaru-kimura-hacarus
Update SGD.step.
64520bc 
* Update src/TorchSharp/Optimizers/SGD.cs.
  + Update SGD.step.
    - Declare TorchSharp.Scalar explicitly.
    - Cache momentum != 0.
    - Cache dampening != 1.
    - Cache weight_decay != 0.
    - Omit unused TorchSharp.Scalar construction.
@hiyuh hiyuh force-pushed the dispose-implicitly-converted-scalar-tensor branch from 4a7cbd2 to 64520bc Compare September 11, 2025 09:04
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 11, 2025 •
edited
Loading

  • nuked torch.Tensor.add(torch.Tensor, Scalar) w/ implicit conversion to TorchSharp.Scalar.
    image
  • i'll continue same kind fix based on "Find All References" by Visual Studio.

@masaru-kimura-hacarus
Update griffinlim.
2f3c9e3 
* Update src/TorchAudio/Functional.cs.
  + Update griffinlim.
    - Declare TorchSharp.Scalar explicitly.
      - Introduce eps_scalar.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 11, 2025 •
edited
Loading

  • nuked torch.Tensor.add(Scalar) w/ implicit conversion to TorchSharp.Scalar.
    image
    • intentionally, i won't touch any unit tests if possible, since leaking due to test code incorrectness is not critical, as long as CI/CD passed.

@masaru-kimura-hacarus
Update AdamW.step.
d168161 
* Update src/TorchSharp/Optimizers/AdamW.cs.
  + Update AdamW.step.
    - Declare TorchSharp.Scalar explicitly.
    - Dispose denom explicitly.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 11, 2025

  • ditto for torch.Tensor.add_(torch.Tensor, Scalar).
    image

@masaru-kimura-hacarus
Update BatchNorm.forward.
90f4065 
* Update src/TorchSharp/NN/Normalization/BatchNorm.cs.
  + Update BatchNorm.forward.
    - Declare TorchSharp.Scalar explicitly.
    - Add FIXME for cache over training.
@masaru-kimura-hacarus
Update torch.normal.
a86d880 
* Update src/TorchSharp/Tensor/Factories/Tensor.Factories.cs.
  + Update torch.normal.
    - Declare TorchSharp.Scalar explicitly.
@masaru-kimura-hacarus
Update torchvision.utils.save_image.
6e54694 
* Update src/TorchVision/Utils.cs.
  + Update torchvision.utils.save_image.
    - Declare TorchSharp.Scalar explicitly.
    - Add FIXME for possible torch.Tensor.round_ use.
    - Add FIXME for no torch.min_int_value.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 12, 2025

  • ditto for torch.Tensor.add_(Scalar).
    image
  • CI/CD errors on Azure DevOps are irrelevant to this MR.
    • Windows_x64_NetFX Release_Build fails on "Run Tests" due to network connection closed by remote host.
    • MacOS_arm64 Release_Build fails on "Build" due to cmake_minimum_required doesn't met.

@masaru-kimura-hacarus
Update Rprop.step.
67300a9 
* Update src/TorchSharp/Optimizers/Rprop.cs.
  + Update Rprop.step.
    - Declare TorchSharp.Scalar explicitly.
      - Add FIXME for unused lr.
    - Add FIXME for possible torch.Tensor.sign_ use.
    - Cache eta{minus,plus} and 1 as torch.Tensor.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 12, 2025

  • ditto for torch.Tensor.addcmul_(Tensor, Tensor, Scalar).
    image

@masaru-kimura-hacarus
Update torchvision.ops.stochastic_depth.
f252383 
* Update src/TorchVision/Ops/StochasticDepth.cs.
  + Update torchvision.ops.stochastic_depth.
    - Declare TorchSharp.Scalar explicitly.
@masaru-kimura-hacarus
Update torchvision.utils.make_grid.
1523386 
* Update src/TorchVision/Utils.cs.
  + Update torchvision.utils.make_grid.
    - Declare TorchSharp.Scalar explicitly.
@masaru-kimura-hacarus
Update torch.distributions.constraints._PositiveSemiDefinite.check.
fc1b893 
* Update src/TorchSharp/Distributions/Constraints.cs.
  + Update torch.distributions.constraints._PositiveSemiDefinite.check.
    - Declare TorchSharp.Scalar explicitly.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 16, 2025

  • ditto for torch.Tensor.ge(Scalar).
    image

@masaru-kimura-hacarus
Update torch.distributions.constraints._CorrCholesky.check.
d552c5a 
* Update src/TorchSharp/Distributions/Constraints.cs.
  + Update torch.distributions.constraints._CorrCholesky.check.
    - Declare TorchSharp.Scalar explicitly.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 16, 2025

  • ditto for torch.Tensor.le(Scalar).
    image

@masaru-kimura-hacarus
Update TransformerModel.GenerateSquareSubsequentMask.
636303a 
* Update src/Examples/SequenceToSequence.cs.
  + Update TransformerModel.GenerateSquareSubsequentMask.
    - Declare TorchSharp.Scalar explicitly.
@masaru-kimura-hacarus
Update TorchSharp.Modules.Tacotron2.forward.
6c8656a 
* Update src/TorchAudio/Modules/Tacotron2.cs.
  + Update TorchSharp.Modules.Tacotron2.forward.
    - Declare TorchSharp.Scalar explicitly.
@masaru-kimura-hacarus
Update TorchSharp.Modules.Tacotron2.Attention.forward.
bb92724 
* Update src/TorchAudio/Modules/Tacotron2.cs.
  + Update TorchSharp.Modules.Tacotron2.Attention.forward.
    - Declare TorchSharp.Scalar explicitly.
@masaru-kimura-hacarus
Update TorchSharp.Modules.NegativeBinomial.log_prob.
0858332 
* Update src/TorchSharp/Distributions/NegativeBinomial.cs.
  + Update TorchSharp.Modules.NegativeBinomial.log_prob.
    - Declare TorchSharp.Scalar explicitly.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 16, 2025

  • ditto for torch.Tensor.masked_fill(torch.Tensor, Scalar).
    image

@masaru-kimura-hacarus
Simplify.
8880435
@masaru-kimura-hacarus
Add more operators.
23f60dc 
This is an exceptional change.
Since comparison operators are widely used, I'd have to give up fix one
by one for now.
Introducing operators take {int,long,float,double} would cover most cases
to prevent missing TorchSharp.Scalar.Dispose.
However, these would leave TorchSharp.Scalar construction cost as is.

* Update src/TorchSharp/Tensor/Tensor.cs.
  + Add more operator ==.
  + Add more operator !=.
  + Add more operator <.
  + Add more operator <=.
  + Add more operator >.
  + Add more operator >=.
@masaru-kimura-hacarus
Add more operators.
f5f7c5c 
Support other types which are covered by implicit conversion to
TorchSharp.Scalar; byte, sbyte, short, Half, bool, (float, float),
System.Numerics.Complex.

* Update src/TorchSharp/Tensor/Tensor.cs.
  + Add more operator ==.
  + Add more operator !=.
  + Add more operator <.
  + Add more operator <=.
  + Add more operator >.
  + Add more operator >=.
@masaru-kimura-hacarus
Call PrintValue w/ explicitly declared TorchSharp.Scalar.
c2da2a7 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Call PrintValue w/ explicitly declared TorchSharp.Scalar.
@masaru-kimura-hacarus
Update TensorExtensionMethods.To*(this Tensor value).
2c469da 
* Update src/TorchSharp/Tensor/TensorExtensionMethods.cs.
  + Update TensorExtensionMethods.To*(this Tensor value).
    - Declare TorchSharp.Scalar explicitly.
@masaru-kimura-hacarus
Introduce torch.Tensor.fill_ overloads.
f52c6c4 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.fill_ overloads.
@masaru-kimura-hacarus
Update TorchSharp.Modules.Rprop.
cee753b 
* Update src/TorchSharp/Optimizers/Rprop.cs.
  + Update TorchSharp.Modules.Rprop.step.
    - Cosmetic.
    - Use torch.Tensor.masked_fill_.
  + Update TorchSharp.Modules.Rprop.State.Initialize.
    - Use a torch.Tensor.fill_ overload.
@masaru-kimura-hacarus
Introduce torch.Tensor.index_put_ overloads.
b077e6a 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.index_put_ overloads.
@masaru-kimura-hacarus
Introduce torch.Tensor.index_add{,_} overloads.
ea78740 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.index_add{,_} overloads.
@masaru-kimura-hacarus
Introduce torch.Tensor.index_fill{,_} overloads.
16f7e9b 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.index_fill{,_} overloads.
@masaru-kimura-hacarus
Introduce torch.Tensor.threshold{,_} overloads.
effda51 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.threshold{,_} overloads.
@masaru-kimura-hacarus
Update torch.nn.functional.threshold.
48281ed 
* Update src/TorchSharp/NN/Activation/Threshold.cs.
  + Update torch.nn.functional.threshold.
    - Use torch.Tensor.threshold{,_} overloads.
@masaru-kimura-hacarus
Introduce torch.Tensor.softplus overloads.
e90139f 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.softplus overloads.
@masaru-kimura-hacarus
Add more torch.Tensor.celu{,_} overloads.
1287b2d 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Add more torch.Tensor.celu{,_} overloads.
@masaru-kimura-hacarus
Update torch.nn.functional.celu.
3da9849 
* Update src/TorchSharp/NN/Activation/CELU.cs.
  + Update torch.nn.functional.celu.
    - Use torch.Tensor.celu{,_} overloads.
@masaru-kimura-hacarus
Add more torch.Tensor.elu{,_} overloads.
2a2b66e 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Add more torch.Tensor.elu{,_} overloads.
@masaru-kimura-hacarus
Introduce torch.Tensor.hardtanh{,_} overloads.
41a0db8 
* Update src/TorchSharp/NN/Activation/Hardtanh.cs.
  + Introduce torch.Tensor.hardtanh{,_} overloads.
@masaru-kimura-hacarus
Update torch.nn.functional.hardtanh.
b230428 
* Update src/TorchSharp/NN/Activation/Hardtanh.cs.
  + Update torch.nn.functional.hardtanh.
    - Use torch.Tensor.hardtanh{,_} overloads.
@masaru-kimura-hacarus
Introduce torch.Tensor.leaky_relu{,_} overloads.
789555b 
* Update src/TorchSharp/Tensor/Tensor.cs.
  + Introduce torch.Tensor.leaky_relu{,_} overloads.
@masaru-kimura-hacarus
Update torch.nn.functional.leaky_relu.
48e0ca8 
* Update src/TorchSharp/NN/Activation/LeakyReLU.cs.
  + Update torch.nn.functional.leaky_relu.
    - Use torch.Tensor.leaky_relu{,_} overloads.
@hiyuh
Copy link
Contributor Author

hiyuh commented Sep 18, 2025 •
edited
Loading

  • introducing overloads on callee side, rather than fix on caller side.
    • although it might be suboptimal (sometimes, caller side could be reorganized with explicitly cached scalars/tensors), changeset would be consist (lower review cost) and conservative (more robust against uncareful caller side).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hiyuh @masaru-kimura-hacarus @ds5678