Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training Terminated due to NaN in tensors #79

Open
a6375336 opened this issue Nov 20, 2024 · 1 comment
Open

Training Terminated due to NaN in tensors #79

a6375336 opened this issue Nov 20, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@a6375336
Copy link

Describe the bug
Hello everyone. First, I'd like to thank the developers of Legged Gym, with this RL implementation I have successfully trained multiple robots in the simulation environment .
But a problem occurred when I introduced a new robot. During training, the robot inexplicably starts flying, followed by training termination with an error message indicating "NaN in tensors" or "ValueError". This issue has been bothering me for a long time. I have tried some methods, but it is not fully solved yet.

The methods I have tried are:

1.reset the robot when its body speed becomes too large, but this approach affects the training and sometimes causes the robot to enter a cycle of termination and restart.
2. Checking the URDF file. I have confirmed that the velocity and effort bounds corresponding to each joint are properly limited in my URDF file. Some people online also mentioned that this issue could be caused by incorrect inertia parameters in URDF files exported from SolidWorks, but even after resetting the inertia parameters based on SolidWorks' mass inspection function, the issue remains unresolved.
3. Adjusting the reward function: I have adjusted the reward function for a while, but this does not seem to solve the problem.

Is there any good solution for this? Thank you.

To Reproduce
Steps to reproduce the behavior:

  1. Execute ./path/script arg1 arg2
  2. Then ...
  3. See error: ...

Expected behavior
A clear and concise description of what you expected to happen.

System (please complete the following information):

  • Commit: [e.g. 8f3b9ca]
  • OS: Ubuntu 20.04
  • GPU: RTX 4080 Super
  • CUDA: 12.4
  • GPU Driver: 550.54

Additional context
Add any other context about the problem here.

@a6375336 a6375336 added the bug Something isn't working label Nov 20, 2024
@hyspirnov
Copy link

I have the same problem. Have you solved it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants