[1.x] Janus server stalls at rooms_mutex lock in janus_videoroom.c

[ramprakash110109]

What version of Janus is this happening on?
v1.2.1

Have you tested a more recent version of Janus too?
No, since it was a rare issue and unable to reproduce again

Was this working before?
This is the second time I am facing this issue in past 6 months

Is there a gdb or libasan trace of the issue?
https://pastebin.com/eVLkvU3H

Additional context
Sessions were running smoothly for more than 3 months.
But suddenly, threads seems to get locked and no http requests or rmq requests were processed in mediaserver.
When I debugged, I came to know that rooms_mutex lock was not unlocked in one of the previous request. But I was unable to find the exact request which caused this issue.

I have pasted the debug_lock output where I hit new http request after the mediaserver gets locked due to some previous request.

I suspect one of the stop_rtp_forward request could have caused this. But I am not sure.

Could you please guide on this?

[lminiero]

Please test master, there have been a ton of fixes recently on potential deadlocks.

[ramprakash110109]

videoroom.c code is almost same inmy version too. anyways, we will try to check latest master as well. Thanks

[lminiero]

@ramprakash110109 any update?

[ramprakash110109]

We have pulled latest master and using it for past 2 weeks. We have not faced any issues till now. Thanks

…

On Thu, 12 Dec 2024 at 5:20 PM, Lorenzo Miniero ***@***.***> wrote: @ramprakash110109 <https://github.com/ramprakash110109> any update? — Reply to this email directly, view it on GitHub <#3478 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGAOGAGEXRO3MRPCUH7HY2L2FF2AFAVCNFSM6AAAAABR274JKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZYGY4DKNBVGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Dev2Trailblazer]

We have been running janus-gateway v1.3.0 for ~1 month, and we occasionally get similar behavior to this one. Randomly, one of our instances stops responding to all HTTP requests, for example:

• POST /janus/{sessionID} (attach janus.plugin.videoroom)
• POST /janus/{sessionID} (keepalive)

Instances keep failing from time to time, so it's possible that we can reproduce the issue. I'm not sure this is the same issue as this ticket. Do you have any suggestions for us on debugging it or providing more data?

Thank you!

[atoppi]

The usual steps for similar issues:

• try to reproduce with AddressSanitizer
• enable lock debugging
  Expect a downgrade in performance and an increase in disk usage if you are saving the logs.