Need guidance on how to process audio waveform to remove filler words and sounds?

[siddhsql]

Hello,

I don't know much about speech processing but I have an audio signal and all I want to do is to process it to remove occurrences of filler words and sounds like umm/ugh etc. can someone guide me how to do this? i don't think i should be transcribing the signal (speech to text) and then re-encoding text to speech as that will result in loss of precise timestamps, intonations etc. can someone give me any pointers how to do this and what libraries are available if any? thanks.

edit: to refine my question, is it possible to use whisper to process an audio and get timestamps (start and end) of the pieces in the audio where the speech is unintelligible?

[EtienneAb3d]

You may try something like this Audio Tagger:
https://github.com/YuanGongND/whisper-at