Improper Handling of Guillemets (»«) cause hallucinations

[clemcer]

I believe this type of quotaton mark (»«) seems to trigger hallucinations somewhat consistently.
At least in German but probably other languages as well.
This type of quotation mark (»«), known as Guillemets or Chevrons, is not commonly used in English, at least ChatGPT told me so lol, so maybe the code doesnt account for them?

Here is an example video, the red lines represent hallucinations.

hallucinations.mp4

Here is the text file:
german_hallucinations_1.txt

[ROBERT-MCDOWELL]

no, it's not the double quotes making hallucinations, it's more the double quotes and the whole empty space + new lines making troubles...

[ROBERT-MCDOWELL]

could you provide the real original text so I can check if it's space, tabs or else....
usually the doubl quotation marks are opening, then text then closing. In German it seems the opposite right? or it's another fancy book where they decided to play with the quotation marks?

[clemcer]

I edited my original post and added the text file.

And I made another video from the text I uploaded on another issue.

german_hallucinations_2.mp4

If you want only the text visible in the video here it is:
german_hallucinations_2.txt

usually the doubl quotation marks are opening, then text then closing. In German it seems the opposite right? or it's another fancy book where they decided to play with the quotation marks?

No, this type of quotation mark is pretty common in German, if not standard, at least for novels. I went through my ebooks and havent found one where these quotation marks aren't being used.

And just as a heads up I even found one where they are reversed:

«Passen Sie auf, wo Sie hintreten, Dr. Hunter»,

edit:
I kind of read over that part

usually the doubl quotation marks are opening, then text then closing

and thought you were asking about the use of this type of quotation mark in general but to give a more precise answer: In German these quotation mark are almost exclusively used with their tips pointing inward like this (»«} instead of («»} although I have found one ebook where they are being used like this («»}. (but that is definetly the exception).

[ROBERT-MCDOWELL]

ok I'm going to make some tests to see if my patch works

[ROBERT-MCDOWELL]

Tell me if the result of the audio from german_hallucinations_2.txt is ok for you
german_hallucinations_2.zip

[clemcer]

No, the hallucinations are still there unfortunately :(

[ROBERT-MCDOWELL]

but is it better or exactly the same?

[clemcer]

about the same. The hallucinations are in the same places

[ROBERT-MCDOWELL]

ok, I'm working on it.

[ROBERT-MCDOWELL]

this one seems better no?
german_hallucinations_2.zip

[ROBERT-MCDOWELL]

FYI, the first word is "unknown" which is normal since there is no title.

[clemcer]

it's a little bit better, yes! If you look at my video the first two hallucinations and the last one are gone, the other two are still there.

Maybe there are still three hallucinations. The fourth pink mark with which I highlighted the hallucinations in the video could be one long hallucination but there seem to be two separate hallucinations for both the end of the sentence and the beginning of the next one.

[ROBERT-MCDOWELL]

ok keep in mind that the default voice used is from English. so be aware that glitches can occur.
if you use your custom voice with good quality sound and Deutsch pronunciation so the result will be better.
hallucinations are strange words. if it's small glitches it's something else

[clemcer]

the "strange words" are definitely too long to be small glitches though. I am pretty sure these are still proper hallucinations.

Where I put HERE there are still hallucinations:

wohl von dieser Schule geworfen werden würde.
»Ganz ruhig.« Meine Mom klang überhaupt nicht ruhig. HERE »Das ist schließlich nur ein Schnuppertreffen. Und denk daran, Lieber, das hier ist Pauls Schule. Also versuch bitte, sie nicht zu … du weißt schon.« HERE
HERE »Zerstören?«
»Ja.«

[ROBERT-MCDOWELL]

the converted text is this:
wohl von dieser Schule geworfen werden würde.
"Ganz ruhig." Meine Mom klang überhaupt nicht ruhig. "Das ist schließlich nur ein Schnuppertreffen. Und denk daran, Lieber, das hier ist Pauls Schule. Also versuch bitte, sie nicht zu ... du weißt schon."
"Zerstören?"
"Ja."
Paul Blofis, der Freund meiner Mutter, stand vor

is it still buggy?

[ROBERT-MCDOWELL]

I wonder if it does not come from the fact there is no space between the guillemets and the word....

[clemcer]

the converted text is this: wohl von dieser Schule geworfen werden würde. "Ganz ruhig." Meine Mom klang überhaupt nicht ruhig. "Das ist schließlich nur ein Schnuppertreffen. Und denk daran, Lieber, das hier ist Pauls Schule. Also versuch bitte, sie nicht zu ... du weißt schon." "Zerstören?" "Ja." Paul Blofis, der Freund meiner Mutter, stand vor

is it still buggy?

I am going to try this tomorrow, don't know why I didn't think of trying it with these "" earlier

[ROBERT-MCDOWELL]

you don't need to try, eb2ab script is doing it. all chars making hallucinations or glitches on various languages are converted with standard ascii chars.

[ROBERT-MCDOWELL]

this one seems to be better after some new punctuation replacements.
german_hallucinations_2.zip

[clemcer]

They are a bit shorter but definitely still there.

And one other thing, don't know if it's relevant but in German these quotation marks (" ") are a little bit different as well. The mark at the beginning of a sentence is placed at the bottom and the one at the end is placed at the top.
Like this:

„Hallo, so funktionieren Anführungszeichen"

If you push the update I could also do some more testing, if you need help :)

[ROBERT-MCDOWELL]

could you tell me if the last conversion is better?
doesn't matter the punctuation used, all special punctuation other than asciii must be replaced to something similar and if possible with a space before and after, or you will get hallucinations. everything is done under eb2ab filters, same for repetitive spaces, new line etc.... the update is not wrapped yet, I still have 3 important issues to fix and feature to add.

[clemcer]

You mean this audio ?
The hallucinations from here

wohl von dieser Schule geworfen werden würde.
»Ganz ruhig.« Meine Mom klang überhaupt nicht ruhig. HERE »Das ist schließlich nur ein Schnuppertreffen. Und denk daran, Lieber, das hier ist Pauls Schule. Also versuch bitte, sie nicht zu … du weißt schon.« HERE(?)
HERE »Zerstören?«
»Ja.«`

are still there but the "strange words" are a bit shorter. But that could probably also be a coincidence.

If you want you could try a longer text with more quotation marks:
harry_potter_hagrid.html.zip

[ChristopherS]

I have also experienced this and solved it by adding a search and replace to calibre and replacing («|») with ". After that there were no further hallucinations. As you convert the uploaded files into .txt anyway (if I am ot mistaken), maybe the easiest solution would be to just do a replacement on those?

[ROBERT-MCDOWELL]

@ChristopherS you should read the whole issue rather than comment without to know what it was said..... try to read before to write....

[ChristopherS]

Super sorry... Just missed the last few comments.
Knowing you already did this now, I can however confirm, that a custom model pretty much get's rid of all the hallucinations and vastly reduces glitches. Since I trained a fine tuned model on a german voice I hardly get any hallucinations anymore in (by now) over 30h of audio in german.

[ROBERT-MCDOWELL]

as you read above, I fixed it already many times... also the hallucinations comes from the voice you choose, its quality etc... but also which TTS engine, fairseq and xtts-v2 are radically different..

[clemcer]

Maybe you could share your trained voice with the devs @ChristopherS? They have mentioned before they would like to add german voices. Of course only if the voice you chose is okay to share and if you and the devs are open to it :)
You could join the discord server, that would be the easiest. I for one would be very happy if a german voice is ready with the next update.