HTML entities in input to Rouge

[lowjoel]

Continuation from #6.

In HTML, editors like to use     to preserve whitespace. Non-breaking space isn't recognised by Rouge to be same as space, so it does not recognise the syntax properly.

(look at the pink whitespace)

Maybe we should extend the "replace_br" configuration to be more general, "preprocess HTML" or some configuration setting like that, where contiguous non-breaking whitespace is replaced with normal spaces instead?

This is getting pretty nasty.

[lowjoel]

For now I've got this:

@@ -16,2 +16 @@ module HTML
-          node.css("br").each { |br| br.replace("\n") } if replace_br
-          text = node.inner_text
+          text = preprocess_html(node)
@@ -30,0 +30,9 @@ module HTML
+      def preprocess_html(node)
+        node.css("br").each { |br| br.replace("\n") } if replace_br?
+        result = node.inner_text
+        return result unless preprocess_html?
+
+        result.gsub!("\u00A0", ' ')
+        result
+      end
+
@@ -43,2 +51,2 @@ module HTML
-      def replace_br
-        context[:replace_br] || false
+      def replace_br?
+        context[:replace_br] || preprocess_html?
@@ -46,0 +55,4 @@ module HTML
+      def preprocess_html?
+        context[:preprocess_html] || false
+      end
+

@JuanitoFatas if you're fine with this I'll cook up a spec and PR.

[lowjoel]

@JuanitoFatas any comments?

[JuanitoFatas]

@JuanitoFatas if you're fine with this I'll cook up a spec and PR.

Sorry for late response, let's go with this first :)