Character encoding issues with articles from previous-gen site

[bjacobel]

Example here. Fairly common with pre-2010 articles.

This may be unfixable. Have to look and see if they're actually saved that way in the DB, or if this is just a presentation layer thing.

[bjacobel]

May be related to #21

[tophtucker]

i think this is basically unfixable at this point. even by the time i first
saw the db, they were all straight-up ascii question marks. if the browser
were just rendering special chars as ?s we'd be in better shape but that's
not it. :-/

i flirted with whipping up a data cleaning mini-app - i think a simple
regex could spot many of the bad ?s and then a human could choose from
likely possible corrections - prob not worth it, idk
On Sun, Mar 29, 2015 at 19:16 Brian Jacobel [email protected]
wrote:

May be related to #21 #21

—
Reply to this email directly or view it on GitHub
#118 (comment).

[bjacobel]

s/(\?)\b\w+\b/‘/g and s/\b\w+\b(\?)/’/g would fix a large number of the issues, probably.

[bjacobel]

alternately: s/\?(\b\w+\b)\?/“$1”/g

I love regex golf 💃