I tested the script in python 2.7 and it works perfect. The problem is in python 3.7(and may be only in windows as you were not getting the issue) and I was getting the following error
UnicodeEncodeError: 'charmap' codec can't encode character '\u0100' in position 0: character maps to <undefined>
I went through the python script and found that the stdout encoding is set to utf-8 only if python version is <=2.
I have made the same change for python version 3 as well. Please find the patch for the same.Let me know if it makes sense
Regards,
Ram
Hi Ram,
I took a look at this, and unfortunately the proposed fix breaks Python 2 (sys.stdout.encoding isn't a writable attribute in Python 2) :-(. I've attached a patch which is compatible with both versions, and have confirmed that the output is identical across Python 2 and 3 and across both Windows and Linux. The output on Windows and Linux is identical, once the difference in line endings is accounted for.
I've also opened the Unicode data file in UTF-8 and added a "with" block which ensures we close the file when we are done with it. The change makes the Python2 compatibility a little more complex (2 blocks to remove), but it's the cleanest I could achieve.
The attached patch goes on top of patch 02 (not on top of the broken, committed 03). I'm hoping that's not a problem. If it is, let me know and I'll factor out the changes.
