How can I detect the encoding/codepage of a text file
|
2:46 AM |
No comments |
How can I detect the encoding/codepage of a text file
In our application, we receive text files (.txt, .csv, etc.) from diverse sources. When reading, these files sometimes contain garbage, because the files where created in a different/unknown codepage.
Is there a way to (automatically) detect the codepage of a text file?
The detectEncodingFromByteOrderMarks, on the StreamReader constructor, works for UTF8 and other unicode marked files, but I'm looking for a way to detect code pages, like ibm850, windows1252.
Thanks for your answers, this is what I've done.
The files we receive are from end-users, they do not have a clue about codepages. The receivers are also end-users, by now this is what they know about codepages: Codepages exist, and are annoying.
Solution:
- Open the received file in Notepad, look at a garbled piece of text. If somebody is called François or something, with your human intelligence you can guess this.
- I've created a small app that the user can use to open the file with, and enter a text that user knows it will appear in the file, when the correct codepage is used.
- Loop through all codepages, and display the ones that give a solution with the user provided text.
- If more as one codepage pops up, ask the user to specify more text.
Answer by leppie for How can I detect the encoding/codepage of a text file
The StreamReader class's constructor takes a 'detect encoding' parameter.
Answer by Tomer Gabel for How can I detect the encoding/codepage of a text file
If you're looking to detect non-UTF encodings (i.e. no BOM), you're basically down to heuristics and statistical analysis of the text. You might want to take a look at the Mozilla paper on universal charset detection (same link, with better formatting via Wayback Machine).
Answer by DeeCee for How can I detect the encoding/codepage of a text file
Got the same problem but didn't found a good solution yet for detecting it automatically . Now im using PsPad (www.pspad.com) for that ;) Works fine
Answer by JV. for How can I detect the encoding/codepage of a text file
You can't detect the codepage, you need to be told it. You can analyse the bytes and guess it, but that can give some bizarre (sometimes amusing) results. I can't find it now, but I'm sure Notepad can be tricked into displaying English text in Chinese.
Anyway, this is what you need to read: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).
Specifically Joel says:
The Single Most Important Fact About Encodings
If you completely forget everything I just explained, please remember one extremely important fact. It does not make sense to have a string without knowing what encoding it uses. You can no longer stick your head in the sand and pretend that "plain" text is ASCII. There Ain't No Such Thing As Plain Text.
If you have a string, in memory, in a file, or in an email message, you have to know what encoding it is in or you cannot interpret it or display it to users correctly.
Answer by tzot for How can I detect the encoding/codepage of a text file
I've done something similar in Python. Basically, you need lots of sample data from various encodings, which are broken down by a sliding two-byte window and stored in a dictionary (hash), keyed on byte-pairs providing values of lists of encodings.
Given that dictionary (hash), you take your input text and:
- if it starts with any BOM character ('\xfe\xff' for UTF-16-BE, '\xff\xfe' for UTF-16-LE, '\xef\xbb\xbf' for UTF-8 etc), I treat it as suggested
- if not, then take a large enough sample of the text, take all byte-pairs of the sample and choose the encoding that is the least common suggested from the dictionary.
If you've also sampled UTF encoded texts that do not start with any BOM, the second step will cover those that slipped from the first step.
So far, it works for me (the sample data and subsequent input data are subtitles in various languages) with diminishing error rates.
Answer by shoosh for How can I detect the encoding/codepage of a text file
You can't detect the codepage
This is clearly false. Every web browser has some kind of universal charset detector to deal with pages which have no indication whatsoever of an encoding. Firefox has one. You can download the code and see how it does it. See some documentation here. Basically, it is a heuristic, but one that works really well.
Given a reasonable amount of text, it is even possible to detect the language.
Here's another one I just found using Google:
Answer by devstuff for How can I detect the encoding/codepage of a text file
Since it basically comes down to heuristics, it may help to use the encoding of previously received files from the same source as a first hint.
Most people (or applications) do stuff in pretty much the same order every time, often on the same machine, so its quite likely that when Bob creates a .csv file and sends it to Mary it'll always be using Windows-1252 or whatever his machine defaults to.
Where possible a bit of customer training never hurts either :-)
Answer by hegearon for How can I detect the encoding/codepage of a text file
Notepad++ has this feature out-of-the-box. It also supports changing it.
Answer by Intraday Tips for How can I detect the encoding/codepage of a text file
I was actually looking for a generic, not programming way of detecting the file encoding, but I didn't find that yet. What I did find by testing with different encodings was that my text was UTF-7.
So where I first was doing: StreamReader file = File.OpenText(fullfilename);
I had to change it to: StreamReader file = new StreamReader(fullfilename, System.Text.Encoding.UTF7);
OpenText assumes it's UTF-8.
you can also create the StreamReader like this new StreamReader(fullfilename, true), the second parameter meaning that it should try and detect the encoding from the byteordermark of the file, but that didn't work in my case.
Answer by Tao for How can I detect the encoding/codepage of a text file
I know it's very late for this question and this solution won't appeal to some (because of its english-centric bias and its lack of statistical/empirical testing), but it's worked very well for me, especially for processing uploaded CSV data:
http://www.architectshack.com/TextFileEncodingDetector.ashx
Advantages:
- BOM detection built-in
- Default/fallback encoding customizable
- pretty reliable (in my experience) for western-european-based files containing some exotic data (eg french names) with a mixture of UTF-8 and Latin-1-style files - basically the bulk of US and western european environments.
Note: I'm the one who wrote this class, so obviously take it with a grain of salt! :)
Answer by Thommy Johansson for How can I detect the encoding/codepage of a text file
I use this code to detect Unicode and windows default ansi codepage when reading a file. For other codings a check of content is necessary, manually or by programming. This can de used to save the text with the same encoding as when it was opened. (I use VB.NET)
'Works for Default and unicode (auto detect) Dim mystreamreader As New StreamReader(LocalFileName, Encoding.Default) MyEditTextBox.Text = mystreamreader.ReadToEnd() Debug.Print(mystreamreader.CurrentEncoding.CodePage) 'Autodetected encoding mystreamreader.Close() Answer by ITmeze for How can I detect the encoding/codepage of a text file
Have you tried C# port for Mozilla Universal Charset Detector
Example from http://code.google.com/p/ude/
public static void Main(String[] args) { string filename = args[0]; using (FileStream fs = File.OpenRead(filename)) { Ude.CharsetDetector cdet = new Ude.CharsetDetector(); cdet.Feed(fs); cdet.DataEnd(); if (cdet.Charset != null) { Console.WriteLine("Charset: {0}, confidence: {1}", cdet.Charset, cdet.Confidence); } else { Console.WriteLine("Detection failed."); } } } Answer by Kundor for How can I detect the encoding/codepage of a text file
If you can link to a C library, you can use libenca. See http://cihar.com/software/enca/. From the man page:
Enca reads given text files, or standard input when none are given, and uses knowledge about their language (must be supported by you) and a mixture of parsing, statistical analysis, guessing and black magic to determine their encodings.
It's GPL v2.
Answer by TarmoPikaro for How can I detect the encoding/codepage of a text file
Looking for different solution, I found that
https://code.google.com/p/ude/
this solution is kinda heavy.
I needed some basic encoding detection, based on 4 first bytes and probably xml charset detection - so I've took some sample source code from internet and added slightly modified version of
http://lists.w3.org/Archives/Public/www-validator/2002Aug/0084.html
written for Java.
public static Encoding DetectEncoding(byte[] fileContent) { if (fileContent == null) throw new ArgumentNullException(); if (fileContent.Length < 2) return Encoding.ASCII; // Default fallback if (fileContent[0] == 0xff && fileContent[1] == 0xfe && (fileContent.Length < 4 || fileContent[2] != 0 || fileContent[3] != 0 ) ) return Encoding.Unicode; if (fileContent[0] == 0xfe && fileContent[1] == 0xff ) return Encoding.BigEndianUnicode; if (fileContent.Length < 3) return null; if (fileContent[0] == 0xef && fileContent[1] == 0xbb && fileContent[2] == 0xbf) return Encoding.UTF8; if (fileContent[0] == 0x2b && fileContent[1] == 0x2f && fileContent[2] == 0x76) return Encoding.UTF7; if (fileContent.Length < 4) return null; if (fileContent[0] == 0xff && fileContent[1] == 0xfe && fileContent[2] == 0 && fileContent[3] == 0) return Encoding.UTF32; if (fileContent[0] == 0 && fileContent[1] == 0 && fileContent[2] == 0xfe && fileContent[3] == 0xff) return Encoding.GetEncoding(12001); String probe; int len = fileContent.Length; if( fileContent.Length >= 128 ) len = 128; probe = Encoding.ASCII.GetString(fileContent, 0, len); MatchCollection mc = Regex.Matches(probe, "^<\\?xml[^<>]*encoding[ \\t\\n\\r]?=[\\t\\n\\r]?['\"]([A-Za-z]([A-Za-z0-9._]|-)*)", RegexOptions.Singleline); // Add '[0].Groups[1].Value' to the end to test regex if( mc.Count == 1 && mc[0].Groups.Count >= 2 ) { // Typically picks up 'UTF-8' string Encoding enc = null; try { enc = Encoding.GetEncoding( mc[0].Groups[1].Value ); }catch (Exception ) { } if( enc != null ) return enc; } return Encoding.ASCII; // Default fallback } It's enough to read probably first 1024 bytes from file, but I'm loading whole file.
Answer by Erik Aronesty for How can I detect the encoding/codepage of a text file
The tool "uchardet" does this well using character frequency distribution models for each charset. Larger files and more "typical" files have more confidence (obviously).
On ubuntu, you just apt-get install uchardet.
On other systems, get the source, usage & docs here: https://github.com/BYVoid/uchardet
Answer by PrivatePyle for How can I detect the encoding/codepage of a text file
As addon to ITmeze post, I've used this function to convert the output of C# port for Mozilla Universal Charset Detector
private Encoding GetEncodingFromString(string encoding) { try { return Encoding.GetEncoding(encoding); } catch { return Encoding.ASCII; } } Answer by Magu for How can I detect the encoding/codepage of a text file
If someone is looking for a 93.9% solution. This works for me:
public static class StreamExtension { /// /// Convert the content to a string. /// /// The stream. /// Answer by plavozont for How can I detect the encoding/codepage of a text file
Open file in AkelPad(or just copy/paste a garbled text), go to Edit -> Selection -> Recode... -> check "Autodetect".
Fatal error: Call to a member function getElementsByTagName() on a non-object in D:\XAMPP INSTALLASTION\xampp\htdocs\endunpratama9i\www-stackoverflow-info-proses.php on line 72
0 comments:
Post a Comment