«

Does exist any module/script that can 100% detect text language..
for example English, German, French, ... (European languages, at least
English...)

AR <> wrote:
> Does exist any module/script that can 100% detect text language..
> for example English, German, French, ... (European languages, at least
> English...)

100%? No. What language is this string: "hotel"?

Ben

--
Joy and Woe are woven fine,
A Clothing for the Soul divine William Blake
Under every grief and pine 'Auguries of Innocence'
Runs a joy with silken twine.

Ben Morrow <> wrote:

> AR <> wrote:
> > Does exist any module/script that can 100% detect text language..
> > for example English, German, French, ... (European languages, at least
> > English...)
>
> 100%? No. What language is this string: "hotel"?

Swahili?

--
JBM
"Everything is futile." -- Marvin of Borg

On Wed, 21 Jan 2004, Ben Morrow wrote:

> 100%? No. What language is this string: "hotel"?

Yeah, ask a German speaker what language this is: "Gift".

Ben Morrow <> wrote:
>
> AR <> wrote:
>> Does exist any module/script that can 100% detect text language..
>> for example English, German, French, ... (European languages, at least
>> English...)
>
> 100%? No. What language is this string: "hotel"?


Military? (the letter "H") ?


--
Tad McClellan SGML consulting
Perl programming
Fort Worth, Texas

J.B. Moreno wrote:
> Ben Morrow <> wrote:
>
>
>>AR <> wrote:
>>
>>>Does exist any module/script that can 100% detect text language..
>>>for example English, German, French, ... (European languages, at least
>>>English...)
>>
>>100%? No. What language is this string: "hotel"?
>
>
> Swahili?
Start by adding all words from all the dictionaries in the world in a file.
Then using statistics you get the most likely one.

or why not just?

#!/usr/bin/perl -w

print "String is in any known language or some constructed language such
as Esperanto, Volapuk, Glosa, Loglan, or even klingon.\n";


Now that would almost certainly cover 95% of all the languages (missed
adding the tolkien languages, but i leave that as a programmin excercise
). But im not sure if its 100% future proof. The "any known language"
could be interpreted as "known" to the person running the program.

Best Regards
Martin Quensel

Martin Quensel <> wrote:

> J.B. Moreno wrote:
> > Ben Morrow <> wrote:
> >
> >>AR <> wrote:
> >>
> >>>Does exist any module/script that can 100% detect text language..
> >>>for example English, German, French, ... (European languages, at least
> >>>English...)
> >>
> >>100%? No. What language is this string: "hotel"?
> >
> > Swahili?
>
> Start by adding all words from all the dictionaries in the world in a
> file. Then using statistics you get the most likely one.

The phrases "100%" and "most likely one" aren't equivalent.

And look up the James Nicoll quote on the purity of the english
language.

--
JBM
"Everything is futile." -- Marvin of Borg

Ben Morrow <> wrote in comp.lang.perl.misc:
>
> AR <> wrote:
> > Does exist any module/script that can 100% detect text language..
> > for example English, German, French, ... (European languages, at least
> > English...)
>
> 100%? No. What language is this string: "hotel"?

Well, one-word-samples are hard, and 100% is unattainable.

Entirely off topic, I have recently heard of an approach to text
classification (with an eye to language recognition) that I found
interesting.

Use a Ziv-Lempel-like method to compress your sample. Then concatenate
it with texts of similar lengths taken from known languages and compress
again. If the compression rate is similar or better than that of the
original text, the appended text is similar to the original one. If
the compression deteriorates, the texts are dissimilar.

The source (some idle chat on IRC, sorry) said that this works for
rather small samples of fewer than a hundred words. I have always been
meaning to play with it, but haven't got around.

Anno

On Thu, 22 Jan 2004 00:35:12 -0600, J.B. Moreno wrote:

>> Start by adding all words from all the dictionaries in the world in a
>> file. Then using statistics you get the most likely one.
>
> The phrases "100%" and "most likely one" aren't equivalent

This is true, but in the real world, something which gives a 99.9%
probability is about as good as we are going to get. No sense in
refusing to use a circle simply because it is impossible to make a
perfect one.

IMO, 99.9% might be a low estimate even if the program takes a naive
approach. If the dictionaries include "adopted" phrases (e.g. Latin
expressions which are often cited in English, etc.) and some kind of
best-fit spell check is used, you might push the probabilities into
99.99%. Now feed some works of literature from each language into a
phrase-counter and use phrases as well, and you might find that a text of
100 words or more can be predicted correctly 99.9999% of the time.

If that isn't good enough (missing 1 of 10^6), you're going to be working
on the thing for so long that half of the languages in use at its
conception are out of use before you reach the prototype.

--Eric

Ben Morrow () wrote:

: AR <> wrote:
: > Does exist any module/script that can 100% detect text language..
: > for example English, German, French, ... (European languages, at least
: > English...)

: 100%? No. What language is this string: "hotel"?

I can say with 100% certainty that that is an english word.