«

Suppose I need to classify 10000 web pages based on their languages.
What should I look for to determine the language of each web page? Any
advice is welcome.

On Apr 11, 6:29 am, "us...@yahoo.com" <us...@yahoo.com> wrote:
> Suppose I need to classify 10000 web pages based on their languages.
> What should I look for to determine the language of each web page? Any
> advice is welcome.

You can look for encoding in the header, but if (as many do) the page
is using utf-8, then it can not be done. Again, UTF-8 allows multiple
langauges on one page, which one will you want to pick?

wrote:
> Suppose I need to classify 10000 web pages based on their languages.
> What should I look for to determine the language of each web page? Any
> advice is welcome.

You could search the content for a common word in a given language that
is used in neither HTML nor script: " the " (including the spaces) would
be, I guess, a reasonable choice to identify English, although there's
no guarantee some bright spark hasn't named a script variable 'the', or
used the word in a comment.

Are you sure this is a JavaScript question?

"" <> writes:

> Suppose I need to classify 10000 web pages based on their languages.
> What should I look for to determine the language of each web page? Any
> advice is welcome.

Well, if the pages provide them, the lang attributes are probably correct.
http://www.w3.org/TR/html401/struct/dirlang.html#h-8.1

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

In comp.lang.javascript message <cda0f617-c9a0-4389-b79e-a02ad24852a6@k1
0g2000prm.googlegroups.com>, Thu, 10 Apr 2008 22:29:01,
"" <> posted:
>Suppose I need to classify 10000 web pages based on their languages.
>What should I look for to determine the language of each web page? Any
>advice is welcome.

Consider <URL:http://www.merlyn.demon.co.uk/zel-82px.htm> and siblings.

--
(c) John Stockton, nr London, UK. ?@merlyn.demon.co.uk Turnpike v6.05 MIME.
Web <URL:http://www.merlyn.demon.co.uk/> - FAQish topics, acronyms, & links.
Proper <= 4-line sig. separator as above, a line exactly "-- " (SonOfRFC1036)
Do not Mail News to me. Before a reply, quote with ">" or "> " (SonOfRFC1036)

On Apr 12, 2:35 am, Dr J R Stockton <j...@merlyn.demon.co.uk> wrote:
> In comp.lang.javascript message <cda0f617-c9a0-4389-b79e-a02ad24852a6@k1
> 0g2000prm.googlegroups.com>, Thu, 10 Apr 2008 22:29:01,
> "us...@yahoo.com" <us...@yahoo.com> posted:
>
> >Suppose I need to classify 10000 web pages based on their languages.
> >What should I look for to determine the language of each web page? Any
> >advice is welcome.
>
> Consider <URL:http://www.merlyn.demon.co.uk/zel-82px.htm> and siblings.

<OT>
more for ciwah, so OT, but still:
is there language code for multilanguage document, like lang="multi"
or something?
</OT>

VK <> writes:

> <OT>
> more for ciwah, so OT, but still:
> is there language code for multilanguage document, like lang="multi"
> or something?
> </OT>

All (or most, I'm not sure, pretty much all though) tags can have a lang
attribute, so the "multi-languageness" is built-in already.

http://www.w3.org/TR/REC-html40/stru...html#adef-lang



--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/