Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Javascript > identify the language of a web page

Reply
Thread Tools

identify the language of a web page

 
 
usgog@yahoo.com
Guest
Posts: n/a
 
      04-11-2008
Suppose I need to classify 10000 web pages based on their languages.
What should I look for to determine the language of each web page? Any
advice is welcome.
 
Reply With Quote
 
 
 
 
GArlington
Guest
Posts: n/a
 
      04-11-2008
On Apr 11, 6:29 am, "(E-Mail Removed)" <(E-Mail Removed)> wrote:
> Suppose I need to classify 10000 web pages based on their languages.
> What should I look for to determine the language of each web page? Any
> advice is welcome.


You can look for encoding in the header, but if (as many do) the page
is using utf-8, then it can not be done. Again, UTF-8 allows multiple
langauges on one page, which one will you want to pick?
 
Reply With Quote
 
 
 
 
pr
Guest
Posts: n/a
 
      04-11-2008
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Suppose I need to classify 10000 web pages based on their languages.
> What should I look for to determine the language of each web page? Any
> advice is welcome.


You could search the content for a common word in a given language that
is used in neither HTML nor script: " the " (including the spaces) would
be, I guess, a reasonable choice to identify English, although there's
no guarantee some bright spark hasn't named a script variable 'the', or
used the word in a comment.

Are you sure this is a JavaScript question?
 
Reply With Quote
 
Joost Diepenmaat
Guest
Posts: n/a
 
      04-11-2008
"(E-Mail Removed)" <(E-Mail Removed)> writes:

> Suppose I need to classify 10000 web pages based on their languages.
> What should I look for to determine the language of each web page? Any
> advice is welcome.


Well, if the pages provide them, the lang attributes are probably correct.
http://www.w3.org/TR/html401/struct/dirlang.html#h-8.1

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
 
Reply With Quote
 
Dr J R Stockton
Guest
Posts: n/a
 
      04-11-2008
In comp.lang.javascript message <cda0f617-c9a0-4389-b79e-a02ad24852a6@k1
0g2000prm.googlegroups.com>, Thu, 10 Apr 2008 22:29:01,
"(E-Mail Removed)" <(E-Mail Removed)> posted:
>Suppose I need to classify 10000 web pages based on their languages.
>What should I look for to determine the language of each web page? Any
>advice is welcome.


Consider <URL:http://www.merlyn.demon.co.uk/zel-82px.htm> and siblings.

--
(c) John Stockton, nr London, UK. ?@merlyn.demon.co.uk Turnpike v6.05 MIME.
Web <URL:http://www.merlyn.demon.co.uk/> - FAQish topics, acronyms, & links.
Proper <= 4-line sig. separator as above, a line exactly "-- " (SonOfRFC1036)
Do not Mail News to me. Before a reply, quote with ">" or "> " (SonOfRFC1036)
 
Reply With Quote
 
VK
Guest
Posts: n/a
 
      04-12-2008
On Apr 12, 2:35 am, Dr J R Stockton <(E-Mail Removed)> wrote:
> In comp.lang.javascript message <cda0f617-c9a0-4389-b79e-a02ad24852a6@k1
> 0g2000prm.googlegroups.com>, Thu, 10 Apr 2008 22:29:01,
> "(E-Mail Removed)" <(E-Mail Removed)> posted:
>
> >Suppose I need to classify 10000 web pages based on their languages.
> >What should I look for to determine the language of each web page? Any
> >advice is welcome.

>
> Consider <URL:http://www.merlyn.demon.co.uk/zel-82px.htm> and siblings.


<OT>
more for ciwah, so OT, but still:
is there language code for multilanguage document, like lang="multi"
or something?
</OT>
 
Reply With Quote
 
Joost Diepenmaat
Guest
Posts: n/a
 
      04-12-2008
VK <(E-Mail Removed)> writes:

> <OT>
> more for ciwah, so OT, but still:
> is there language code for multilanguage document, like lang="multi"
> or something?
> </OT>


All (or most, I'm not sure, pretty much all though) tags can have a lang
attribute, so the "multi-languageness" is built-in already.

http://www.w3.org/TR/REC-html40/stru...html#adef-lang



--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to identify double bytes language? sqlcamel Perl Misc 8 11-14-2009 11:46 PM
identify the language of a web page usgog@yahoo.com XML 2 04-11-2008 01:34 PM
Identify the language of a String literal javadev Java 2 04-14-2006 03:50 AM
Identify which page is loaded in main frame of a frameset Andrew K Javascript 1 02-23-2005 02:06 PM
Page.Controls to identify server controls KK ASP .Net 2 01-25-2004 02:09 PM



Advertisments