«

Dear all:
I want to build a lex analyzer,
and streamtokenizer is seems a powerful API
but I have 2 question about it
1).how to use method slashSlashComments(boolean flag)
I try to set its flag as true,but it doesn't work when the comment
char be encountered. I also get a token that contains comments.
slashStarComments() has the same condition
2).I want to parse the java source code, and get the token
for example:
str1 = str2;
str1.compareTo("abc");
====================>use whitespace(' ','\t') and semicolon(';') to delimte
str1
=
str2
str1.compareTo("abc")
====================>but I want semicolon is also a token, like this
str1
=
str2
;
str1.compareTo("abc")
;

can only use streamtokenizer to implement this function?
thanks
Eric

Eric wrote
> Dear all:
> I want to build a lex analyzer,
> and streamtokenizer is seems a powerful API
> but I have 2 question about it
> 1).how to use method slashSlashComments(boolean flag)
> I try to set its flag as true,but it doesn't work when the comment
> char be encountered. I also get a token that contains comments.
> slashStarComments() has the same condition
> 2).I want to parse the java source code, and get the token
> for example:
> str1 = str2;
> str1.compareTo("abc");
> ====================>use whitespace(' ','\t') and semicolon(';') to
delimte
> str1
> =
> str2
> str1.compareTo("abc")
> ====================>but I want semicolon is also a token, like this
> str1
> =
> str2
> ;
> str1.compareTo("abc")
> ;
>
> can only use streamtokenizer to implement this function?
> thanks
> Eric

Hello Eric,
Have you considered using StringTokenizer instead of StreamTokenizer.
It's constructor StringTokenizer(String str, String delim, boolean
returnDelims) allows you to choose whether to return the delimiters as
tokens or not. Just set the returnDelims flag to true.

Regards,
Perica

> Hello Eric,
> Have you considered using StringTokenizer instead of StreamTokenizer.
> It's constructor StringTokenizer(String str, String delim, boolean
> returnDelims) allows you to choose whether to return the delimiters as
> tokens or not. Just set the returnDelims flag to true.
>
> Regards,
> Perica


thanx for your suggestion
but, I want to parse the java program source code
the java code is so complex that I need powerful function to do this
job
1.I need more than one delimiter to get token once a time (e.g. '
','\t','\n')
2.comments in java code must ignore
3.white space char mention in 1. must ignore
4.some delimiter(';','{','}') must be reserve, and others must
discard(e.g. ' ','\t','\n')

StringTokenizer has not enough functionality for my requirement

StreamTokenizer is "almost" perfect, but can't return the delimiters
as tokens

any one can give more help?

Regards, Eric

"Eric" <> wrote in message
news: om...
> > Hello Eric,
> > Have you considered using StringTokenizer instead of StreamTokenizer.
> > It's constructor StringTokenizer(String str, String delim, boolean
> > returnDelims) allows you to choose whether to return the delimiters as
> > tokens or not. Just set the returnDelims flag to true.
> >
> > Regards,
> > Perica
>
>
> thanx for your suggestion
> but, I want to parse the java program source code
> the java code is so complex that I need powerful function to do this
> job
> 1.I need more than one delimiter to get token once a time (e.g. '
> ','\t','\n')
> 2.comments in java code must ignore
> 3.white space char mention in 1. must ignore
> 4.some delimiter(';','{','}') must be reserve, and others must
> discard(e.g. ' ','\t','\n')
>
> StringTokenizer has not enough functionality for my requirement
>
> StreamTokenizer is "almost" perfect, but can't return the delimiters
> as tokens
>
> any one can give more help?
>
> Regards, Eric

it's not all that complicated to do yourself
go for it