Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > using strtok to mark delimiters as tokens

Reply
Thread Tools

using strtok to mark delimiters as tokens

 
 
gpaps87@gmail.com
Guest
Posts: n/a
 
      03-12-2008
hi,
i wanted to know whether we can use strtok command to mark delimiters
as tokens as well.In Java,we have a command:

StringTokennizer(String str, String delimiters, boolean
delimAsToken)

which considers the delimiters as tokens,too.Can strtok accomplish
this requirement?or could you please let me know if there is any other
command in C that would carry out this task?
 
Reply With Quote
 
 
 
 
santosh
Guest
Posts: n/a
 
      03-12-2008
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> hi,
> i wanted to know whether we can use strtok command to mark delimiters
> as tokens as well.In Java,we have a command:
>
> StringTokennizer(String str, String delimiters, boolean
> delimAsToken)
>
> which considers the delimiters as tokens,too.Can strtok accomplish
> this requirement?


You could force strtok to consider delimiters as tokens by specifying a
different delimiter string at the next call to it. But this might not
be quite what you want though.

> or could you please let me know if there is any other
> command in C that would carry out this task?


You can use a string function like strstr or strcspn to locate the
necessary substring and tokenise the string yourself. In fact, it may
be more robust to use these functions and write your own tokeniser, as
strtok has a number of caveats you need to be aware of to use it
safely.

 
Reply With Quote
 
 
 
 
Falcon Kirtaran
Guest
Posts: n/a
 
      03-12-2008
(E-Mail Removed) wrote:
> hi,
> i wanted to know whether we can use strtok command to mark delimiters
> as tokens as well.In Java,we have a command:
>
> StringTokennizer(String str, String delimiters, boolean
> delimAsToken)
>
> which considers the delimiters as tokens,too.Can strtok accomplish
> this requirement?or could you please let me know if there is any other
> command in C that would carry out this task?


I don't think there is. This is because the function is generally
implemented by replacing the delimiter with a null.

It'd be really easy to write your own, however, that preserves the
delimiting character. For instance, it might look something like:

char * strtok_d(char * str, const char * tokens, char * delim) {
int i1 = 0;
int i2 = 0;

static char * istr;
if (str) istr = str; /*if str is non-null, use str instead of the*/
/*stored value*/

while (istr[i1]) {
while (tokens[i2]) {
if (istr[i1] == tokens[i2]) {
*delim = tokens[i2]; /*save the token*/
tokens[i2] = 0; /*tokenize*/
istr += i2; /*set a new istr for the next call*/
return istr;
};
};

i2 = 0; /*re-init after use*/
i1++; /*iterate through*/
};

return 0; /*should not arrive here*/
};

The only thing you'd need to take care of is that both parameters must
be null-terminated strings. Also, you'd have to check if the third
parameter when dereferenced is not 0. Obviously, the third parameter is
just a single char (by reference), not a string.

The function could return it as a char *, but that would massively
modify the behaviour of the function, and would probably require it to
allocate a char array that you'd later have to free(). Calls to this
can't be mixed interchangeably with calls to strtok().

I've not tested this function at all; it's purely theoretical

--
--Falcon Kirtaran
 
Reply With Quote
 
Mark Bluemel
Guest
Posts: n/a
 
      03-12-2008
(E-Mail Removed) wrote:
> hi,
> i wanted to know whether we can use strtok command to mark delimiters
> as tokens as well.In Java,we have a command:
>
> StringTokennizer(String str, String delimiters, boolean
> delimAsToken)
>
> which considers the delimiters as tokens,too.Can strtok accomplish
> this requirement?


No, it can't.

> or could you please let me know if there is any other
> command in C that would carry out this task?


You could use one of the techniques others have already suggested, and
write your own.

Or you could do what's recommended with later versions of Java and move
to using a regular expression parser to pick apart the string.
 
Reply With Quote
 
Mark Bluemel
Guest
Posts: n/a
 
      03-12-2008
Falcon Kirtaran wrote:
> (E-Mail Removed) wrote:
>> hi,
>> i wanted to know whether we can use strtok command to mark delimiters
>> as tokens as well.In Java,we have a command:


[snip]

> It'd be really easy to write your own, however, that preserves the
> delimiting character. For instance, it might look something like:
>
> char * strtok_d(char * str, const char * tokens, char * delim) {


[snip]

> };
>
> The only thing you'd need to take care of is that both parameters must
> be null-terminated strings. Also, you'd have to check if the third
> parameter when dereferenced is not 0. Obviously, the third parameter is
> just a single char (by reference), not a string.
>
>[snip] Calls to this
> can't be mixed interchangeably with calls to strtok().


And due to the (very dubious) design decision to hold state in a static
variable, it is not thread-safe or even capable of being used on more
than one string at a time, for example if you wanted to further parse
a field extracted from a string before moving on to the next.

There are better solutions out there, I'm sure.
 
Reply With Quote
 
Richard Bos
Guest
Posts: n/a
 
      03-12-2008
Falcon Kirtaran <(E-Mail Removed)> wrote:

> (E-Mail Removed) wrote:
> > i wanted to know whether we can use strtok command to mark delimiters
> > as tokens as well.In Java,we have a command:


> I don't think there is. This is because the function is generally
> implemented by replacing the delimiter with a null.


s/generally/required to be/.

> It'd be really easy to write your own, however, that preserves the
> delimiting character. For instance, it might look something like:
>
> char * strtok_d(char * str, const char * tokens, char * delim) {


Don't call it that, though. Names starting in str and a letter are
reserved for the implementation's use.

> The only thing you'd need to take care of is that both parameters must
> be null-terminated strings.


No ****. This is generally true of string-mangling functions: they take
strings, not randomly filled arrays.

Another useful modification for your home-grown strtok() replacement
could be (depending on your requirements) that it does not skip empty
tokens. strtok() does so; this is sometimes desired behaviour, but to
me, it's usually a hindrance.

Richard
 
Reply With Quote
 
Default User
Guest
Posts: n/a
 
      03-12-2008
santosh wrote:

> (E-Mail Removed) wrote:
>
> > hi,
> > i wanted to know whether we can use strtok command to mark
> > delimiters as tokens as well.In Java,we have a command:
> >
> > StringTokennizer(String str, String delimiters, boolean
> > delimAsToken)
> >
> > which considers the delimiters as tokens,too.Can strtok accomplish
> > this requirement?

>
> You could force strtok to consider delimiters as tokens by specifying
> a different delimiter string at the next call to it. But this might
> not be quite what you want though.


Not really. strtok() punches a hole in the string where that first
delimiter was. No way to save it. Besides, presumably the OP wants to
keep using the delimiter set.

I'm not sure what the exact semantics of the Java call are, it would
have to be specified better. What I would do would be an approach more
like PHP explode(), and produce an array or list of strings. Then it
would be relatively easy to save the delimiters too.




Brian
 
Reply With Quote
 
Falcon Kirtaran
Guest
Posts: n/a
 
      03-12-2008
Mark Bluemel wrote:
> Falcon Kirtaran wrote:
>> (E-Mail Removed) wrote:
>>> hi,
>>> i wanted to know whether we can use strtok command to mark delimiters
>>> as tokens as well.In Java,we have a command:

>
> [snip]
>
>> It'd be really easy to write your own, however, that preserves the
>> delimiting character. For instance, it might look something like:
>>
>> char * strtok_d(char * str, const char * tokens, char * delim) {

>
> [snip]
>
>> };
>>
>> The only thing you'd need to take care of is that both parameters must
>> be null-terminated strings. Also, you'd have to check if the third
>> parameter when dereferenced is not 0. Obviously, the third parameter
>> is just a single char (by reference), not a string.
>>
>> [snip] Calls to this can't be mixed interchangeably with calls to
>> strtok().

>
> And due to the (very dubious) design decision to hold state in a static
> variable, it is not thread-safe or even capable of being used on more
> than one string at a time, for example if you wanted to further parse
> a field extracted from a string before moving on to the next.
>
> There are better solutions out there, I'm sure.


strtok() itself is not thread-safe, however, and also can't be used on
more than one string at a time.

--
--Falcon Kirtaran
 
Reply With Quote
 
Falcon Kirtaran
Guest
Posts: n/a
 
      03-12-2008
Richard Bos wrote:
> Falcon Kirtaran <(E-Mail Removed)> wrote:
>
>> (E-Mail Removed) wrote:
>>> i wanted to know whether we can use strtok command to mark delimiters
>>> as tokens as well.In Java,we have a command:

>
>> I don't think there is. This is because the function is generally
>> implemented by replacing the delimiter with a null.

>
> s/generally/required to be/.
>
>> It'd be really easy to write your own, however, that preserves the
>> delimiting character. For instance, it might look something like:
>>
>> char * strtok_d(char * str, const char * tokens, char * delim) {

>
> Don't call it that, though. Names starting in str and a letter are
> reserved for the implementation's use.
>
>> The only thing you'd need to take care of is that both parameters must
>> be null-terminated strings.

>
> No ****. This is generally true of string-mangling functions: they take
> strings, not randomly filled arrays.
>
> Another useful modification for your home-grown strtok() replacement
> could be (depending on your requirements) that it does not skip empty
> tokens. strtok() does so; this is sometimes desired behaviour, but to
> me, it's usually a hindrance.
>
> Richard


I don't think mine actually does skip empty tokens. If it finds an
empty token, it'll just return the next delimiter and a pointer to a
string that starts with null.

--
--Falcon Kirtaran
 
Reply With Quote
 
Malcolm McLean
Guest
Posts: n/a
 
      03-12-2008
<(E-Mail Removed)> wrote in message
>
> Can strtok accomplish
> this requirement?or could you please let me know if there is any other
> command in C that would carry out this task?
>

sscanf() might be what you are looking for.

Alternatively, if you are implementing something like a BASIC interpreter,
you need a struct with the string, the read position, and the top token
read. Then you have a match() function to get rid of the current token and
get a new one, triggering an error if there is no match (for instance for
closing paretheses].


--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
To Mark Fitzpatrick, Juan Libre, Mark Rae, Steve Orr, Cowboy and other MVPs Cirene ASP .Net 5 05-17-2008 07:17 PM
strtok behavior with multiple consecutive delimiters Geometer C Programming 34 05-09-2006 02:32 PM
strtok behavior with multiple consecutive delimiters Geometer C++ 33 05-09-2006 02:32 PM
Problems with strtok() returning one too many tokens... Adam Balgach C++ 2 11-28-2004 01:12 AM



Advertisments