«

hi,
i wanted to know whether we can use strtok command to mark delimiters
as tokens as well.In Java,we have a command:

StringTokennizer(String str, String delimiters, boolean
delimAsToken)

which considers the delimiters as tokens,too.Can strtok accomplish
this requirement?or could you please let me know if there is any other
command in C that would carry out this task?

wrote:

> hi,
> i wanted to know whether we can use strtok command to mark delimiters
> as tokens as well.In Java,we have a command:
>
> StringTokennizer(String str, String delimiters, boolean
> delimAsToken)
>
> which considers the delimiters as tokens,too.Can strtok accomplish
> this requirement?

You could force strtok to consider delimiters as tokens by specifying a
different delimiter string at the next call to it. But this might not
be quite what you want though.

> or could you please let me know if there is any other
> command in C that would carry out this task?

You can use a string function like strstr or strcspn to locate the
necessary substring and tokenise the string yourself. In fact, it may
be more robust to use these functions and write your own tokeniser, as
strtok has a number of caveats you need to be aware of to use it
safely.

wrote:
> hi,
> i wanted to know whether we can use strtok command to mark delimiters
> as tokens as well.In Java,we have a command:
>
> StringTokennizer(String str, String delimiters, boolean
> delimAsToken)
>
> which considers the delimiters as tokens,too.Can strtok accomplish
> this requirement?or could you please let me know if there is any other
> command in C that would carry out this task?

I don't think there is. This is because the function is generally
implemented by replacing the delimiter with a null.

It'd be really easy to write your own, however, that preserves the
delimiting character. For instance, it might look something like:

char * strtok_d(char * str, const char * tokens, char * delim) {
int i1 = 0;
int i2 = 0;

static char * istr;
if (str) istr = str; /*if str is non-null, use str instead of the*/
/*stored value*/

while (istr[i1]) {
while (tokens[i2]) {
if (istr[i1] == tokens[i2]) {
*delim = tokens[i2]; /*save the token*/
tokens[i2] = 0; /*tokenize*/
istr += i2; /*set a new istr for the next call*/
return istr;
};
};

i2 = 0; /*re-init after use*/
i1++; /*iterate through*/
};

return 0; /*should not arrive here*/
};

The only thing you'd need to take care of is that both parameters must
be null-terminated strings. Also, you'd have to check if the third
parameter when dereferenced is not 0. Obviously, the third parameter is
just a single char (by reference), not a string.

The function could return it as a char *, but that would massively
modify the behaviour of the function, and would probably require it to
allocate a char array that you'd later have to free(). Calls to this
can't be mixed interchangeably with calls to strtok().

I've not tested this function at all; it's purely theoretical

--
--Falcon Kirtaran

wrote:
> hi,
> i wanted to know whether we can use strtok command to mark delimiters
> as tokens as well.In Java,we have a command:
>
> StringTokennizer(String str, String delimiters, boolean
> delimAsToken)
>
> which considers the delimiters as tokens,too.Can strtok accomplish
> this requirement?

No, it can't.

> or could you please let me know if there is any other
> command in C that would carry out this task?

You could use one of the techniques others have already suggested, and
write your own.

Or you could do what's recommended with later versions of Java and move
to using a regular expression parser to pick apart the string.

Falcon Kirtaran wrote:
> wrote:
>> hi,
>> i wanted to know whether we can use strtok command to mark delimiters
>> as tokens as well.In Java,we have a command:

[snip]

> It'd be really easy to write your own, however, that preserves the
> delimiting character. For instance, it might look something like:
>
> char * strtok_d(char * str, const char * tokens, char * delim) {

[snip]

> };
>
> The only thing you'd need to take care of is that both parameters must
> be null-terminated strings. Also, you'd have to check if the third
> parameter when dereferenced is not 0. Obviously, the third parameter is
> just a single char (by reference), not a string.
>
>[snip] Calls to this
> can't be mixed interchangeably with calls to strtok().

And due to the (very dubious) design decision to hold state in a static
variable, it is not thread-safe or even capable of being used on more
than one string at a time, for example if you wanted to further parse
a field extracted from a string before moving on to the next.

There are better solutions out there, I'm sure.

Falcon Kirtaran <> wrote:

> wrote:
> > i wanted to know whether we can use strtok command to mark delimiters
> > as tokens as well.In Java,we have a command:

> I don't think there is. This is because the function is generally
> implemented by replacing the delimiter with a null.

s/generally/required to be/.

> It'd be really easy to write your own, however, that preserves the
> delimiting character. For instance, it might look something like:
>
> char * strtok_d(char * str, const char * tokens, char * delim) {

Don't call it that, though. Names starting in str and a letter are
reserved for the implementation's use.

> The only thing you'd need to take care of is that both parameters must
> be null-terminated strings.

No ****. This is generally true of string-mangling functions: they take
strings, not randomly filled arrays.

Another useful modification for your home-grown strtok() replacement
could be (depending on your requirements) that it does not skip empty
tokens. strtok() does so; this is sometimes desired behaviour, but to
me, it's usually a hindrance.

Richard

santosh wrote:

> wrote:
>
> > hi,
> > i wanted to know whether we can use strtok command to mark
> > delimiters as tokens as well.In Java,we have a command:
> >
> > StringTokennizer(String str, String delimiters, boolean
> > delimAsToken)
> >
> > which considers the delimiters as tokens,too.Can strtok accomplish
> > this requirement?
>
> You could force strtok to consider delimiters as tokens by specifying
> a different delimiter string at the next call to it. But this might
> not be quite what you want though.

Not really. strtok() punches a hole in the string where that first
delimiter was. No way to save it. Besides, presumably the OP wants to
keep using the delimiter set.

I'm not sure what the exact semantics of the Java call are, it would
have to be specified better. What I would do would be an approach more
like PHP explode(), and produce an array or list of strings. Then it
would be relatively easy to save the delimiters too.




Brian

Mark Bluemel wrote:
> Falcon Kirtaran wrote:
>> wrote:
>>> hi,
>>> i wanted to know whether we can use strtok command to mark delimiters
>>> as tokens as well.In Java,we have a command:
>
> [snip]
>
>> It'd be really easy to write your own, however, that preserves the
>> delimiting character. For instance, it might look something like:
>>
>> char * strtok_d(char * str, const char * tokens, char * delim) {
>
> [snip]
>
>> };
>>
>> The only thing you'd need to take care of is that both parameters must
>> be null-terminated strings. Also, you'd have to check if the third
>> parameter when dereferenced is not 0. Obviously, the third parameter
>> is just a single char (by reference), not a string.
>>
>> [snip] Calls to this can't be mixed interchangeably with calls to
>> strtok().
>
> And due to the (very dubious) design decision to hold state in a static
> variable, it is not thread-safe or even capable of being used on more
> than one string at a time, for example if you wanted to further parse
> a field extracted from a string before moving on to the next.
>
> There are better solutions out there, I'm sure.

strtok() itself is not thread-safe, however, and also can't be used on
more than one string at a time.

--
--Falcon Kirtaran

Richard Bos wrote:
> Falcon Kirtaran <> wrote:
>
>> wrote:
>>> i wanted to know whether we can use strtok command to mark delimiters
>>> as tokens as well.In Java,we have a command:
>
>> I don't think there is. This is because the function is generally
>> implemented by replacing the delimiter with a null.
>
> s/generally/required to be/.
>
>> It'd be really easy to write your own, however, that preserves the
>> delimiting character. For instance, it might look something like:
>>
>> char * strtok_d(char * str, const char * tokens, char * delim) {
>
> Don't call it that, though. Names starting in str and a letter are
> reserved for the implementation's use.
>
>> The only thing you'd need to take care of is that both parameters must
>> be null-terminated strings.
>
> No ****. This is generally true of string-mangling functions: they take
> strings, not randomly filled arrays.
>
> Another useful modification for your home-grown strtok() replacement
> could be (depending on your requirements) that it does not skip empty
> tokens. strtok() does so; this is sometimes desired behaviour, but to
> me, it's usually a hindrance.
>
> Richard

I don't think mine actually does skip empty tokens. If it finds an
empty token, it'll just return the next delimiter and a pointer to a
string that starts with null.

--
--Falcon Kirtaran

<> wrote in message
>
> Can strtok accomplish
> this requirement?or could you please let me know if there is any other
> command in C that would carry out this task?
>
sscanf() might be what you are looking for.

Alternatively, if you are implementing something like a BASIC interpreter,
you need a struct with the string, the read position, and the top token
read. Then you have a match() function to get rid of the current token and
get a new one, triggering an error if there is no match (for instance for
closing paretheses].


--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm