Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Simple question: combine a quoted string into a single token

Reply
Thread Tools

Simple question: combine a quoted string into a single token

 
 
Squeamizh
Guest
Posts: n/a
 
      08-07-2006
Hi,

I have a program which separates each line of a text file into tokens,
using whitespace as a delimiter (I do this with String.split). This
suits my needs for the most part, but now I need the ability to treat
quoted strings as single tokens. Note that the quoted string could be
multiple words, or even a 0-length string.

Could anyone recommend a basic strategy for doing this? Should I deal
with this when I first tokenize each line, or should I combine tokens
appropriately during parsing when I see a double-quote?

Help would be greatly appreciated.

 
Reply With Quote
 
 
 
 
Kevin Olbrich
Guest
Posts: n/a
 
      08-07-2006

On Tuesday, August 08, 2006, at 2:40 AM, Squeamizh wrote:
>Hi,
>
>I have a program which separates each line of a text file into tokens,
>using whitespace as a delimiter (I do this with String.split). This
>suits my needs for the most part, but now I need the ability to treat
>quoted strings as single tokens. Note that the quoted string could be
>multiple words, or even a 0-length string.
>
>Could anyone recommend a basic strategy for doing this? Should I deal
>with this when I first tokenize each line, or should I combine tokens
>appropriately during parsing when I see a double-quote?
>
>Help would be greatly appreciated.
>
>


You could pull out all the quoted strings into an array and then delete
them from the original before processing it normally.


_Kevin
www.sciwerks.com

--
Posted with http://DevLists.com. Sign up and save your mailbox.

 
Reply With Quote
 
 
 
 
Robert Klemme
Guest
Posts: n/a
 
      08-07-2006
Squeamizh wrote:
> Hi,
>
> I have a program which separates each line of a text file into tokens,
> using whitespace as a delimiter (I do this with String.split). This
> suits my needs for the most part, but now I need the ability to treat
> quoted strings as single tokens. Note that the quoted string could be
> multiple words, or even a 0-length string.
>
> Could anyone recommend a basic strategy for doing this? Should I deal
> with this when I first tokenize each line, or should I combine tokens
> appropriately during parsing when I see a double-quote?
>
> Help would be greatly appreciated.
>


Something along the lines of

line.scan %r{
"[^"]*" |
\S+
}x

HTH

robert
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
IMAP: unexpected token LPAR (expected QUOTED or LITERAL) Shea Barton Ruby 2 08-17-2010 05:11 PM
"EOL while scanning single-quoted string", what is the meaning? excite Python 0 11-02-2006 10:57 PM
Re: EOL - scanning single-quoted string Ajay Python 2 08-04-2004 03:09 PM
EOL - scanning single-quoted string Ajay Python 0 08-04-2004 01:20 PM
preprocessor, token concatenation, no valid preprocessor token Cronus C++ 1 07-14-2004 11:10 PM



Advertisments