Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > question about speed of sequential string replacement vs regex or

Reply
Thread Tools

question about speed of sequential string replacement vs regex or

 
 
Xah Lee
Guest
Posts: n/a
 
      09-28-2011
curious question.

suppose you have 300 different strings and they need all be replaced
to say "aaa".

is it faster to replace each one sequentially (i.e. replace first
string to aaa, then do the 2nd, 3rd,...)
, or is it faster to use a regex with “or” them all and do replace one
shot? (i.e. "1ststr|2ndstr|3rdstr|..." -> aaa)

let's say the sourceString this replacement to be done on is 500k
chars.

Anyone? i suppose the answer will be similar for perl, python, ruby.

btw, the origin of this question is about writing a emacs lisp
function that replace ~250 html named entities to unicode char.

Xah
 
Reply With Quote
 
 
 
 
Randal L. Schwartz
Guest
Posts: n/a
 
      09-28-2011
>>>>> "Xah" == Xah Lee <(E-Mail Removed)> writes:

Xah> curious question.
Xah> suppose you have 300 different strings and they need all be replaced
Xah> to say "aaa".

And then suppose this isn't the *real* question, but one entirely of
Fiction by Xah Lee.

How helpful do you want to be?

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<(E-Mail Removed)> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.posterous.com/ for Smalltalk discussion
 
Reply With Quote
 
 
 
 
Xah Lee
Guest
Posts: n/a
 
      09-28-2011
On Sep 28, 3:57*am, (E-Mail Removed) (Randal L. Schwartz) wrote:
> >>>>> "Xah" == Xah Lee <(E-Mail Removed)> writes:

>
> Xah> curious question.
> Xah> suppose you have 300 different strings and they need all be replaced
> Xah> to say "aaa".
>
> And then suppose this isn't the *real* question, but one entirely of
> Fiction by Xah Lee.
>
> How helpful do you want to be?


it's a interesting question anyway.

the question originally came from when i was coding elisp of a
function that changes html entities to unicode char literal. The
problem is slightly complicated, involving a few questions about speed
in emacs. e.g. string vs buffer, and much more... i spent several
hours on this but it's probably too boring to detail (but i'll do so
if anyone wishes). But anyway, while digging these questions that's
not clear in my mind, i thought of why not generate a regex or
construct and do it in one shot, and wondered if that'd be faster. But
afterwards, i realized this wouldn't be applicable to my problem
because for my problem each string needs to be changed to a unique
string, not all to the same string.

Xah
 
Reply With Quote
 
Xah Lee
Guest
Posts: n/a
 
      09-28-2011
here's more detail about the origin of this problem. Relevant to emacs
lisp only.

------------------------------

in the definition of “replace-regexp-in-string”, there's this comment:

;; To avoid excessive consing from multiple matches in long strings,
;; don't just call `replace-match' continually. Walk down the
;; string looking for matches of REGEXP and building up a (reversed)
;; list MATCHES. This comprises segments of STRING which weren't
;; matched interspersed with replacements for segments that were.
;; [For a `large' number of replacements it's more efficient to
;; operate in a temporary buffer; we can't tell from the function's
;; args whether to choose the buffer-based implementation, though it
;; might be reasonable to do so for long enough STRING.]

note: «For a `large' number of replacements it's more efficient to
operate in a temporary buffer».

my question is, anyone got some idea, how “large” is large?

currently, i have a function replace-pairs-in-string which is
implemented by repeatedly calling “replace-pairs-in-string” like this:

(while (< ii (length pairs))
(setq mystr (replace-regexp-in-string
(elt tempMapPoints ii)
(elt (elt pairs ii) 1)
mystr t t))
(setq ii (1+ ii))
)

When there are 260 pairs of strings to be replaced on a file that's
26k in size, my function takes about 3 seconds (which i think is too
slow). I'm at pain deciding whether my function should be implemented
like this or whether it should create a temp buffer. The problem with
temp buffer is that, if you repeatedly call it, the overhead of
creating buffer is going to make it much slower.

i was actually surprised that replace-regexp-in-string isn't written
in C, which i thought it was.

is there technical reason the replace-regexp-in-string isn't C? (i
suppose only because nobody every did it and the need for speed didn't
suffice?) and also, shouldn't there also be a replace-in-string
(literal, not regex)? because i thought implementing replacement for
string should be much simpler and faster, because buffers comes with
it a whole structure such as “point”, text properties, buffer names,
buffier modifier, etc.

Xah

On Sep 28, 5:28*am, Xah Lee <(E-Mail Removed)> wrote:
> On Sep 28, 3:57*am, (E-Mail Removed) (Randal L. Schwartz) wrote:
>
> > >>>>> "Xah" == Xah Lee <(E-Mail Removed)> writes:

>
> > Xah> curious question.
> > Xah> suppose you have 300 different strings and they need all be replaced
> > Xah> to say "aaa".

>
> > And then suppose this isn't the *real* question, but one entirely of
> > Fiction by Xah Lee.

>
> > How helpful do you want to be?

>
> it's a interesting question anyway.
>
> the question originally came from when i was coding elisp of a
> function that changes html entities to unicode char literal. The
> problem is slightly complicated, involving a few questions about speed
> in emacs. e.g. string vs buffer, and much more... i spent several
> hours on this but it's probably too boring to detail (but i'll do so
> if anyone wishes). But anyway, while digging these questions that's
> not clear in my mind, i thought of why not generate a regex or
> construct and do it in one shot, and wondered if that'd be faster. But
> afterwards, i realized this wouldn't be applicable to my problem
> because for my problem each string needs to be changed to a unique
> string, not all to the same string.
>
> *Xah


 
Reply With Quote
 
Willem
Guest
Posts: n/a
 
      09-28-2011
Xah Lee wrote:
) the question originally came from when i was coding elisp of a
) function that changes html entities to unicode char literal. The
) problem is slightly complicated, involving a few questions about speed
) in emacs. e.g. string vs buffer, and much more... i spent several
) hours on this but it's probably too boring to detail (but i'll do so
) if anyone wishes). But anyway, while digging these questions that's
) not clear in my mind, i thought of why not generate a regex or
) construct and do it in one shot, and wondered if that'd be faster. But
) afterwards, i realized this wouldn't be applicable to my problem
) because for my problem each string needs to be changed to a unique
) string, not all to the same string.

In Perl, it would be applicable. You see, in Perl, you can call a function
in the replacement of the regex substitution, which can then look up the
html entity and return the wanted unicode literal.

I think you can do that in some other languages as well.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
 
Reply With Quote
 
Willem
Guest
Posts: n/a
 
      09-28-2011
Eli the Bearded wrote:
) In comp.lang.perl.misc, Willem <(E-Mail Removed)> wrote:
)> In Perl, it would be applicable. You see, in Perl, you can call a function
)> in the replacement of the regex substitution, which can then look up the
)> html entity and return the wanted unicode literal.
)
) A function? I'd use a hash.

A function can return a sensible value for unknown substitutions.
In the case where you prebuild a giant regex or-list, that is not an issue,
but I would match html entities generically.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
 
Reply With Quote
 
John Bokma
Guest
Posts: n/a
 
      09-28-2011
Willem <(E-Mail Removed)> writes:

> Eli the Bearded wrote:
> ) In comp.lang.perl.misc, Willem <(E-Mail Removed)> wrote:
> )> In Perl, it would be applicable. You see, in Perl, you can call a function
> )> in the replacement of the regex substitution, which can then look up the
> )> html entity and return the wanted unicode literal.
> )
> ) A function? I'd use a hash.
>
> A function can return a sensible value for unknown substitutions.


You can do that also in the RHS of the substitution and still keep it
readable if you use something like

s{..}{

your
code
goes
here
}ge;

However, a function can be easier on the eye:

s{...}{ some_good_name( ... ) }ge;

--
John Bokma j3b

Blog: http://johnbokma.com/ Perl Consultancy: http://castleamber.com/
Perl for books: http://johnbokma.com/perl/help-in-ex...for-books.html
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
question about speed of sequential string replacement vs regex or Xah Lee Python 15 09-28-2011 09:23 PM
regex =~ string or string =~ regex? Ruby Newbee Ruby 3 01-04-2010 06:04 PM
How make regex that means "contains regex#1 but NOT regex#2" ?? seberino@spawar.navy.mil Python 3 07-01-2008 03:06 PM
String.replaceAll(String regex, String replacement) question Mladen Adamovic Java 3 12-05-2003 04:20 PM
Re: String.replaceAll(String regex, String replacement) question Mladen Adamovic Java 0 12-04-2003 04:40 PM



Advertisments