Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > How to edit a file without loading it all into memory

Reply
Thread Tools

How to edit a file without loading it all into memory

 
 
Jonathan Dobbie
Guest
Posts: n/a
 
      11-09-2007
I need to change the texture paths in maya files. The lines are easy to
find with grep, and I'm able to create the new line easily enough, but
how can I do this without holding the entire file into memory? The
files can be hundreds of megabytes.

Thanks.
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
 
Reid Thompson
Guest
Posts: n/a
 
      11-09-2007
newer versions of sed can replace text in place in files.

there's also perl's

perl -pi -e 's/to change this/to change that/' filename


On Sat, 2007-11-10 at 05:51 +0900, Jonathan Dobbie wrote:
> I need to change the texture paths in maya files. The lines are easy to
> find with grep, and I'm able to create the new line easily enough, but
> how can I do this without holding the entire file into memory? The
> files can be hundreds of megabytes.
>
> Thanks.


 
Reply With Quote
 
 
 
 
Robert Klemme
Guest
Posts: n/a
 
      11-09-2007
On 09.11.2007 21:51, Jonathan Dobbie wrote:
> I need to change the texture paths in maya files. The lines are easy to
> find with grep, and I'm able to create the new line easily enough, but
> how can I do this without holding the entire file into memory? The
> files can be hundreds of megabytes.


As you said: do it line by line. This is the usual approach to this
problem. You might want to have a look at Ruby's command line options.
Basically you can do something like this for an inplace edit:

ruby -ni.bak -e 'your replacement code'

Kind regards

robert
 
Reply With Quote
 
7stud --
Guest
Posts: n/a
 
      11-09-2007
Jonathan Dobbie wrote:
> The lines are easy to
> find with grep, and I'm able to create the new line easily enough, but
> how can I do this without holding the entire file into memory? The
> files can be hundreds of megabytes.
>


File.foreach("data.txt") do |line|
#look for something in line
end

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Jonathan Dobbie
Guest
Posts: n/a
 
      11-09-2007
7stud -- wrote:
> Jonathan Dobbie wrote:
>> The lines are easy to
>> find with grep, and I'm able to create the new line easily enough, but
>> how can I do this without holding the entire file into memory? The
>> files can be hundreds of megabytes.
>>

>
> File.foreach("data.txt") do |line|
> #look for something in line
> end


Here is what I originally wrote, and what I just tried. Am I missing
something obvious? The first one duplicates the lines, and the second
one does nothing. (doesn't save the changes)

def change_texture_paths
File.open(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |file|
file.grep( /setAttr ".fileTextureName" -type "string"/ ) do |line|
texture_file_path = line[52..-2]
texture_file_name = texture_file_path.split("\/").last
line = " setAttr \".fileTextureName\" -type \"string\"
\"" + ENV['HOME'] + "/include/" + texture_file_name + "\n"
file.print line
end
end


File.foreach(Dir.glob(ENV['HOME'] + "/*.ma").first, "r+") do |line|
if (line =~ /setAttr ".fileTextureName" -type "string"/ )
texture_file_path = line[52..-2]
texture_file_name = texture_file_path.split("\/").last
line = " setAttr \".fileTextureName\" -type \"string\"
\"" + ENV['HOME'] + "/include/" + texture_file_name + "\n"
file.print line
end
end
end
--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
Kyle Schmitt
Guest
Posts: n/a
 
      11-09-2007
Err.

Well only one of these textures is open at once right?
If this box is actually capable of running maya well, it's sure to
have more than a few hundred megs of free ram (when maya isn't running
of course .

My guess is that you'll get a hefty performance hit for parsing the
files line by line. The box has memory to spare, so why not just use
it?

Of course if there was a way to memap the file under ruby that would
probably be the best...but I dunno how to do that.

--Kyle

 
Reply With Quote
 
Kyle Schmitt
Guest
Posts: n/a
 
      11-09-2007
mem map. not memap..but you get the idea.

 
Reply With Quote
 
7stud --
Guest
Posts: n/a
 
      11-10-2007
Jonathan Dobbie wrote:
> 7stud -- wrote:
>> Jonathan Dobbie wrote:
>>> The lines are easy to
>>> find with grep, and I'm able to create the new line easily enough, but
>>> how can I do this without holding the entire file into memory? The
>>> files can be hundreds of megabytes.
>>>

>>
>> File.foreach("data.txt") do |line|
>> #look for something in line
>> end

>
> Here is what I originally wrote, and what I just tried. Am I missing
> something obvious? The first one duplicates the lines, and the second
> one does nothing. (doesn't save the changes)
>


1) Are you under the impression that every method in ruby has the same
parameters?

2) When you see an error message, do you just ignore it?

--
Posted via http://www.ruby-forum.com/.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Create thumbnails without loading entire file into memory Phil Johnson ASP .Net 3 04-25-2008 07:57 PM
Snapshot restraint - edit, edit, edit Alan Browne Digital Photography 24 05-10-2005 10:15 PM
Snapshot restraint - edit, edit, edit Patrick Digital Photography 0 05-06-2005 10:53 PM
Need to concatenate all files in a dir together into one file and read the first 225 characters from each file into another file. Tony Perl Misc 5 04-19-2004 03:28 PM



Advertisments