Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Avoiding fragmentation

Reply
Thread Tools

Avoiding fragmentation

 
 
Roedy Green
Guest
Posts: n/a
 
      03-18-2011
I just notice how when I create files with Java, they often are badly
fragmented, with ten or more fragments.

This could be avoided with a simple strategy. When you open the file
for write you specify the estimated size of the file. The OS then
looks for a continuous hunk of disk space to satisfy. If you guess too
big, you return the tail to the free space pool. If you guess too
small, it adds another extent or two.
--
Roedy Green Canadian Mind Products
http://mindprod.com
If you think it’s expensive to hire a professional to do the job, wait until you hire an amateur.
~ Red Adair

 
Reply With Quote
 
 
 
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      03-18-2011
In message <(E-Mail Removed)>, Roedy Green wrote:

> I just notice how when I create files with Java, they often are badly
> fragmented, with ten or more fragments.


On Windows, is it?
 
Reply With Quote
 
 
 
 
Owen Jacobson
Guest
Posts: n/a
 
      03-19-2011
On 2011-03-18 15:33:45 -0400, Roedy Green said:

> I just notice how when I create files with Java, they often are badly
> fragmented, with ten or more fragments.
>
> This could be avoided with a simple strategy. When you open the file
> for write you specify the estimated size of the file. The OS then
> looks for a continuous hunk of disk space to satisfy. If you guess too
> big, you return the tail to the free space pool. If you guess too
> small, it adds another extent or two.


You can do that, or something much like it, by building up larger
chunks of the file in memory before writing them to disk (using NIO,
RandomAccessFile, your own buffering, or BufferedOutputStream). There's
no guarantee that the filesystem will fragment less given larger
writes, but a large block written at once at least propagates some size
information to the OS, whereas a small write to a stream indicates
nearly nothing about the volume of data following it.

This isn't really a Java issue, though: the lowest common denominator
for file creation is POSIX-like, which assumes infintely-appendable
files with no fixed size (or size hints). Blame Bell, basically.

-o

 
Reply With Quote
 
Joshua Cranmer
Guest
Posts: n/a
 
      03-19-2011
On 03/18/2011 07:43 PM, Lawrence D'Oliveiro wrote:
> In message<(E-Mail Removed) >, Roedy Green wrote:
>
>> I just notice how when I create files with Java, they often are badly
>> fragmented, with ten or more fragments.

>
> On Windows, is it?


Actually, this may come as a shock to you, but Linux filesystems can and
do have fragmentation problems. See
<http://blog.mozilla.com/tglek/2010/07/22/file-fragmentation/> for some
statistics (one of my files had "only" 400 fragments). But, hey, why let
truth get in the way of Windows-bashing?

--
Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth
 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      03-19-2011
In message <im19mj$73k$(E-Mail Removed)-september.org>, Joshua Cranmer wrote:

> On 03/18/2011 07:43 PM, Lawrence D'Oliveiro wrote:
>
>> In message<(E-Mail Removed) >, Roedy Green
>> wrote:
>>
>>> I just notice how when I create files with Java, they often are badly
>>> fragmented, with ten or more fragments.

>>
>> On Windows, is it?

>
> Actually, this may come as a shock to you, but Linux filesystems can and
> do have fragmentation problems.


And yet I have never known any of my customers’ Linux servers—or my own
Linux machines, for that matter—to slow down over time.
 
Reply With Quote
 
Joshua Cranmer
Guest
Posts: n/a
 
      03-19-2011
On 03/19/2011 01:56 AM, Lawrence D'Oliveiro wrote:
> In message<im19mj$73k$(E-Mail Removed)-september.org>, Joshua Cranmer wrote:
>
>> On 03/18/2011 07:43 PM, Lawrence D'Oliveiro wrote:
>>
>>> In message<(E-Mail Removed) >, Roedy Green
>>> wrote:
>>>
>>>> I just notice how when I create files with Java, they often are badly
>>>> fragmented, with ten or more fragments.
>>>
>>> On Windows, is it?

>>
>> Actually, this may come as a shock to you, but Linux filesystems can and
>> do have fragmentation problems.

>
> And yet I have never known any of my customers’ Linux servers—or my own
> Linux machines, for that matter—to slow down over time.


And I have sped up cold startup by an order of magnitude just by
defragging my harddrive. Note that Linux's filesystems are geared more
towards server use, not towards desktop use.

--
Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      03-19-2011
On 03/19/2011 09:21 AM, Joshua Cranmer wrote:
> On 03/19/2011 01:56 AM, Lawrence D'Oliveiro wrote:
>> In message<im19mj$73k$(E-Mail Removed)-september.org>, Joshua Cranmer wrote:
>>
>>> On 03/18/2011 07:43 PM, Lawrence D'Oliveiro wrote:
>>>
>>>> In message<(E-Mail Removed) >, Roedy Green
>>>> wrote:
>>>>
>>>>> I just notice how when I create files with Java, they often are badly
>>>>> fragmented, with ten or more fragments.
>>>>
>>>> On Windows, is it?


I am going to undo your presumption in eliding vital parts of Joshua's post,
to revert the effect of your intellectual dishonesty, "Lawrence".

>>> Actually, this may come as a shock to you, but Linux filesystems can and
>>> do have fragmentation problems.
>>> See <http://blog.mozilla.com/tglek/2010/07/22/file-fragmentation/>
>>> for some statistics (one of my files had "only" 400 fragments).
>>> But, hey, why let truth get in the way of Windows-bashing?


>> And yet I have never known any of my customers’ Linux servers—or my own
>> Linux machines, for that matter—to slow down over time.


Not surprising. Also not surprising that you use your own very, very limited
personal experience to try to countervail the documented, technical evidence
that Joshua presented, then *omit* the reference lest the blinding light of
truth reveal your pathetic ignorance, "Lawrence".

To paraphrase Joshua, "But, hey, 'Lawrence', why let truth get in the way of
your ignorance?"

> And I have sped up cold startup by an order of magnitude just by defragging my
> harddrive. Note that Linux's filesystems are geared more towards server use,
> not towards desktop use.


So Joshua not only has facts, dear boy, but personal experience that supports
the truth and not your narrow, blindered and inaccurate view of the world,
"Lawrence". So, "Larry"-boy, how 'bout you accept truth and allow your world
view to expand? It's not in your own best interest to fight facts, "Lar".

--
Lew
Honi soit qui mal y pense.
 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      03-20-2011
In message <im2alc$lv9$(E-Mail Removed)-september.org>, Joshua Cranmer wrote:

> And I have sped up cold startup by an order of magnitude just by
> defragging my harddrive.


On a Linux system?

> Note that Linux's filesystems are geared more towards server use, not
> towards desktop use.


What does that mean?
 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      03-20-2011
On Fri, 18 Mar 2011 22:46:54 -0400, Owen Jacobson
<(E-Mail Removed)> wrote, quoted or indirectly quoted someone
who said :

>
>This isn't really a Java issue, though: the lowest common denominator


If the various Java stream open methods developed an optional size
parameter, either that fact could be passed to the OS for
optimisation, or Java could explicitly allocate that much space, as
per random i/o, and then discard the excess on close.

I remember discovering that writing 0 byte record in DOS was the way
to chop the file. At first I thought it was a bug, then a joke.
--
Roedy Green Canadian Mind Products
http://mindprod.com
If you think it’s expensive to hire a professional to do the job, wait until you hire an amateur.
~ Red Adair

 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      03-20-2011
On Fri, 18 Mar 2011 12:33:45 -0700, Roedy Green
<(E-Mail Removed)> wrote, quoted or indirectly quoted
someone who said :

>This could be avoided with a simple strategy. When you open the file
>for write you specify the estimated size of the file. The OS then
>looks for a continuous hunk of disk space to satisfy. If you guess too
>big, you return the tail to the free space pool. If you guess too
>small, it adds another extent or two.


The current approach is to use after-the-fact defragging. I have
written extensive reviews on Windows defraggers.

See http://mindprod.com/jgloss/defragger.html

This does you no good at all for temp files or short-lived files.

--
Roedy Green Canadian Mind Products
http://mindprod.com
If you think it’s expensive to hire a professional to do the job, wait until you hire an amateur.
~ Red Adair

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Will memory upgrade help my SOHO77 fragmentation problems? Arnoud Helmantel Cisco 13 10-18-2004 06:57 PM
packet fragmentation with ipsec users edward Cisco 0 06-29-2004 10:25 AM
The proper size of ping packet (is fragmentation allowed) Yang Jun Cisco 6 12-20-2003 01:42 PM
Re: tough question: MTU Path discovery , IPsec, VPN, Pre-fragmentation. Nick Bailey Cisco 0 10-20-2003 07:57 PM
l2tp fragmentation prob mac Cisco 1 07-12-2003 02:20 PM



Advertisments