FPGA to solve the two most annoying problems on usenet - SuggestionsWelcome

As you all know, downloading files from usenet leaves you with two
sets of files.. The rar files from what you're downloading and the
par2 files for incomplete file repair.

If anyone has attempted to download anything in the 8GB range, you'll
find that well.. if you're missing enough parts of the file, the par2
recovery can be a painful, painful process taking up to three hours in
some cases.

I'm sick of it.

So, here's the idea. Use a FPGA to do the reed-solomon decoding to
accelerate the PAR2 repair/recovery process. The system should utilize
a USB connection to pipe data directly from the disk to the FPGA that
will do the offboard processing of the data. The data transfer should
be controlled by an application on the computer.

Second Problem..

The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
Converting from x.264 to h.264 could be done offboard on an FPGA
because it takes for-ever to complete on my system (8 hours). This
should have the same premise as the previous issue, minus using a x.
264 decoding core and possibly directly converting it to h.264 or
doing a decompression-recompression..

SPECS: The development system i'm using is the XILINX ML-505 board
with the Virtex 5 chip. This is a open-source project being done for
fun and learning btw.

Suggestions / Comments / Complaints?

Charles Xavier

Seems like a really nice idea. Please notify your progress.

Atilla Filiz

> The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
> Converting from x.264 to h.264 could be done offboard on an FPGA
> because it takes for-ever to complete on my system (8 hours). This
> should have the same premise as the previous issue, minus using a x.
> 264 decoding core and possibly directly converting it to h.264 or
> doing a decompression-recompression..

There are chips that will do this already. Here's one:

http://www.aspex-semi.com/q/product_features.shtml

Cheers,
Jon

Jon Beniston

On Jun 11, 11:24*am, Jon Beniston <j...@beniston.com> wrote:
> > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
> > Converting from x.264 to h.264 could be done offboard on an FPGA
> > because it takes for-ever to complete on my system (8 hours). This
> > should have the same premise as the previous issue, minus using a x.
> > 264 decoding core and possibly directly converting it to h.264 or
> > doing a decompression-recompression..
>
> There are chips that will do this already. Here's one:
>
> http://www.aspex-semi.com/q/product_features.shtml
>
> Cheers,
> Jon

Thanks for the reply. Aspects of this design can certainly be used and
incorporated into this design, but as you know.. Their product isn't
free/open source. As this is also a learning project, I think it would
be better to build it from the ground up (using some pre-existing open
source IP if possible).

Charles Xavier

> So, here's the idea. Use a FPGA to do the reed-solomon decoding to
> accelerate the PAR2 repair/recovery process. The system should utilize
> a USB connection to pipe data directly from the disk to the FPGA that
> will do the offboard processing of the data. The data transfer should
> be controlled by an application on the computer.

Why USB ? Gigabit Ethernet would be much faster
Don't tell me a Virtex5 can't handle a measly 100 MB/s

And... is it a PCI express slot that I see there ?
That's even better !

> Second Problem..
>
> The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
> Converting from x.264 to h.264 could be done offboard on an FPGA
> because it takes for-ever to complete on my system (8 hours). This
> should have the same premise as the previous issue, minus using a x.
> 264 decoding core and possibly directly converting it to h.264 or
> doing a decompression-recompression..

I was under the impression that x264 was h264 in disguise. Perhaps
tweaking the file format would be enough ?

Also you could recode the thing in a much simpler codec like MPEG2 or
something, who cares if it takes 10x the size, just stream it in realtime


Stick a DVI port on your FPGA !

> SPECS: The development system i'm using is the XILINX ML-505 board
> with the Virtex 5 chip. This is a open-source project being done for
> fun and learning btw.

PFC

On Jun 13, 5:46*pm, Eric Smith <e...@brouhaha.com> wrote:
> Charles Xavier wrote:
> > If anyone has attempted to download anything in the 8GB range, you'll
> > find that well.. if you're missing enough parts of the file, the par2
> > recovery can be a painful, painful process taking up to three hours in
> > some cases.
> > The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
> > Converting from x.264 to h.264 could be done offboard on an FPGA
>
> I have a hard time believing that those are the "two most annoying problems
> on usenet". *I've been using Usenet since 1984, and I've never personally
> encountered either problem.
>
> I've never tried to download an 8GB file from Usenet, and don't have any
> clue why anyone would want to do such a thing. *There are much better
> ways to distributed 8GB files. *But if I *did* want to download an 8GB
> file from Usenet, I can't imagine that waiting three hours for a
> reconstruction of missing pieces would really bother me that much.
> There's never been a single thing that I've needed from Usenet so
> urgently that it couldn't have waited three more hours.
>
> I watch a fair number of movies, but I've never had any problem with
> x.264. *It sounds like you just need better player software. *That has
> nothing to do with Usenet.
>
> Since it doesn't appear that there is any real problem here, it also
> doesn't appear that there is any need for an FPGA-based "solution".

Thank you for sharing. Let your next post be in contribution to the
project.

If it doesn't apply to you and you don't see the need, why even reply?

Charles Xavier

Charles Xavier wrote:
> On Jun 13, 5:46 pm, Eric Smith <e...@brouhaha.com> wrote:
>> Charles Xavier wrote:
>>> If anyone has attempted to download anything in the 8GB range,
>>> you'll find that well.. if you're missing enough parts of the file,
>>> the par2 recovery can be a painful, painful process taking up to
>>> three hours in some cases.
>>> The XBOX 360 doesn't play x.264 and all the good movies are in
>>> x.264. Converting from x.264 to h.264 could be done offboard on an
>>> FPGA
>>
>> I have a hard time believing that those are the "two most annoying
>> problems on usenet". I've been using Usenet since 1984, and I've
>> never personally encountered either problem.
>>
>>
>> Since it doesn't appear that there is any real problem here, it also
>> doesn't appear that there is any need for an FPGA-based "solution".
>
> Thank you for sharing. Let your next post be in contribution to the
> project.
>
> If it doesn't apply to you and you don't see the need, why even reply?

Hi Charles,
Maybe to share the benefit of his experience to help you avoid wasting your
time on a solution without a problem?
Good luck, Syms.

Symon

On Jun 14, 12:39*pm, "Symon" <symon_bre...@hotmail.com> wrote:
> Charles Xavier wrote:
> > On Jun 13, 5:46 pm, Eric Smith <e...@brouhaha.com> wrote:
> >> Charles Xavier wrote:
> >>> If anyone has attempted to download anything in the 8GB range,
> >>> you'll find that well.. if you're missing enough parts of the file,
> >>> the par2 recovery can be a painful, painful process taking up to
> >>> three hours in some cases.
> >>> The XBOX 360 doesn't play x.264 and all the good movies are in
> >>> x.264. Converting from x.264 to h.264 could be done offboard on an
> >>> FPGA
>
> >> I have a hard time believing that those are the "two most annoying
> >> problems on usenet". I've been using Usenet since 1984, and I've
> >> never personally encountered either problem.
>
> >> Since it doesn't appear that there is any real problem here, it also
> >> doesn't appear that there is any need for an FPGA-based "solution".
>
> > Thank you for sharing. Let your next post be in contribution to the
> > project.
>
> > If it doesn't apply to you and you don't see the need, why even reply?
>
> Hi Charles,
> Maybe to share the benefit of his experience to help you avoid wasting your
> time on a solution without a problem?
> Good luck, Syms.

Engineers can often be of a single track mind, so let me restate. This
is a problem that I am having. I could certainly get a faster
computer, or try to finesse the algorithms for the decoding, but I
want to learn about this specific algorithm implementation on an FPGA,
so I'm using the FPGA to solve this problem (that is a problem for
me).

To me, this is a practical solution and an excellent gateway for
myself and others who want to get into FPGA-based hardware
acceleration of software algorithms. You give me a better way that
solves a practical problem for _me_ and I'll take it.

This reminds of the idiots at In-N-Out Burger who created the drive-
through fast food restaurant (they claim). It doesn't affect the
geriatric who only wants to eat inside because they can't imagine
driving while eating. But apparently it turned out to be a good idea.

Responding to this type of feedback is the real waste of time.

Charles Xavier

On 2008-06-11, Charles Xavier <> wrote:
> The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
> Converting from x.264 to h.264 could be done offboard on an FPGA
> because it takes for-ever to complete on my system (8 hours). This
> should have the same premise as the previous issue, minus using a x.
> 264 decoding core and possibly directly converting it to h.264 or
> doing a decompression-recompression..

I'm not sure what the problem with x.264 is, but lets just say that
it is far from trivial to write a H.264 decoder/encoder. The decoder
is especially tricky as you need to be able to parse all parts of the
standard whereas you can select only the parts you care about for the
encoder.

If you are an expert on video codecs and a newcomer to FPGAs, this
would probably be a quite ambitiouis project to start with. If you
are not an expert on video codecs, you will have to spend a lot of
time to understand H.264 before you can even get started on the FPGA
parts.

This is written from personal experience by the way, I have written
a grayscale H.264 decoder in C to get an understanding of how the
standard works. Lets just say that it took way more time than I wanted
to...

In summary: I don't recommend a H.264 codec as a first FPGA project.

The Reed-Solomon application seems like a much more reasonable project.
You will still have a lot of things to figure out like how you are going
to communicate with your FPGA board, but the problem itself is much more
limited.

/Andreas

Andreas Ehliar

I have an idea simular or not to your mind..(but I still
thinking....hope you are now starting ..your project).
my project is that

FILE COMPRESSION (zip or other)
read files from SD card, then compress it, and write a new zipped file
back to SD card.

it will not include the PC to do it.

On Jun 11, 9:50*am, Charles Xavier <skelo...@gmail.com> wrote:
> As you all know, downloading files from usenet leaves you with two
> sets of files.. The rar files from what you're downloading and the
> par2 files for incomplete file repair.
>
> If anyone has attempted to download anything in the 8GB range, you'll
> find that well.. if you're missing enough parts of the file, the par2
> recovery can be a painful, painful process taking up to three hours in
> some cases.
>
> I'm sick of it.
>
> So, here's the idea. Use a FPGA to do the reed-solomon decoding to
> accelerate the PAR2 repair/recovery process. The system should utilize
> a USB connection to pipe data directly from the disk to the FPGA that
> will do the offboard processing of the data. The data transfer should
> be controlled by an application on the computer.
>
> Second Problem..
>
> The XBOX 360 doesn't play x.264 and all the good movies are in x.264.
> Converting from x.264 to h.264 could be done offboard on an FPGA
> because it takes for-ever to complete on my system (8 hours). This
> should have the same premise as the previous issue, minus using a x.
> 264 decoding core and possibly directly converting it to h.264 or
> doing a decompression-recompression..
>
> SPECS: The development system i'm using is the XILINX ML-505 board
> with the Virtex 5 chip. This is a open-source project being done for
> fun and learning btw.
>
> Suggestions / Comments / Complaints?

Aiken