Q: Which one is better for parsing 30MB files ? ATL/COM or VB/COM

Hello,

I am working on a B2B web serivce application. I am
Systems Architect for this application. I am in the
process of desigining my components. I would like to know
which one is better over the other for parsing 30MB
files. We would like to use one of the following two
choices: ATL/COM or VB/COM. Can you please share your
thoughts on this ?

Regards,
Ramie

 

Until another response is posted, I would go with VB only
because of future maintainability.

Obviously I don't know your requirements.

Is this file created from the webservice?, and after it's
creation you need to call a component to do something
with it?

Or is the file a feed from a partner that's updated at
different times (outside of the web service)? If so you
might consider using an NT service and filewatcher.

 

Sorry. I did not specify my requirements cleary. Web
Service gets the input file (as big as 30MB) via HTTP
post. The user posts file to an ASP page. Then, the ASP
page will call MTS component, which will parse the file to
extract three piece of info: User Id, Password, and some
text data (called as BLOB). Now, the question is on
choosing the language to develop the MTS component (called
as App Controller), in VB/COM or ATL/COM ?

 

XML may not be the best method of moving 30mb of data across the internet,
especially is most of it is a BLOB.

Let me bounce this idea off you:

1. Store your BLOB in a text or binary file, store any header or
structured info in a smaller .xml or comma delimited file.

2. ZIP the files on the client and then FTP to the server.

3. Bulk load the data into SQL Server using DTS or BCP.

WKidd

 

Depends on the format of the files and what information you are
parsing for.

 

Web

Review your architectural design. I mean, 30 megs via a web service
using asp, then processing through an mts component.....

 

2. ZIP the files on the client and then FTP to the server.

Better still use a compressing stream, i.e.
GZIPInputStream/GZIPOutputStream or the like of

 

Yeah - I've used Gzip and it's good. I serialized a large
dataset using the binaryFormatter to a memorystream, get
and Gzipped it, and converted it to a byte() to send it
accross the wire and it worked great!

 

Cool!

 

Architecture for the Web Service is fixed, or shoud say,
dictated by the user. We must use HTTP as the
transportation protocol, and data is in ASCII (X12)
format, which is encrypted before posted to the web
service. Data is posted to an ASP page in multi-part
binary post. Therefore, I do not have architecture design
choice between HTTP and FTP.

 

Architecture for the Web Service is fixed, or shoud say,

Deflate it before it is sent. Having said that, If the number of times
this 'process' gets used is small and operates through vpn or like of
(thus not open to the whole wide world to bring your network to it's
knees), it won't make too much difference.

Is there no way this information can be 'parsed' at the client end?
Thus only returning the relevant information.

Seems a bit over kill when a simple socket will do the job.

Processing the file in an MTS component is a very bad move. Operations
conducted in as an environment should be completed in milliseconds, I
think what you will be doing will take somewhat longer than that. In
any case given the time the file may take to 'download', the choice of
programming language used to parse the file will not make too much
difference, although my choice for COM programming would always be ATL

An encrypted socket would be faster, but if you're stuck to using what
some dumb client reckons, oh well could be worse - I think!

 

Sorry, but coding it in VB won't give you future maintainability. VB
won't be around in 20 years but C++ will always be around.