Q: Which one is better for parsing 30MB files ? ATL/COM or VB/COM

Discussion in 'MCSD' started by Ramie, Oct 30, 2003.

  1. Ramie

    Ramie Guest

    Hello,

    I am working on a B2B web serivce application. I am
    Systems Architect for this application. I am in the
    process of desigining my components. I would like to know
    which one is better over the other for parsing 30MB
    files. We would like to use one of the following two
    choices: ATL/COM or VB/COM. Can you please share your
    thoughts on this ?

    Regards,
    Ramie
     
    Ramie, Oct 30, 2003
    #1
    1. Advertisements

  2. Ramie

    Jay Walters Guest

    Until another response is posted, I would go with VB only
    because of future maintainability.

    Obviously I don't know your requirements.

    Is this file created from the webservice?, and after it's
    creation you need to call a component to do something
    with it?

    Or is the file a feed from a partner that's updated at
    different times (outside of the web service)? If so you
    might consider using an NT service and filewatcher.
     
    Jay Walters, Oct 30, 2003
    #2
    1. Advertisements

  3. Ramie

    Ramie Guest

    Sorry. I did not specify my requirements cleary. Web
    Service gets the input file (as big as 30MB) via HTTP
    post. The user posts file to an ASP page. Then, the ASP
    page will call MTS component, which will parse the file to
    extract three piece of info: User Id, Password, and some
    text data (called as BLOB). Now, the question is on
    choosing the language to develop the MTS component (called
    as App Controller), in VB/COM or ATL/COM ?
     
    Ramie, Oct 30, 2003
    #3
  4. Ramie

    Guest Guest

    XML may not be the best method of moving 30mb of data across the internet,
    especially is most of it is a BLOB.

    Let me bounce this idea off you:

    1. Store your BLOB in a text or binary file, store any header or
    structured info in a smaller .xml or comma delimited file.

    2. ZIP the files on the client and then FTP to the server.

    3. Bulk load the data into SQL Server using DTS or BCP.

    WKidd
     
    Guest, Oct 30, 2003
    #4
  5. Ramie

    Kline Sphere Guest

    Depends on the format of the files and what information you are
    parsing for.
     
    Kline Sphere, Oct 30, 2003
    #5
  6. Ramie

    Kline Sphere Guest

    Web
    Review your architectural design. I mean, 30 megs via a web service
    using asp, then processing through an mts component.....
     
    Kline Sphere, Oct 30, 2003
    #6
  7. Ramie

    Kline Sphere Guest

    2. ZIP the files on the client and then FTP to the server.

    Better still use a compressing stream, i.e.
    GZIPInputStream/GZIPOutputStream or the like of
     
    Kline Sphere, Oct 30, 2003
    #7
  8. Ramie

    Jay Walters Guest

    Yeah - I've used Gzip and it's good. I serialized a large
    dataset using the binaryFormatter to a memorystream, get
    and Gzipped it, and converted it to a byte() to send it
    accross the wire and it worked great!
     
    Jay Walters, Oct 30, 2003
    #8
  9. Ramie

    Kline Sphere Guest

    Cool!
     
    Kline Sphere, Oct 30, 2003
    #9
  10. Ramie

    Ramie Guest

    Architecture for the Web Service is fixed, or shoud say,
    dictated by the user. We must use HTTP as the
    transportation protocol, and data is in ASCII (X12)
    format, which is encrypted before posted to the web
    service. Data is posted to an ASP page in multi-part
    binary post. Therefore, I do not have architecture design
    choice between HTTP and FTP.
     
    Ramie, Oct 30, 2003
    #10
  11. Ramie

    Kline Sphere Guest

    Architecture for the Web Service is fixed, or shoud say,
    Deflate it before it is sent. Having said that, If the number of times
    this 'process' gets used is small and operates through vpn or like of
    (thus not open to the whole wide world to bring your network to it's
    knees), it won't make too much difference.

    Is there no way this information can be 'parsed' at the client end?
    Thus only returning the relevant information.
    Seems a bit over kill when a simple socket will do the job.

    Processing the file in an MTS component is a very bad move. Operations
    conducted in as an environment should be completed in milliseconds, I
    think what you will be doing will take somewhat longer than that. In
    any case given the time the file may take to 'download', the choice of
    programming language used to parse the file will not make too much
    difference, although my choice for COM programming would always be ATL
    An encrypted socket would be faster, but if you're stuck to using what
    some dumb client reckons, oh well could be worse - I think!
     
    Kline Sphere, Oct 30, 2003
    #11
  12. Sorry, but coding it in VB won't give you future maintainability. VB
    won't be around in 20 years but C++ will always be around.
     
    General Protection Fault, Oct 30, 2003
    #12
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.