Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > How to Parse a CSV formatted text file

Reply
Thread Tools

How to Parse a CSV formatted text file

 
 
Ram Laxman
Guest
Posts: n/a
 
      02-07-2004
Hi all,
I have a text file which have data in CSV format.
"empno","phonenumber","wardnumber"
12345,2234353,1000202
12326,2243653,1000098
Iam a beginner of C/C++ programming.
I don't know how to tokenize the comma separated values.I used strtok
function reading line by line using fgets.but it gives some weird
behavior.It doesnot stripout the "" fully.Could any body have sample
code for the same so that it will be helfful for my reference?

Ram Laxman



Ram Laxman
 
Reply With Quote
 
 
 
 
Phlip
Guest
Posts: n/a
 
      02-07-2004
Ram Laxman wrote:

> I have a text file which have data in CSV format.
> "empno","phonenumber","wardnumber"
> 12345,2234353,1000202
> 12326,2243653,1000098
> Iam a beginner of C/C++ programming.
> I don't know how to tokenize the comma separated values.I used strtok
> function reading line by line using fgets.but it gives some weird
> behavior.It doesnot stripout the "" fully.Could any body have sample
> code for the same so that it will be helfful for my reference?


Parsing is tricky. Consider these rules:

- \n is absolute. All lines must be unbroken
- "" precedes , - so commas inside strings are text, not delimiters
- quotes inside "" need an escape, either \n or ""
- escapes need escapes - \\ is \

Try this project to learn more:

http://c2.com/cgi/wiki?MsWindowsResourceLint

First, we express those rules (one by one) as test cases:

TEST_(TestCase, pullNextToken_comma)
{

Source aSource("a , b\nc, \n d");

string
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("a", token);
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("b", token);
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("c", token);
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("d", token);
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("", token);
// EOF!

}

struct
TestTokens: TestCase
{

void
test_a_b_d(string input)
{
Source aSource(input);
string
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("a", token);
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("b", token);
// token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("c",
token);
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("d", token);
token = aSource.pullNextToken(); CPPUNIT_ASSERT_EQUAL("", token);
// EOF!
}

};

TEST_(TestTokens?, elideComments)
{
test_a_b_d("a b\n //c\n d");
test_a_b_d("a b\n//c \n d");
test_a_b_d("a b\n // c \"neither\" \n d");
test_a_b_d("a b\n // c \"neither\" \n d//");
test_a_b_d("//\na b\n // c \"neither\" \n d//");
test_a_b_d("//c\na b\n // c \"neither\" \n d//");
test_a_b_d("// c\na b\n // c \"neither\" \n d//");
test_a_b_d("//c \na b\n // c \"neither\" \n d//");
test_a_b_d("// \na b\n // c \"neither\" \n d//");
test_a_b_d(" // \na b\n // c \"neither\" \n d//");
}

TEST_(TestTokens?, elideStreamComments)
{
test_a_b_d("a b\n /*c*/\n d");
test_a_b_d("a b\n/*c*/ \n d");
test_a_b_d("a b\n /* c \"neither\" */\n d");
test_a_b_d("a b\n /* c \"neither\" \n */ d//");
test_a_b_d("//\na b\n /* c \"neither\" */ \n d/**/");
test_a_b_d("//c\na b\n // c \"neither\" \n d/* */");
test_a_b_d("/* c\n*/a b\n // c \"neither\" \n d//");
test_a_b_d("//c \na b\n // c \"neither\" \n d//");
test_a_b_d("// \na b\n // c \"neither\" \n d//");
test_a_b_d(" // \na b\n // c \"neither\" \n d//");
}

Those tests re-use the fixture test_a_b_d() to ensure that every one of
those strings parse into a, b, & d, skipping (for whatever reason) c.

You will need tests that show slightly different behaviors. But write your
tests one at a time. I wrote every single line you see here, essentially in
order, and got it to work before adding the next line. Don't write all your
tests at once, because when programming you should never go more than 1~10
edits before passing all tests.

Now here's the source of Source (which means "source of tokens"):

class
Source
{
public:
Source(string const & rc = ""):
m_rc(rc),
m_bot(0),
m_eot(0)
{}

void setResource(string const & rc) { m_rc = rc; }
size_type getBOT() { return m_bot; }
string const & getPriorToken() { return m_priorToken; }
string const & getCurrentToken() { return m_currentToken; }

string const &
pullNextToken()
{
m_priorToken = m_currentToken;
extractNextToken();
return m_currentToken;
}

size_type
getLineNumber(size_type at)
{
size_type lineNumber = 1;

for(size_type idx(0); idx < at; ++idx)
if ('\n' == m_rc[idx])
++lineNumber;

return lineNumber;
}

string
getLine(size_type at)
{
size_type bol = m_rc.rfind('\n', at);
if (string::npos == bol) bol = 0; else ++bol;
size_type eol = m_rc.find('\n', at);
if (string::npos == eol) eol = m_rc.length(); else ++eol;
return m_rc.substr(bol, eol - bol);
}

private:

string const &
extractNextToken()
{
char static const delims[] = " \t\n,";

m_bot = m_rc.find_first_not_of(delims, m_eot);

if (string::npos == m_bot)
m_currentToken = "";
else if (m_rc[m_bot] == '"')
m_currentToken = parseString();
else if (m_rc.substr(m_bot, 2) == "//")
{
if (skipUntil("\n"))
return extractNextToken();
}
else if (m_rc.substr(m_bot, 2) == "/*")
{
if (skipUntil("*/"))
return extractNextToken();
}
/* else if (m_rc.substr(m_bot, 1) == "#")
{
string line = getLine(m_bot);
size_type at(0);
while(isspace(line[at]) && at < line.size()) ++at;


if ('#' == line[at])
{
m_eot = m_bot + 1;
if (skipUntil("\n"))
return extractNextToken();
}
}*/
else
{
m_eot = m_rc.find_first_of(" \n,/", m_bot);
m_currentToken = m_rc.substr(m_bot, m_eot - m_bot);
}

if ('#' == m_currentToken[0])
{
// assert(m_rc.substr(m_bot, 1) == "#");
string line = getLine(m_bot);
size_type at(0);
while(isspace(line[at]) && at < line.size()) ++at;

if ('#' == line[at])
{
--m_eot;
if (skipUntil("\n"))
return extractNextToken();
}
}
return m_currentToken;
}

bool
skipUntil(char const * delimiter)
{
m_eot = m_rc.find(delimiter, m_eot + 1);

if (string::npos == m_eot)
{
m_currentToken = "";
return false;
}
m_eot += strlen(delimiter);
return true;
}

char
parseStringChar()
{
if (m_rc[m_eot] == '\\')
{
m_eot += 1;
char escapee(m_rc[m_eot++]);

switch (escapee)
{
case 'n' : return '\n';
case 'r' : return '\r';
case 't' : return '\t';
case '0' : return '\0';
case '\\': return '\\';
case 'a' : return '\a';
default : // TODO \x, \v \b, \f
if (isdigit(escapee))
{
string slug = m_rc.substr(m_eot - 1, 3);
return char(strtol(slug.c_str(), NULL, );
}
else
//assert(false);
return escapee;
}
}
else if (m_rc[m_eot] == '"' && m_rc[m_eot+1] == '"')
m_eot++;

return m_rc[m_eot++];
}

string
parseString()
{
m_eot = m_bot + 1;
string z;

while ( m_eot < m_rc.length() &&
( m_rc[m_eot] != '"' ||
m_rc[m_eot + 1] == '"' ) )
z += parseStringChar();

if (m_eot < m_rc.length())
m_eot += 1;

return z;
}

string m_rc;
size_type m_bot;
size_type m_eot;
string m_priorToken;
string m_currentToken;
};

That looks really ugly & long, because it hides so much behind such a narrow
interface. (I don't know if I copied all of it in, either.) But it
demonstrates (possibly) correct usage of std::string and std::vector.

Do not copy my source into your editor and try to run it. It will not parse
CVS. Start your project like this:

#include <assert.h>
#include <string>
#include <vector>
typedef std::vector<std::string> strings_t;

strings_t parse(std::string input)
{
strings_t result;
return result;
}

int main()
{
assert("a" == parse("a,b")[0]);
}

If that compiles, it >will< crash if you run it.

Now fix parse() so that it _only_ does not crash, and passes this test. Make
the implementation as stupid as you like.

Then add a test:

assert("a" == parse("a,b")[0]);
assert("b" == parse("a,b")[1]);

Keep going. Make the implementation just a little better after each test.
Write a set of tests for each of the parsing rules I listed. When the new
parse() function is full-featured, put it to work in your program.

All programs should be written by generating long lists of simple tests like
this. That keeps the bug count very low, and prevents wasting hours and
hours with a debugger.

--
Phlip
http://www.xpsd.org/cgi-bin/wiki?Tes...UserInterfaces


 
Reply With Quote
 
 
 
 
Phlip
Guest
Posts: n/a
 
      02-07-2004
Willem wrote:

> Ram wrote:
> ) Hi all,
> ) I have a text file which have data in CSV format.
> ) "empno","phonenumber","wardnumber"
> ) 12345,2234353,1000202
> ) 12326,2243653,1000098
> ) Iam a beginner of C/C++ programming.
> ) I don't know how to tokenize the comma separated values.I used strtok
> ) function reading line by line using fgets.but it gives some weird
> ) behavior.It doesnot stripout the "" fully.Could any body have sample
> ) code for the same so that it will be helfful for my reference?
>
> Here's a tip: Look for a library that scans CSV files.


Hi Willem! Welcome to the first hard projects of this semester. So far, a
professor somewhere has assumed their class was reading the right chapters
in their tutorial, and has hit them with the first non-Hello World project.

Someone just posted the same question to news:comp.programming .

--
Phlip
http://www.xpsd.org/cgi-bin/wiki?Tes...UserInterfaces


 
Reply With Quote
 
Willem
Guest
Posts: n/a
 
      02-07-2004
Ram wrote:
) Hi all,
) I have a text file which have data in CSV format.
) "empno","phonenumber","wardnumber"
) 12345,2234353,1000202
) 12326,2243653,1000098
) Iam a beginner of C/C++ programming.
) I don't know how to tokenize the comma separated values.I used strtok
) function reading line by line using fgets.but it gives some weird
) behavior.It doesnot stripout the "" fully.Could any body have sample
) code for the same so that it will be helfful for my reference?

Here's a tip: Look for a library that scans CSV files.

And if you really want to do it yourself, you really don't want to be using
stuff like strtok. Assuming you have one complete line in memory, you're
better off searching for the commas (and quotes) yourself, that's really
not so hard. Just put NULs where the commas are, and point to the
beginning of the strings (just after the comma). You can then pass these
pointers as strings to another parsing routine that turns stuff without
quotes into integers, and stuff with quotes into strings or whatever.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
 
Reply With Quote
 
Mike Wahler
Guest
Posts: n/a
 
      02-07-2004

"Ram Laxman" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) om...
> Hi all,
> I have a text file which have data in CSV format.
> "empno","phonenumber","wardnumber"
> 12345,2234353,1000202
> 12326,2243653,1000098
> Iam a beginner of C/C++ programming.
> I don't know how to tokenize the comma separated values.I used strtok
> function reading line by line using fgets.but it gives some weird
> behavior.It doesnot stripout the "" fully.Could any body have sample
> code for the same so that it will be helfful for my reference?
>
> Ram Laxman


#include <cstdlib>
#include <fstream>
#include <ios>
#include <iomanip>
#include <iostream>
#include <sstream>
#include <string>

int main()
{
std::ifstream ifs("csv.txt");
if(!ifs)
{
std::cerr << "Cannot open input\n";
return EXIT_FAILURE;
}

const std::streamsize width(15);
std::cout << std::left;

std::string line;
while(std::getline(ifs, line))
{
std::string tok1;
std::istringstream iss(line);
while(std::getline(iss, tok1, ','))
{
if(tok1.find('"') != std::string::npos)
{
std::string tok2;
std::istringstream iss(tok1);
while(std::getline(iss, tok2, '"'))
{
if(!tok2.empty())
std::cout << std::setw(width) << tok2;
}
}
else
std::cout << std::setw(width) << tok1;

std::cout << ' ';

}
std::cout << " \n";
}

if(!ifs && !ifs.eof())
std::cerr << "Error reading input\n";

return 0;
}

Input file:

"empno","phonenumber","wardnumber"
12345,2234353,1000202
12326,2243653,1000098



Output:

empno phonenumber wardnumber
12345 2234353 1000202
12326 2243653 1000098



-Mike


 
Reply With Quote
 
Phlip
Guest
Posts: n/a
 
      02-07-2004
Mike Wahler wrote:

> #include <cstdlib>


Hi Mike!

I just wanted to be the first to remind you that the FAQ advises against
doing others' homework - fun though it may be. (Advising the newbie to throw
in a few Design Patterns is better sport, of course...)

--
Phlip
http://www.xpsd.org/cgi-bin/wiki?Tes...UserInterfaces


 
Reply With Quote
 
Jon Bell
Guest
Posts: n/a
 
      02-07-2004
In article <mgaVb.17848$(E-Mail Removed). net>,
Mike Wahler <(E-Mail Removed)> wrote:

[code snipped]

>Input file:
>
>"empno","phonenumber","wardnumber"
>12345,2234353,1000202
>12326,2243653,1000098


Try changing the first line so one of the tokens contains a comma, e.g.

"empno","phone, number","wardnumber"



I started to work on a solution, too, and then I thought about embedded
commas, and went, "uh oh..."

--
Jon Bell <(E-Mail Removed)> Presbyterian College
Dept. of Physics and Computer Science Clinton, South Carolina USA
 
Reply With Quote
 
Mike Wahler
Guest
Posts: n/a
 
      02-07-2004

"Phlip" <(E-Mail Removed)> wrote in message
news:YtaVb.19462$(E-Mail Removed). com...
> Mike Wahler wrote:
>
> > #include <cstdlib>

>
> Hi Mike!
>
> I just wanted to be the first to remind you that the FAQ advises against
> doing others' homework - fun though it may be.


Yes, I realize that.

> (Advising the newbie to throw
> in a few Design Patterns is better sport, of course...)


I very much doubt that the code would be accepted 'as is'
by an instructor -- unless the student can explain it --
in which case he would have actually studied and learned...
Anyway, it seems that OP isn't quite sure whether he's learning
C or C++.

-Mike


 
Reply With Quote
 
Mike Wahler
Guest
Posts: n/a
 
      02-07-2004

"Jon Bell" <(E-Mail Removed)> wrote in message
news:c03c7s$mq0$(E-Mail Removed)...
> In article <mgaVb.17848$(E-Mail Removed). net>,
> Mike Wahler <(E-Mail Removed)> wrote:
>
> [code snipped]
>
> >Input file:
> >
> >"empno","phonenumber","wardnumber"
> >12345,2234353,1000202
> >12326,2243653,1000098

>
> Try changing the first line so one of the tokens contains a comma, e.g.
>
> "empno","phone, number","wardnumber"
>
>
>
> I started to work on a solution, too, and then I thought about embedded
> commas, and went, "uh oh..."


Well, yes I did think about bad input, but thought I'd leave that
to the OP. IOW I gave a very 'literal' answer that only addressed
the exact input cited by the OP.


-Mike


 
Reply With Quote
 
Jon Bell
Guest
Posts: n/a
 
      02-07-2004
In article <vGaVb.17879$(E-Mail Removed) .net>,
Mike Wahler <(E-Mail Removed)> wrote:
>
>Well, yes I did think about bad input, but thought I'd leave that
>to the OP. IOW I gave a very 'literal' answer that only addressed
>the exact input cited by the OP.


It would be interesting to find out if the instructor actually intended
the students to go whole hog and deal with embedded commas, escaped
quotes, etc. If it's an introductory programming course, it's quite
possible they don't need to worry about those details for the purposes of
the assignment.

--
Jon Bell <(E-Mail Removed)> Presbyterian College
Dept. of Physics and Computer Science Clinton, South Carolina USA
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Best way to parse a csv...... a csv that has CRLF in the fields sso Java 20 04-26-2009 11:04 AM
Parse specific text in email body to CSV file dpw.asdf@gmail.com Python 2 03-09-2008 06:26 PM
Writing a nice formatted csv file redcic Python 6 05-02-2007 03:14 PM
How to Parse a CSV formatted text file Ram Laxman C Programming 22 02-11-2004 06:19 PM



Advertisments