DtTokenizer Class Reference

A generic tokenizer class that can be used to read in a stream of data and parse it into tokens. More...

Public Member Functions
	DtTokenizer ()
	~DtTokenizer ()
void	setFilename (const string &filename)
	sets the file to use as the input stream
void	setStream (istream &stream)
	sets the input stream to get the data from
void	setRecordSeparator (const string &separator)
	Functions to set the different kinds of separators and special characters.
void	setFieldSeparators (const string &separators)
void	setQuoteChars (const string &quoteChars)
void	setEscapeChars (const string &escapeChars)
void	setSeparatorData (const string &recordSeparator, const string &fieldSeparators=",", const string &quoteChars="\"", const string &escapeChars="\\")
void	setTrimWhitespace (bool bTrim)
	determines if whitespace will be trimmed from a token
void	setAddEmptyTokens (bool bAdd)
	determines if empty tokens should be added
void	setSkipNextRecord (bool bSkip)
	setSkipNextLine - especially useful for skipping the first line which could be a header
bool	nextRecord (vector< string > &record)
	function used to read in and parse the next record into tokens populates a vector of strings representing all of the tokens parsed returns a bool indicating whether tokens are being returned or the stream is still good

Protected Member Functions
virtual bool	isEndOfRecord (const string &str)
	determines if we are at the end of a record

Private Member Functions
void	readRecord (string &record)
	Read chars until end of a record.

Private Attributes
string	myRecordSeparator
string	myFieldSeparators
string	myQuoteChars
string	myEscapeChars
bool	myTrimWhitespace
bool	myAddEmptyTokens
bool	mySkipNextRecord
ifstream	myFileStream
istream *	pMyStream

Detailed Description

A generic tokenizer class that can be used to read in a stream of data and parse it into tokens.

You can specify the separators between each record as well as each field and any escape chars or quote chars. The default is set to be used with csv files.

Constructor & Destructor Documentation

DtTokenizer::DtTokenizer

(

)

DtTokenizer::~DtTokenizer

(

)

Member Function Documentation

void DtTokenizer::setFilename

(

const string &

filename

)

sets the file to use as the input stream

void DtTokenizer::setStream

(

istream &

stream

)

sets the input stream to get the data from

void DtTokenizer::setRecordSeparator

(

const string &

separator

)

Functions to set the different kinds of separators and special characters.

void DtTokenizer::setFieldSeparators

(

const string &

separators

)

void DtTokenizer::setQuoteChars

(

const string &

quoteChars

)

void DtTokenizer::setEscapeChars

(

const string &

escapeChars

)

void DtTokenizer::setSeparatorData	(	const string &	recordSeparator,
		const string &	fieldSeparators = ",",
		const string &	quoteChars = "\"",
		const string &	escapeChars = "\\"
	)

void DtTokenizer::setTrimWhitespace

(

bool

bTrim

)

determines if whitespace will be trimmed from a token

void DtTokenizer::setAddEmptyTokens

(

bool

bAdd

)

determines if empty tokens should be added

void DtTokenizer::setSkipNextRecord

(

bool

bSkip

)

setSkipNextLine - especially useful for skipping the first line which could be a header

bool DtTokenizer::nextRecord

(

vector< string > &

record

)

function used to read in and parse the next record into tokens populates a vector of strings representing all of the tokens parsed returns a bool indicating whether tokens are being returned or the stream is still good

virtual bool DtTokenizer::isEndOfRecord ( const string &  str )

protectedvirtual

determines if we are at the end of a record

void DtTokenizer::readRecord ( string &  record )

private

Read chars until end of a record.

Member Data Documentation

string DtTokenizer::myRecordSeparator

private

string DtTokenizer::myFieldSeparators

private

string DtTokenizer::myQuoteChars

private

string DtTokenizer::myEscapeChars

private

bool DtTokenizer::myTrimWhitespace

private

bool DtTokenizer::myAddEmptyTokens

private

bool DtTokenizer::mySkipNextRecord

private

ifstream DtTokenizer::myFileStream

private

istream* DtTokenizer::pMyStream

private

---

The documentation for this class was generated from the following file:

• /exports/nfedora7-1/nightly/docs/vrforcesDevelopment.vrf4-6-branch/include/vrfutil/tokenizer.h