DtTokenizer Class Reference

A generic tokenizer class that can be used to read in a stream of data and parse it into tokens. You can specify the separators between each record as well as each field and any escape chars or quote chars. The default is set to be used with csv files.

Public Member Functions
	DtTokenizer ()
	~DtTokenizer ()
void	setFilename (const string &filename)
void	setStream (istream &stream)
void	setRecordSeparator (const string &separator)
void	setFieldSeparators (const string &separators)
void	setQuoteChars (const string &quoteChars)
void	setEscapeChars (const string &escapeChars)
void	setSeparatorData (const string &recordSeparator, const string &fieldSeparators=",", const string &quoteChars="\"", const string &escapeChars="\\")
void	setTrimWhitespace (bool bTrim)
void	setAddEmptyTokens (bool bAdd)
void	setSkipNextRecord (bool bSkip)
bool	nextRecord (vector< string > &record)

Protected Member Functions
virtual bool	isEndOfRecord (const string &str)

Private Member Functions
void	readRecord (string &record)

Private Attributes
string	myRecordSeparator
string	myFieldSeparators
string	myQuoteChars
string	myEscapeChars
bool	myTrimWhitespace
bool	myAddEmptyTokens
bool	mySkipNextRecord
ifstream	myFileStream
istream *	pMyStream

Constructor & Destructor Documentation

DtTokenizer::DtTokenizer

(

)

DtTokenizer::~DtTokenizer

(

)

Member Function Documentation

void DtTokenizer::setFilename

(

const string &

filename

)

sets the file to use as the input stream

void DtTokenizer::setStream

(

istream &

stream

)

sets the input stream to get the data from

void DtTokenizer::setRecordSeparator

(

const string &

separator

)

Functions to set the different kinds of separators and special characters.

void DtTokenizer::setFieldSeparators

(

const string &

separators

)

void DtTokenizer::setQuoteChars

(

const string &

quoteChars

)

void DtTokenizer::setEscapeChars

(

const string &

escapeChars

)

void DtTokenizer::setSeparatorData	(	const string &	recordSeparator,
		const string &	fieldSeparators = ",",
		const string &	quoteChars = "\"",
		const string &	escapeChars = "\\"
	)

void DtTokenizer::setTrimWhitespace

(

bool

bTrim

)

determines if whitespace will be trimmed from a token

void DtTokenizer::setAddEmptyTokens

(

bool

bAdd

)

determines if empty tokens should be added

void DtTokenizer::setSkipNextRecord

(

bool

bSkip

)

setSkipNextLine - especially useful for skipping the first line which could be a header

bool DtTokenizer::nextRecord

(

vector< string > &

record

)

function used to read in and parse the next record into tokens populates a vector of strings representing all of the tokens parsed returns a bool indicating whether tokens are being returned or the stream is still good

virtual bool DtTokenizer::isEndOfRecord ( const string &  str )

protectedvirtual

determines if we are at the end of a record

void DtTokenizer::readRecord ( string &  record )

private

Read chars until end of a record.

Member Data Documentation

string DtTokenizer::myRecordSeparator

private

string DtTokenizer::myFieldSeparators

private

string DtTokenizer::myQuoteChars

private

string DtTokenizer::myEscapeChars

private

bool DtTokenizer::myTrimWhitespace

private

bool DtTokenizer::myAddEmptyTokens

private

bool DtTokenizer::mySkipNextRecord

private

ifstream DtTokenizer::myFileStream

private

istream* DtTokenizer::pMyStream

private

---

The documentation for this class was generated from the following file:

• /data/exports/nrhent7-1/nightly/docs/vrForces.vrf5-2-branch/include/vrfutil/tokenizer.h