Header for tokenizer.c. More...

#include <stdio.h>
#include "typedefs.h"

Data Structures
struct	Tokenizer

Functions
Tokenizer *	Tokenizer_new (int maxtok)
	Tokenizer constructor.

void	Tokenizer_clear (Tokenizer *self)

void	Tokenizer_free (Tokenizer *t)
	Tokenizer destructor.

int	Tokenizer_split (Tokenizer t, char buff, const char *sep)
	Turn string "buff" into an array of tokens, assuming that tokens in the input string may be separated by any of the characters in string "sep". More...

char *	Tokenizer_token (Tokenizer *t, int index)
	Return pointer to token with given index.

int	Tokenizer_ntokens (Tokenizer *t)
	Return number of tokens.

int	Tokenizer_strip (Tokenizer t, const char extraneous)
	Strip extraneous chars (those list in "extraneous") from both ends of each token. More...

int	Tokenizer_find (Tokenizer t, const char s)
	Search for string s among tokens. More...

void	Tokenizer_print (const Tokenizer tkz, FILE ofp)
	Print Tokenizer object.

void	Tokenizer_printSummary (const Tokenizer tkz, FILE ofp)
	Print a summary of the information in a Tokenizer.

Detailed Description

Header for tokenizer.c.

Author: Alan R. Rogers

Copyright: Copyright (c) 2014, Alan R. Rogers roger.nosp@m.s@an.nosp@m.thro..nosp@m.utah.nosp@m..edu. This file is released under the Internet Systems Consortium License, which can be found in file "LICENSE".

Function Documentation

◆ Tokenizer_find()

int Tokenizer_find	(	Tokenizer *	t,
		const char *	s
	)

Search for string s among tokens.

On success, return index of token. On failure, return the current number of tokens. After each call, the returned value should be compared with that of Tokenizer_ntokens.

◆ Tokenizer_split()

int Tokenizer_split	(	Tokenizer *	self,
		char *	buff,
		const char *	sep
	)

Turn string "buff" into an array of tokens, assuming that tokens in the input string may be separated by any of the characters in string "sep".

Supress empty tokens. Return the number of tokens.

References strCountSetChunks().

Referenced by DAFReader_next(), and RAFReader_next().

◆ Tokenizer_strip()

int Tokenizer_strip	(	Tokenizer *	t,
		const char *	extraneous
	)

Strip extraneous chars (those list in "extraneous") from both ends of each token.

If the result is an empty string, this token is removed from the list. The function returns the number of tokens.