Back to General and Gameplay Programming

Tokenizing Strings

General and Gameplay Programming Programming

Started by Xetrov June 30, 2001 10:45 AM

3 comments, last by Xetrov 23 years, 7 months ago

Xetrov

Author

116

June 30, 2001 10:45 AM

I need to tokenize strings! I''ve used strtok() but it crashes when I tell it to look for the second token.... This is very annoying. Are there any alternatives, or will I have to write my own function?

a38750

164

June 30, 2001 12:44 PM

If you''re making a script parser, I''d write your own, you''ll have more control over how it is tokenized. eg. operators like +-*/ should be separated even if there isn''t a space between the variable.

If you''re just trying to extract data from a string separated by spaces, then there is some function which is like the reverse of sprintf, but I can''t remember what it''s called. sorry, that''s not very helpful, but what the hell

Frank

Anonymous

June 30, 2001 01:41 PM

Strtok() sucks ass because it modifies/destroys your source string.

Keep that in mind. It screwed me up big time when I wrote a parser for a little synth I tried to make (using text files for input). Anyway, I don''t remember exactly what happened, but I ended up having to make a copy of my source string, and passing that to strtok().

freshya

122

June 30, 2001 08:04 PM

Strtok is very useful, but, as the AP said, you will need a copy of the source string, otherwise it gets mangled. Notice the "char seps[]", in which you can place anything that you want, such as ''+'' or ''-'' or '' '' etc etc etc. The following code is taken from my console. It isn''t the most versatile, being designed specifically for my needs, but it works a treat.

          // make a copy of the input string	char stream[LINE_LENGTH];	sprintf ( stream, input );	// establish the strings we will need	char seps[] = " ,\t\n";				// the valid word seperators   char *command;							// the first token ( ie. SET, or GIVE )	char *parm1;							// the first argument ( ie. HEALTH or ROCKETLAUNCHER )	char *parm2;							// the second argument ( ie. 50, 70 )	char *parm3;							// it may be used someday...   /* Establish string and get the tokens: */   command = strtok( stream, seps  );   parm1 = strtok( NULL, seps );	parm2 = strtok( NULL, seps );	parm3 = strtok( NULL, seps );	// make sure they typed something:	if ( command==NULL ) return 0;

I think the formatting may get screwed around with the comments, but I''m not sure...

Anyway, cheers

TrIaD

122

June 30, 2001 08:34 PM

If you need to write your own func, you could do it fairly easily, especially if you can hard-code the seperator...

something I just made up... not tested

    void strtok2(char* instr, char * outstr, int& start){    instr += start;    while(*instr != ' ')    {        *(outstr++) = *(instr++);        start++;    }    *outstr = 0;   start++; //skip over seperator next time    }

...instr is the string to tokenize, outstr is a buffer for the token, start is where to start at (init any var to 0 and keep passing the same var w/o modifying it's value)...

edit: btw, if you need more seps, just add " || *instr = ','", etc to the while..

--Tr][aD--

Edited by - TrIaD on June 30, 2001 10:09:03 PM

--Tr][aD--

Tokenizing Strings

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Tokenizing Strings

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines