Tokenizes the string, splitting on spaces. Does not split between consecutive, unquoted
How the tokenizer works:
- Split the string into "characters" where each "character" is either an escaped
character like \" (that is, "\\\"") or a single real character like f (just "f").
- For each "character"
- If it's a space, finish a token unless we're being quoted
- If it's a quotation mark, flip the "we're being quoted" bit
- Otherwise, add it to the token being built
- At EOL, we typically haven't added the final token to the (tokens)
- If the last "character" is an escape character, throw an exception; that's not
- If we're in the middle of a quotation, throw an exception; that's not valid
- Otherwise, add the final token to (tokens)
- Return a String version of (tokens)
String to be tokenized
|A tokenized version of the string
|if the line cannot be parsed