QuotationAwareTokenizer

QuotationAwareTokenizer

QuotationAwareTokenizer ()

combineTokens

String combineTokens (String... tokens)

Perform the reverse of tokenizeLine(String).
Given array of tokens, combine them into a single line.

tokenizeLine

String[] tokenizeLine (String line, 
                String delim)

Tokenizes the string, splitting on specified delimiter. Does not split between consecutive, unquoted double-quote marks.

How the tokenizer works:

1. Split the string into "characters" where each "character" is either an escaped character like \" (that is, "\\\"") or a single real character like f (just "f").
2. For each "character"
   1. If it's a space, finish a token unless we're being quoted
   2. If it's a quotation mark, flip the "we're being quoted" bit
   3. Otherwise, add it to the token being built
3. At EOL, we typically haven't added the final token to the (tokens) ArrayList
   1. If the last "character" is an escape character, throw an exception; that's not valid
   2. If we're in the middle of a quotation, throw an exception; that's not valid
   3. Otherwise, add the final token to (tokens)
4. Return a String[] version of (tokens)

tokenizeLine

String[] tokenizeLine (String line)

Tokenizes the string, splitting on spaces. Does not split between consecutive, unquoted double-quote marks.

See also tokenizeLine(String, String)