QuotationAwareTokenizer

QuotationAwareTokenizer

public QuotationAwareTokenizer ()

combineTokens

public static String combineTokens (String... tokens)

Perform the reverse of tokenizeLine(String).
Given array of tokens, combine them into a single line.

tokenizeLine

public static String[] tokenizeLine (String line, 
                String delim)

tokenizeLine

public static String[] tokenizeLine (String line)

Tokenizes the string, splitting on spaces. Does not split between consecutive, unquoted double-quote marks.

See also tokenizeLine(String, String)

tokenizeLine

public static String[] tokenizeLine (String line, 
                boolean logging)

Tokenizes the string, splitting on spaces. Does not split between consecutive, unquoted double-quote marks.

See also tokenizeLine(String, String)

tokenizeLine

public static String[] tokenizeLine (String line, 
                String delim, 
                boolean logging)

Tokenizes the string, splitting on specified delimiter. Does not split between consecutive, unquoted double-quote marks.

How the tokenizer works:

1. Split the string into "characters" where each "character" is either an escaped character like \" (that is, "\\\"") or a single real character like f (just "f").
2. For each "character"
   1. If it's a space, finish a token unless we're being quoted
   2. If it's a quotation mark, flip the "we're being quoted" bit
   3. Otherwise, add it to the token being built
3. At EOL, we typically haven't added the final token to the (tokens) ERROR(/ArrayList)
   1. If the last "character" is an escape character, throw an exception; that's not valid
   2. If we're in the middle of a quotation, throw an exception; that's not valid
   3. Otherwise, add the final token to (tokens)
4. Return a String[] version of (tokens)