Delimiters, Splits, Tokens, and Meta- Characters

So we have this text file that is full of World information on 10 players.

Each Player has the following information;

• World Name
• Player Name
• Player Health
• Player Strength
• Player Intelligence
• Player Location   -   each Player has a starting location
• each location has 2 coordinates
• X co-ordinate
• Y co-ordinate
• Player Inventory   -   each Player has 10 Items in their Inventory
• each Item has 5 bits of Information 
• Item Name
• Item ID
• Item Weight
• Item Value
• Item Level 

Now this information is all held in this World text file, but how do we read it effectively, it just seems to be a random wall of text.

We use a Delimiter.

In this case the delimiter is set as   ;

What that means is when we scan the text file each character preceded by   ;   will be recorded as a string.

Each of these strings is called a token.

Each token can be broken down further into smaller tokens using the split function.

In this case we have

  First split     :
  Second split     @
  Third split     *
  Fourth split     #
  Fifth split     ?

The issue we come across is that some characters already have assigned functions; these are called Meta- Characters, and using them as delimiters and splits will cause hang and crash issues.

Meta-Characters      .  ^  $  |  *  +  ?  (  )  [  {  \  

If you precede Meta-Characters with two forward slashes \\ it will turn them into a literal character.

eg.    scan.useDelimiter(”\\? “)     or    String[] StringName = Token.split(”\\* “)

Original Post Jun 30th 2012