Skip to content

ali33/simple-tokenize-js

Repository files navigation

Simple tokenizer

Accepts a subject string and an array of seperators Returns an array of token strings

var str = 'this is text. This is next text <a href="proxy.php?url=https%3A%2F%2Fgoogle.com">click here to goto google.com</a>';
var seps = [' ', '.', '"', '<', '>','/','=','\'','\n'];
//result: ["this", " ", "is", " ", "text", ".", "", " ", "This", " ", "is", " ", 
//        "next", " ", "text", " ", "<", "a", " ", "href", "=", "", """, 
//        "https:", "/", "/", "google", ".", "com", """, ">", "click", " ", "here", " ",
//        "to", " ", "goto", " ", "google", ".", "com", "<", "/", "a", ">", ""]

About

Simple tokenizer for javascript

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors