Ve Parser : A .Net Combinatory Parser

Source code checked in, #94877

Project Collection Service Accounts — Mon, 01 Oct 2012 22:38:35 GMT

Upgrade: New Version of LabDefaultTemplate.xaml. To upgrade your build definitions, please visit the following link: http://go.microsoft.com/fwlink/?LinkId=254563

Source code checked in, #94876

Project Collection Service Accounts — Mon, 01 Oct 2012 22:31:06 GMT

Checked in by server upgrade

Closed Task: Fix the problems in parallel implementation or drop the feature if is not useful [16734]

MeysamNaseri — Wed, 11 Jan 2012 01:42:15 GMT

The parallel execution of some combinators and related structures are not working well, remove them from the main source codes, fix the problems and then brought them back to the main source codes.
Comments:

Closed Task: Improve parse monitoring and debugging [16735]

MeysamNaseri — Wed, 11 Jan 2012 01:42:02 GMT

Lots of things can be done to improve the debugging experience:
1- Implementing DebuggerDisplay for Parser and TokenChecker
2- Implementing DebuggerVisualizer for Parser
3- Providing virtual methods to be called on executation of each parser or token checker
4- The ParserModel should store a better definition for the parser.
Comments:

Closed Issue: Update code structures to comply with MSDN's library design guidelines [16737]

MeysamNaseri — Wed, 11 Jan 2012 01:37:15 GMT

Consider a namespace for all types.
Using public fields is not a good practice, most of fields should be converted into property.
The library forces the AST types to expose fileds by not supporting the properties, change fill methods to support properties as well.
Move each code file into new seperate files inside directories.
Comments:

Updated Wiki: Home

MeysamNaseri — Thu, 01 Dec 2011 13:15:33 GMT

Project Description
Ve Parser is an implementation of Combinatory Parser concept in .Net environment using C# language. It allows to very easily write parsers for your language by writing its grammar using your favorite .Net language. Not only it does parse the input, but also it generates a decent output.

Update

Just released the new version, in order to have some improvements I forced to change some names or functionalities that may affect your parsers(if you have written using version 1.0). Lets me know if you have any problem in migrating your existing parser to use this new version.

Nuget

To add a reference to Ve Parser in your project:
PM> Install-Package veparser

Introuduction

Like other combinatory parsers, we used functional structures available in .Net and specially High Order Functions in order to design Ve Parser; however, the functions, which are used in the Ve parser, have a different structure in comparison with others; and consequently, a number of remarkable merits are provided. For instance, this parser is by far the simplest and easiest to understand among other combinatory parsers and it has the possibility of parallel processing for choice combinators. On the other hand, the usage of object-oriented approaches will lead to avoid of restrictions of pure functional programming. Moreover, Ve Parser has the ability of generating Abstract Syntax Tree in an integrated way and the output of parsing will be a concrete conceptual object.

What is a Combinatory Parser How Ve Parser works?

The only jargon I used in the project and even in the source codes are 'Combinator Parser' and 'Lexer'. I dont want to make it more complex by explaining what a combinator parser is, but if you are curious you can see its page in wiki pedia. You see, its a little complicated subject but Ve Parser is much simpler that the name 'combinatory parser' suggests. So instead of describing what is a Combinatory Parser I will describe how Ve Parser works:
Ve Parser is defined only a delegate type named Parser which has no input and has a boolean output. Parser is a delegate type, so variables of type Parser are functions that can be executed, when a Parser executes it returns a boolean value. The result value from a Parser specifies if it was successful for or not.
Then there is ParserBase class which is an abstract class you have to derive from. When you derive from it you have to implement the abstract method -GetRootParser-. To implement this GetRootParser the BaseParser class provided you with a set of functions which all of them get inputs of type Parser and produce a Parser as result, as long as all of them have the output type of Parser and all of them have input of type Parser they can used combined toghether to form a more mature parser from a more basic parser and finaly make a root parser. For example it would look like 'seq(oneOrMore(....),any(...),seq(any(zeroOrMore(...))))'. Here the seq function is being called by passing the result of oneOrMore parser, any and seq. Then seq parser returns a parser as result.

Features

Simple to use. Simple to learn. Although hard to understand how it works internally.
No dependency on any grammar language, only pure C# code can be used to design and implement a parser.
Rich grammar authoring preset. The ParserBase is shiped with a whole bunch of useful parser combinators like : seq, any, zeroOrMore, zeroOrOne, oneOrMore, zeroOrMoreAny, zeroOeMoreSeq, oneOrMoreAny, oneOrMoreSeq and deleimitedList and mixture.
Producing output result from the parsed content. (A.k.a Abstract Syntax Tree generation).
Extensible. Although there are lots of useful combinators you can wrote your own combinators to the library. I am looking to hearing from you about any new combinator that you may suggest.
Probably fast, if you wrote a neat grammar it should be fast. I need to evaluate performance more percisely to claim about it speed.
Grammar debugging. Yes that's it, you can monitor the executation of the parser, and also you can put breakpoints in your parser code to see how it is performing. (_Although this features require some improvements, so the next version would be even better).
Shiped with a default Lexer.
Lightweight library, with no dependency to any other library. Just you need .Net 4 to use it. And itself is only a couple of essential classes and some other helper classes.

Sample Code

Probably a good way to describe the library is a sample code of how a Parser using Ve Parser would look like:
Here is a small part of a sample CSV parser that parse a simple CSV content and result a list of rows which each row is list of values:
Note: This code is based on the latest source codes (downloadable through Source Code tab) and it would not work with the current released version.

    public class CSVParser : BaseParser<char>
    {
        public const char comma = ',';
        public const char newline = '\n';

        protected override Parser GetRootParser()
        {
            var seperatorParser = toParser(ch => ch == comma);
            var lineBreakParser = seq(toParser(ch => ch == '\r'), toParser(ch => ch == '\n'));
            var itemParser = createNewText(zeroOrMore(appendText(keep(toParser(ch => ch != comma && ch != '\n' && char.IsLetterOrDigit(ch))))));
            var lineParser = createNewList(deleimitedList(seperatorParser, add(itemParser)));
            var fileParser = createNewList(deleimitedList(lineBreakParser, add(lineParser)));
            return seq(fileParser, endOfFile());
        }
    }

And below is the code which describes how the above parser will be used:

[TestMethod]
public void CSVParserTest()
{
    var csvContent = @"column1,column2
19,20";
    var csvParser = new CSVParser();
    dynamic result = csvParser.Parse(csvContent.ToCharArray());
    Assert.IsNotNull(result);
    Assert.AreEqual(result.Count, 2);
    foreach (var row in result)
    {
        Assert.AreEqual(row.Count, 2);
    }
    Assert.AreEqual(result[0][0], "column1");
    Assert.AreEqual(result[0][1], "column2");
    Assert.AreEqual(result[1][0], "19");
    Assert.AreEqual(result[1][1], "20");
}

Created Release: Ve Parser 1.1 (Dec 01, 2011)

meysamnaseri — Thu, 01 Dec 2011 13:11:38 GMT

In this release I added/improved output generation functionalities, most notably the ability to generate output dynamically(using DLR). Also there are lots of changes to some basic combinators which if you have used them your parser will not work/compile with this new version.

Released: Ve Parser 1.1 (Dec 01, 2011)

Thu, 01 Dec 2011 13:11:38 GMT

Updated Wiki: Home

MeysamNaseri — Thu, 17 Nov 2011 04:46:48 GMT

Nuget

To add a reference to Ve Parser in your project:
PM> Install-Package veparser

Introuduction

What is a Combinatory Parser How Ve Parser works?

Features

Simple to use. Simple to learn. Although hard to understand how it works internally.
No dependency on any grammar language, only pure C# code can be used to design and implement a parser.
Rich grammar authoring preset. The ParserBase is shiped with a whole bunch of useful parser combinators like : seq, any, zeroOrMore, zeroOrOne, oneOrMore, zeroOrMoreAny, zeroOeMoreSeq, oneOrMoreAny, oneOrMoreSeq and deleimitedList and mixture.
Producing output result from the parsed content. (A.k.a Abstract Syntax Tree generation).
Extensible. Although there are lots of useful combinators you can wrote your own combinators to the library. I am looking to hearing from you about any new combinator that you may suggest.
Probably fast, if you wrote a neat grammar it should be fast. I need to evaluate performance more percisely to claim about it speed.
Grammar debugging. Yes that's it, you can monitor the executation of the parser, and also you can put breakpoints in your parser code to see how it is performing. (_Although this features require some improvements, so the next version would be even better).
Shiped with a default Lexer.
Lightweight library, with no dependency to any other library. Just you need .Net 4 to use it. And itself is only a couple of essential classes and some other helper classes.

Sample Code

    public class CSVParser : BaseParser<char>
    {
        public const char comma = ',';
        public const char newline = '\n';

        protected override Parser GetRootParser()
        {
            var seperatorParser = toParser(ch => ch == comma);
            var lineBreakParser = seq(toParser(ch => ch == '\r'), toParser(ch => ch == '\n'));
            var itemParser = createNewText(zeroOrMore(appendText(keep(toParser(ch => ch != comma && ch != '\n' && char.IsLetterOrDigit(ch))))));
            var lineParser = createNewList(deleimitedList(seperatorParser, add(itemParser)));
            var fileParser = createNewList(deleimitedList(lineBreakParser, add(lineParser)));
            return seq(fileParser, endOfFile());
        }
    }

And below is the code which describes how the above parser will be used:

[TestMethod]
public void CSVParserTest()
{
    var csvContent = @"column1,column2
19,20";
    var csvParser = new CSVParser();
    dynamic result = csvParser.Parse(csvContent.ToCharArray());
    Assert.IsNotNull(result);
    Assert.AreEqual(result.Count, 2);
    foreach (var row in result)
    {
        Assert.AreEqual(row.Count, 2);
    }
    Assert.AreEqual(result[0][0], "column1");
    Assert.AreEqual(result[0][1], "column2");
    Assert.AreEqual(result[1][0], "19");
    Assert.AreEqual(result[1][1], "20");
}

Updated Wiki: Home

MeysamNaseri — Mon, 14 Nov 2011 01:06:44 GMT

Nuget

To add a reference to Ve Parser in your project:
PM> Install-Package veparser

Introuduction

What is a Combinatory Parser How Ve Parser works?

Features

Simple to use. Simple to learn. Although hard to understand how it works internally.
No dependency on any grammar language, only pure C# code can be used to design and implement a parser.
Rich grammar authoring preset. The ParserBase is shiped with a whole bunch of useful parser combinators like : seq, any, zeroOrMore, zeroOrOne, oneOrMore, zeroOrMoreAny, zeroOeMoreSeq, oneOrMoreAny, oneOrMoreSeq and deleimitedList and mixture.
Producing output result from the parsed content. (A.k.a Abstract Syntax Tree generation).
Extensible. Although there are lots of useful combinators you can wrote your own combinators to the library. I am looking to hearing from you about any new combinator that you may suggest.
Probably fast, if you wrote a neat grammar it should be fast. I need to evaluate performance more percisely to claim about it speed.
Grammar debugging. Yes that's it, you can monitor the executation of the parser, and also you can put breakpoints in your parser code to see how it is performing. (_Although this features require some improvements, so the next version would be even better).
Shiped with a default Lexer.
Lightweight library, with no dependency to any other library. Just you need .Net 4 to use it. And itself is only a couple of essential classes and some other helper classes.

Sample Code

Probably a good way to describe the library is a sample code of how a Parser using Ve Parser would look like:
Here is a small part of a sample parser for C# language (to make it more understandable and to make it smaller only a part of the whole parser is included):

    public class CSVParser : BaseParser<char>
    {
        public const char comma = ',';
        public const char newline = '\n';

        protected override Parser GetRootParser()
        {
            var seperatorParser = toParser(ch => ch == comma);
            var lineBreakParser = seq(toParser(ch => ch == '\r'), toParser(ch => ch == '\n'));
            var itemParser = createNewText(zeroOrMore(appendText(keep(toParser(ch => ch != comma && ch != '\n' && char.IsLetterOrDigit(ch))))));
            var lineParser = createNewList(deleimitedList(seperatorParser, add(itemParser)));
            var fileParser = createNewList(deleimitedList(lineBreakParser, add(lineParser)));
            return seq(fileParser, endOfFile());
        }
    }

Source code checked in, #82830

MeysamNaseri — Mon, 14 Nov 2011 01:02:30 GMT

Better output generatoin functionalities added, dynamic programming features, LOTS OF CHANGES INTO FUNCTION NAMES

Source code checked in, #81837

MeysamNaseri — Thu, 20 Oct 2011 14:27:49 GMT

Solution structure refinements, some , a little xml documentation added.

Updated Wiki: Documentation

MeysamNaseri — Thu, 20 Oct 2011 14:14:23 GMT

Geting Started

Although the task of writing a parser is not so easy and requires you to know some parsing concepts, this parser is realy easy to use. We designed the library to be as simple as is possible and avoided using jargon. So you only need to design your language or analyze how your data structure can be parsed using the available functionalities in veparser.

The steps to write a parser using Ve Parser

Add Ve Parser to your project:
1. Download and reference the veparser.dll in your project.
2. or simply using NuGet you only need to type Install-Package veparser
Derive a class from TokenParser.
Define your parser by implementing the abstract method GetRootParser.
Tokenize your input text calling Lexer.Parse
Call the Parse method of your parser calss.

How Tos

How to write a parser (comming soon..)
How to write my own Lexer/Tokenizer (coming soon...)
How to monitor parser execution (coming soon...)
How to debug the parser I wrote (coming soon...)

FAQ

Is Ve Parser only for parsing programming languages?

Ve Parser is very tricky in its design and in its usage. You can use it for parsing a computer language, a data structure like CSV file, an xml data. So its up to you how to use it. In fact it consumes an IEnumerator of any arbitary data type(rather than char or string), so you can parse a list of data objects to see if they follow a special pattern or not.

What kind of grammar language do I have to wrote in order to utilise Ve Parser?

Ve Parser is a Combinatory Parser writen in C#, so the only thing you need is a .Net language. So you can write your grammar directly in your C# code and actualy your grammar is your parser and your parser is your grammar. Thats it, the grammar and the parser are the same thing.

Whats are Ve Parser requirements?

Due to implementatoin dependencies it is available only for .Net 4.0.

Reference

For now I am writing a brief description of the major combinators, to start the process of documentation. So be aware that this reference documentation is incomplete and I am looking after generating a comprehensive documentation which includes the cod XML documentations.
Lets start with some basic concepts:

Parser

Parser is a delegate defined as the very fundamental type for functions which will represent a parser or generator. This delegate signature is as follow:

public delegate bool Parser();

The delegate simply says that a Parser is a function that has no argument and return a true or false value. If the return value is true it means that the parser did it job successfully and if the return value is false it means the Parser failed to do its job.
I need to clarify here that the Parser finally will work on the input stream but for some reason which requires a separate page I did not set the input stream as the Parser's arguments. The toParser function will create a Parser from a TokenChecker but I tried to hide this function by layers of abstraction so from the perspective of end user, only we have a Parser concept.

Combinator

Ve Parser is a combinatory parser, it works based on combination of parsers. You create more complex parsers by combining the more simpler parsers. You may begin with a simple parser that only expects the following input token(character or word) to be a special value and then create parsers which expect a sequence of some other things to appear in the input stream.

Lets follow the documentation with combinators, I am going to put them in the order of importance.
Parser

any

The input and output : This combinator receives a set of parsers in input and will returns a parser (Parser[] --> Parser).
When wins : The returned parser will success if it reaches a successful parser.
Purpose : To follow different options to parse the input, so it will allow the parser to find a expected input from a range of possibilities and follow the path for that kind of expected input. For example when parsing class members for C# language you may expect a field definition or you may expect a property definition, but in current position both are possible.

seq

The input and output : This combinator receives a set of parsers in input and will returns a parser (Parser[] --> Parser).
When wins : The returned parser will success if all of the passed parsers run successfully one after one.
Purpose: So it will check to see if a sequence of expectations are satisfied by input.

Source code checked in, #81635

MeysamNaseri — Fri, 14 Oct 2011 23:58:43 GMT

Removed some extra irrelevant codes

Created Release: Ve Parser 1.0 (Oct 14, 2011)

meysamnaseri — Fri, 14 Oct 2011 16:38:52 GMT

This is the first stable version of the library available for use. Download and add reference to the library and follow the documentation to create your parser using Ve Parser.

Released: Ve Parser 1.0 (Oct 14, 2011)

Fri, 14 Oct 2011 16:38:52 GMT

This is the first stable version of the library available for use. Download and add reference to the library and follow the documentation to create your parser using Ve Parser.

Closed Feature: Add not/avoid parser [16736]

MeysamNaseri — Fri, 14 Oct 2011 16:34:34 GMT

Add a new parser combinator which takes a parser and produces a parser which calls the original parser but revereses the result. so if the original parser returns true it will return false and if the original parser return true it returns false.
Comments: Resolved with changeset 81612.

Commented Task: Improve parse monitoring and debugging [16735]

MeysamNaseri — Fri, 14 Oct 2011 16:34:33 GMT

Source code checked in, #81612

MeysamNaseri — Fri, 14 Oct 2011 16:34:29 GMT

Project structure updated-Some other changes to make library more stable

Updated Wiki: Home

MeysamNaseri — Fri, 14 Oct 2011 16:24:21 GMT

Nuget

To add a reference to Ve Parser in your project:
PM> Install-Package veparser

Introuduction

What is a Combinatory Parser How Ve Parser works?

Features

Simple to use. Simple to learn. Although hard to understand how it works internally.
No dependency on any grammar language, only pure C# code can be used to design and implement a parser.
Rich grammar authoring preset. The ParserBase is shiped with a whole bunch of useful parser combinators like : seq, any, zeroOrMore, zeroOrOne, oneOrMore, zeroOrMoreAny, zeroOeMoreSeq, oneOrMoreAny, oneOrMoreSeq and deleimitedList and mixture.
Producing output result from the parsed content. (A.k.a Abstract Syntax Tree generation).
Extensible. Although there are lots of useful combinators you can wrote your own combinators to the library. I am looking to hearing from you about any new combinator that you may suggest.
Probably fast, if you wrote a neat grammar it should be fast. I need to evaluate performance more percisely to claim about it speed.
Grammar debugging. Yes that's it, you can monitor the executation of the parser, and also you can put breakpoints in your parser code to see how it is performing. (_Although this features require some improvements, so the next version would be even better).
Shiped with a default Lexer.
Lightweight library, with no dependency to any other library. Just you need .Net 4 to use it. And itself is only a couple of essential classes and some other helper classes.

Sample Code

// Define some keywords using the Keyword_of function and assign it to Parser deleagates
Parser @public = Keyword_of("public");
Parser @private = Keyword_of("private");
Parser @protected = Keyword_of("protected");
Parser @get = Keyword_of("get");
Parser @set = Keyword_of("set");
Parser @enum = Keyword_of("enum");
Parser @class = Keyword_of("class");
Parser @internal = Keyword_of("internal");
Parser @using = Keyword_of("using");
Parser @namespace = Keyword_of("namspace");

// Combine some basic parsers using any and zeroOrOne combinators and create a more high level parser
Parser member_access_level = zeroOrOne(any(
set_choice("access_level", "public", @public),
set_choice("access_level", "private", @private),
set_choice("access_level", "protected", @protected),
set_choice("access_level", "internal", @internal),
set_choice("access_level", "protectedinternal", seq(@protected, @internal))));
// Combine parsers using seq function
Parser property_get = seq(@get, Symbol_of("{"), zeroOrMore(statement), Symbol_of("}"));
Parser property_set = seq(@set, Symbol_of("{"), zeroOrMore(statement), Symbol_of("}"));