If you just wish to add a few operators to the default Pliant syntax to be able to write your programs in a more convenient way, you may use the 'operator' function defined in the extend.pli module to do it very simply.

It's prototype is:

function operator name priority nb_before nb_after
  arg Int priority nb_before nb_after

Operators are handled from the highest priority first down to the lowest priority. At equal priority, the left one is handled first. You'll find the priority of the predefined operators in the presentation of the default Pliant syntax

The 'nb_before' and 'nb_after' parameters specify how many tokens the operator is expected to find before it, and after it. Most operators are binary operators that expect one token before and one after.

You can find an example defining the operator '.and.' which is the bit by bit and operator in the int.pli module.
It is very simple since it's only one line:

operator '.and.' 260h 1 1

The remaining of this document may be hard to read, so if you wish to customize very deeply the Pliant syntax and get troubles due to the poor quality of this document, please email your questions to Hubert Tonneau; i'll be happy to help you and improve this document in the same time provided you agree to send me the source code of your module at the end and allow me to take it as an example of deep extension.

A parser filter is responsible of scanning the source code in order to detect a given kind of object and build the corresponding object.

The prototype of a parser filter function is:

function my_function context line parameter
  arg ParserContext context ; arg Str line ; arg address parameter

'context' is the data structure that stores the current state of the parsing process.
'line' parameter is a string that maps the remaining characters on the current line being parsed.
'parameter' is the address stored in the 'parameter' field of the ParserFilter object which introduces the new operator

In order to record a parser filter function, you have to create a ParserFilter object.

So the whole code for recording the parser filter function will look like:

function my_function context line parameter
  var Pointer:MyType my_parameter :> parameter map MyType
var MyType my_parameter := ...
var ParserFilter my_filter
my_filter function :> fun my_function ParserContext Str address
my_filter parameter := addressof my_parameter
current_module define "pliant parser basic types" false addressof:my_filter

You can find an example of a new filter in int.pli where the function 'parse_bin' is a filter that recognize integers in binary notation such as  01101110b .

It is still possible to get deeper in the Pliant parsing machinery in order to make more important changes. The rest of this chapter try to give you some guidelines on how it works. Anyway you should first read the end of this chapter, then study the details of the engine.c source code.

The general task of the parser is to turn the source code to a tree of expressions, then call the compiler to convert that tree to an executable.

The Pliant parser is made of two stages:

The main function of the Pliant engine is 'compile_text'.
In input, it takes: 'compile_text' will first create a ParserContext temporary structure that will store the status of the parsing process until the full text as been parsed, then will be dropped.
As you can see in the C source code, 'compile_text' mainly loops on 'ParserContext_parse_one_token' until the end of the source code is reached.
'ParserContext_parse_one_token' is responsible for parsing a single token of the source code. In order to do so, it will call in order each of the filters until one moves the parsing cursor forward which means that is successfull parsed a token.

Calling the parser filters in the right order is very important, it is the main job of 'ParserContext_parse_one_token' that uses a two stages scheme:

Nb: We could do all this with a single stage scheme since the many definitions of a single symbol in a dictionary are fully ordered, but then it will be hard to insert new symbols properly since an overdefinition of an existing symbol is always inserted in the dictionary before or behind all other definitions of that symbol.

already covered

There are also pseudo parser filters that dont really scan the source code for tokens but rather check if the parsing status has riched some point where it is time to perform other operations. As an example, 'parser_filter_execute' checks if we have parsed a full expression and if so asks to compile and execute it trough calling 'Expression_execute'.

Another pseudo parser filter is 'parser_filter_handle_pendings'. At some point of the source code, a true parser filter may recognize a pattern that requires to perform some extra operations when another point further in the source code will be riched. In order to do so, the true parser filter will call 'ParserContext_add_pending' that will record the function to be called and when it must be called, then 'parser_filter_handle_pendings' will do the call at the right time. The best example of it is in 'parser_filter_newline' that recognize indentation. When the next line is indented further than the current one, it records where the '{' sign must be inserted and where the corresponding '}' sign must be inserted.

When a token is recognized by a true parser filter, it calls 'ParserContext_add_token' to record the recognized token. The 'ParserContext_add_token' function builds an atomic expression than contains the recognized token, stores it as another sun of the current expression pointed in the ParserContext structure and return the just built expression.

Now if the recognized token is an operator, the true parser filter is responsible for attaching a 'parseroperator' structure to the 'op' field of the newly created expression, that will define how priority the operator is (at folding time the operators are handled most priority first, and left one first in case of equal priority), which function is responsible for folding and an additional argument that will be passed to that function.'parser_operator_t1' is an example or a folding function which will fold a ParserFilterOperatorT1 operator.

The obvious way of folding is to call 'fold_arguments' function. (Should you get so deep in extending or customising Pliant syntax that the two kind of predefined operators below dont match your needs, then you still can call 'fold_arguments' function in your folding function written in Pliant since it is also exported from C to Pliant in startup.c

There are two basic kind of predefined operators:

Just before compiling and executing a parsed expression through calling 'Expression_execute', the parser engine calls 'Expression_precompile_rewrite' which is defined in expression.c. This function will call itself recursively for each sub expressions.

furthermore, for each expression, it will scan the main directory for al 'pliant precompile rewrite' symbols that are visible in the current module. All the associated objects are expected to be functions that rewrite an expression.
An example of such a function is 'parser_rewrite_bloc' that removes the bloc stage when a bloc contains a single instruction.

From a more general point of view the Pliant parsing process is three stages: