Google Git
Sign in
qemu/edk2/refs/heads/UDK2018_TreeCheckIn/./BaseTools/Source/C/VfrCompile/Pccts/CHANGES_SUMMARY.txt
blob: 91defae169c4e7f5c3f059ae52130b94715a9e77 [file]
======================================================================
CHANGES_SUMMARY.TXT
A QUICK overview of changes from 1.33 in reverse order
A summary of additions rather than bug fixes and minor code changes.
Numbers refer to items in CHANGES_FROM_133*.TXT
which may contain additional information.
DISCLAIMER
The software and these notes are provided "as is". They may include
typographical or technical errors and their authors disclaims all
liability of any kind or nature for damages due to error, fault,
defect, or deficiency regardless of cause. All warranties of any
kind, either express or implied, including, but not limited to, the
implied warranties of merchantability and fitness for a particular
purpose are disclaimed.
======================================================================
#258. You can specify a user-defined base class for your parser
The base class must constructor must have a signature similar to
that of ANTLRParser.
#253. Generation of block preamble (-preamble and -preamble_first)
The antlr option -preamble causes antlr to insert the code
BLOCK_PREAMBLE at the start of each rule and block.
The antlr option -preamble_first is similar, but inserts the
code BLOCK_PREAMBLE_FIRST(PreambleFirst_123) where the symbol
PreambleFirst_123 is equivalent to the first set defined by
the #FirstSetSymbol described in Item #248.
#248. Generate symbol for first set of an alternative
rr : #FirstSetSymbol(rr_FirstSet) ( Foo | Bar ) ;
#216. Defer token fetch for C++ mode
When the ANTLRParser class is built with the pre-processor option
ZZDEFER_FETCH defined, the fetch of new tokens by consume() is deferred
until LA(i) or LT(i) is called.
#215. Use reset() to reset DLGLexerBase
#188. Added pccts/h/DLG_stream_input.h
#180. Added ANTLRParser::getEofToken()
#173. -glms for Microsoft style filenames with -gl
#170. Suppression for predicates with lookahead depth >1
Consider the following grammar with -ck 2 and the predicate in rule
"a" with depth 2:
r1 : (ab)* "@"
;
ab : a
| b
;
a : (A B)? => <<p(LATEXT(2))>>? A B C
;
b : A B C
;
Normally, the predicate would be hoisted into rule r1 in order to
determine whether to call rule "ab". However it should *not* be
hoisted because, even if p is false, there is a valid alternative
in rule b. With "-mrhoistk on" the predicate will be suppressed.
If "-info p" command line option is present the following information
will appear in the generated code:
while ( (LA(1)==A)
#if 0
Part (or all) of predicate with depth > 1 suppressed by alternative
without predicate
pred << p(LATEXT(2))>>?
depth=k=2 ("=>" guard) rule a line 8 t1.g
tree context:
(root = A
B
)
The token sequence which is suppressed: ( A B )
The sequence of references which generate that sequence of tokens:
1 to ab r1/1 line 1 t1.g
2 ab ab/1 line 4 t1.g
3 to b ab/2 line 5 t1.g
4 b b/1 line 11 t1.g
5 #token A b/1 line 11 t1.g
6 #token B b/1 line 11 t1.g
#endif
A slightly more complicated example:
r1 : (ab)* "@"
;
ab : a
| b
;
a : (A B)? => <<p(LATEXT(2))>>? (A B | D E)
;
b : <<q(LATEXT(2))>>? D E
;
In this case, the sequence (D E) in rule "a" which lies behind
the guard is used to suppress the predicate with context (D E)
in rule b.
while ( (LA(1)==A || LA(1)==D)
#if 0
Part (or all) of predicate with depth > 1 suppressed by alternative
without predicate
pred << q(LATEXT(2))>>?
depth=k=2 rule b line 11 t2.g
tree context:
(root = D
E
)
The token sequence which is suppressed: ( D E )
The sequence of references which generate that sequence of tokens:
1 to ab r1/1 line 1 t2.g
2 ab ab/1 line 4 t2.g
3 to a ab/1 line 4 t2.g
4 a a/1 line 8 t2.g
5 #token D a/1 line 8 t2.g
6 #token E a/1 line 8 t2.g
#endif
&&
#if 0
pred << p(LATEXT(2))>>?
depth=k=2 ("=>" guard) rule a line 8 t2.g
tree context:
(root = A
B
)
#endif
(! ( LA(1)==A && LA(2)==B ) || p(LATEXT(2)) ) {
ab();
...
#165. (Changed in MR13) option -newAST
To create ASTs from an ANTLRTokenPtr antlr usually calls
"new AST(ANTLRTokenPtr)". This option generates a call
to "newAST(ANTLRTokenPtr)" instead. This allows a user
to define a parser member function to create an AST object.
#161. (Changed in MR13) Switch -gxt inhibits generation of tokens.h
#158. (Changed in MR13) #header causes problem for pre-processors
A user who runs the C pre-processor on antlr source suggested
that another syntax be allowed. With MR13 such directives
such as #header, #pragma, etc. may be written as "\#header",
"\#pragma", etc. For escaping pre-processor directives inside
a #header use something like the following:
\#header
<<
\#include <stdio.h>
>>
#155. (Changed in MR13) Context behind predicates can suppress
With -mrhoist enabled the context behind a guarded predicate can
be used to suppress other predicates. Consider the following grammar:
r0 : (r1)+;
r1 : rp
| rq
;
rp : <<p LATEXT(1)>>? B ;
rq : (A)? => <<q LATEXT(1)>>? (A|B);
In earlier versions both predicates "p" and "q" would be hoisted into
rule r0. With MR12c predicate p is suppressed because the context which
follows predicate q includes "B" which can "cover" predicate "p". In
other words, in trying to decide in r0 whether to call r1, it doesn't
really matter whether p is false or true because, either way, there is
a valid choice within r1.
#154. (Changed in MR13) Making hoist suppression explicit using <<nohoist>>
A common error, even among experienced pccts users, is to code
an init-action to inhibit hoisting rather than a leading action.
An init-action does not inhibit hoisting.
This was coded:
rule1 : <<;>> rule2
This is what was meant:
rule1 : <<;>> <<;>> rule2
With MR13, the user can code:
rule1 : <<;>> <<nohoist>> rule2
The following will give an error message:
rule1 : <<nohoist>> rule2
If the <<nohoist>> appears as an init-action rather than a leading
action an error message is issued. The meaning of an init-action
containing "nohoist" is unclear: does it apply to just one
alternative or to all alternatives ?
#151a. Addition of ANTLRParser::getLexer(), ANTLRTokenStream::getLexer()
You must manually cast the ANTLRTokenStream to your program's
lexer class. Because the name of the lexer's class is not fixed.
Thus it is impossible to incorporate it into the DLGLexerBase
class.
#151b.(Changed in MR12) ParserBlackBox member getLexer()
#150. (Changed in MR12) syntaxErrCount and lexErrCount now public
#149. (Changed in MR12) antlr option -info o (letter o for orphan)
If there is more than one rule which is not referenced by any
other rule then all such rules are listed. This is useful for
alerting one to rules which are not used, but which can still
contribute to ambiguity.
#148. (Changed in MR11) #token names appearing in zztokens,token_tbl
One can write:
#token Plus ("+") "\+"
#token RP ("(") "\("
#token COM ("comment begin") "/\*"
The string in parenthesis will be used in syntax error messages.
#146. (Changed in MR11) Option -treport for locating "difficult" alts
It can be difficult to determine which alternatives are causing
pccts to work hard to resolve an ambiguity. In some cases the
ambiguity is successfully resolved after much CPU time so there
is no message at all.
A rough measure of the amount of work being peformed which is
independent of the CPU speed and system load is the number of
tnodes created. Using "-info t" gives information about the
total number of tnodes created and the peak number of tnodes.
Tree Nodes: peak 1300k created 1416k lost 0
It also puts in the generated C or C++ file the number of tnodes
created for a rule (at the end of the rule). However this
information is not sufficient to locate the alternatives within
a rule which are causing the creation of tnodes.
Using:
antlr -treport 100000 ....
causes antlr to list on stdout any alternatives which require the
creation of more than 100,000 tnodes, along with the lookahead sets
for those alternatives.
The following is a trivial case from the ansi.g grammar which shows
the format of the report. This report might be of more interest
in cases where 1,000,000 tuples were created to resolve the ambiguity.
-------------------------------------------------------------------------
There were 0 tuples whose ambiguity could not be resolved
by full lookahead
There were 157 tnodes created to resolve ambiguity between:
Choice 1: statement/2 line 475 file ansi.g
Choice 2: statement/3 line 476 file ansi.g
Intersection of lookahead[1] sets:
IDENTIFIER
Intersection of lookahead[2] sets:
LPARENTHESIS COLON AMPERSAND MINUS
STAR PLUSPLUS MINUSMINUS ONESCOMPLEMENT
NOT SIZEOF OCTALINT DECIMALINT
HEXADECIMALINT FLOATONE FLOATTWO IDENTIFIER
STRING CHARACTER
-------------------------------------------------------------------------
#143. (Changed in MR11) Optional ";" at end of #token statement
Fixes problem of:
#token X "x"
<<
parser action
>>
Being confused with:
#token X "x" <<lexical action>>
#142. (Changed in MR11) class BufFileInput subclass of DLGInputStream
Alexey Demakov (demakov@kazbek.ispras.ru) has supplied class
BufFileInput derived from DLGInputStream which provides a
function lookahead(char *string) to test characters in the
input stream more than one character ahead.
The class is located in pccts/h/BufFileInput.* of the kit.
#140. #pred to define predicates
+---------------------------------------------------+
| Note: Assume "-prc on" for this entire discussion |
+---------------------------------------------------+
A problem with predicates is that each one is regarded as
unique and capable of disambiguating cases where two
alternatives have identical lookahead. For example:
rule : <<pred(LATEXT(1))>>? A
| <<pred(LATEXT(1))>>? A
;
will not cause any error messages or warnings to be issued
by earlier versions of pccts. To compare the text of the
predicates is an incomplete solution.
In 1.33MR11 I am introducing the #pred statement in order to
solve some problems with predicates. The #pred statement allows
one to give a symbolic name to a "predicate literal" or a
"predicate expression" in order to refer to it in other predicate
expressions or in the rules of the grammar.
The predicate literal associated with a predicate symbol is C
or C++ code which can be used to test the condition. A
predicate expression defines a predicate symbol in terms of other
predicate symbols using "!", "&&", and "||". A predicate symbol
can be defined in terms of a predicate literal, a predicate
expression, or *both*.
When a predicate symbol is defined with both a predicate literal
and a predicate expression, the predicate literal is used to generate
code, but the predicate expression is used to check for two
alternatives with identical predicates in both alternatives.
Here are some examples of #pred statements:
#pred IsLabel <<isLabel(LATEXT(1))>>?
#pred IsLocalVar <<isLocalVar(LATEXT(1))>>?
#pred IsGlobalVar <<isGlobalVar(LATEXT(1)>>?
#pred IsVar <<isVar(LATEXT(1))>>? IsLocalVar || IsGlobalVar
#pred IsScoped <<isScoped(LATEXT(1))>>? IsLabel || IsLocalVar
I hope that the use of EBNF notation to describe the syntax of the
#pred statement will not cause problems for my readers (joke).
predStatement : "#pred"
CapitalizedName
(
"<<predicate_literal>>?"
| "<<predicate_literal>>?" predOrExpr
| predOrExpr
)
;
predOrExpr : predAndExpr ( "||" predAndExpr ) * ;
predAndExpr : predPrimary ( "&&" predPrimary ) * ;
predPrimary : CapitalizedName
| "!" predPrimary
| "(" predOrExpr ")"
;
What is the purpose of this nonsense ?
To understand how predicate symbols help, you need to realize that
predicate symbols are used in two different ways with two different
goals.
a. Allow simplification of predicates which have been combined
during predicate hoisting.
b. Allow recognition of identical predicates which can't disambiguate
alternatives with common lookahead.
First we will discuss goal (a). Consider the following rule:
rule0: rule1
| ID
| ...
;
rule1: rule2
| rule3
;
rule2: <<isX(LATEXT(1))>>? ID ;
rule3: <<!isX(LATEXT(1)>>? ID ;
When the predicates in rule2 and rule3 are combined by hoisting
to create a prediction expression for rule1 the result is:
if ( LA(1)==ID
&& ( isX(LATEXT(1) || !isX(LATEXT(1) ) ) { rule1(); ...
This is inefficient, but more importantly, can lead to false
assumptions that the predicate expression distinguishes the rule1
alternative with some other alternative with lookahead ID. In
MR11 one can write:
#pred IsX <<isX(LATEXT(1))>>?
...
rule2: <<IsX>>? ID ;
rule3: <<!IsX>>? ID ;
During hoisting MR11 recognizes this as a special case and
eliminates the predicates. The result is a prediction
expression like the following:
if ( LA(1)==ID ) { rule1(); ...
Please note that the following cases which appear to be equivalent
*cannot* be simplified by MR11 during hoisting because the hoisting
logic only checks for a "!" in the predicate action, not in the
predicate expression for a predicate symbol.
*Not* equivalent and is not simplified during hoisting:
#pred IsX <<isX(LATEXT(1))>>?
#pred NotX <<!isX(LATEXT(1))>>?
...
rule2: <<IsX>>? ID ;
rule3: <<NotX>>? ID ;
*Not* equivalent and is not simplified during hoisting:
#pred IsX <<isX(LATEXT(1))>>?
#pred NotX !IsX
...
rule2: <<IsX>>? ID ;
rule3: <<NotX>>? ID ;
Now we will discuss goal (b).
When antlr discovers that there is a lookahead ambiguity between
two alternatives it attempts to resolve the ambiguity by searching
for predicates in both alternatives. In the past any predicate
would do, even if the same one appeared in both alternatives:
rule: <<p(LATEXT(1))>>? X
| <<p(LATEXT(1))>>? X
;
The #pred statement is a start towards solving this problem.
During ambiguity resolution (*not* predicate hoisting) the
predicates for the two alternatives are expanded and compared.
Consider the following example:
#pred Upper <<isUpper(LATEXT(1))>>?
#pred Lower <<isLower(LATEXT(1))>>?
#pred Alpha <<isAlpha(LATEXT(1))>>? Upper || Lower
rule0: rule1
| <<Alpha>>? ID
;
rule1:
| rule2
| rule3
...
;
rule2: <<Upper>>? ID;
rule3: <<Lower>>? ID;
The definition of #pred Alpha expresses:
a. to test the predicate use the C code "isAlpha(LATEXT(1))"
b. to analyze the predicate use the information that
Alpha is equivalent to the union of Upper and Lower,
During ambiguity resolution the definition of Alpha is expanded
into "Upper || Lower" and compared with the predicate in the other
alternative, which is also "Upper || Lower". Because they are
identical MR11 will report a problem.
-------------------------------------------------------------------------
t10.g, line 5: warning: the predicates used to disambiguate rule rule0
(file t10.g alt 1 line 5 and alt 2 line 6)
are identical when compared without context and may have no
resolving power for some lookahead sequences.
-------------------------------------------------------------------------
If you use the "-info p" option the output file will contain:
+----------------------------------------------------------------------+
|#if 0 |
| |
|The following predicates are identical when compared without |
| lookahead context information. For some ambiguous lookahead |
| sequences they may not have any power to resolve the ambiguity. |
| |
|Choice 1: rule0/1 alt 1 line 5 file t10.g |
| |
| The original predicate for choice 1 with available context |
| information: |
| |
| OR expr |
| |
| pred << Upper>>? |
| depth=k=1 rule rule2 line 14 t10.g |
| set context: |
| ID |
| |
| pred << Lower>>? |
| depth=k=1 rule rule3 line 15 t10.g |
| set context: |
| ID |
| |
| The predicate for choice 1 after expansion (but without context |
| information): |
| |
| OR expr |
| |
| pred << isUpper(LATEXT(1))>>? |
| depth=k=1 rule line 1 t10.g |
| |
| pred << isLower(LATEXT(1))>>? |
| depth=k=1 rule line 2 t10.g |
| |
| |
|Choice 2: rule0/2 alt 2 line 6 file t10.g |
| |
| The original predicate for choice 2 with available context |
| information: |
| |
| pred << Alpha>>? |
| depth=k=1 rule rule0 line 6 t10.g |
| set context: |
| ID |
| |
| The predicate for choice 2 after expansion (but without context |
| information): |
| |
| OR expr |
| |
| pred << isUpper(LATEXT(1))>>? |
| depth=k=1 rule line 1 t10.g |
| |
| pred << isLower(LATEXT(1))>>? |
| depth=k=1 rule line 2 t10.g |
| |
| |
|#endif |
+----------------------------------------------------------------------+
The comparison of the predicates for the two alternatives takes
place without context information, which means that in some cases
the predicates will be considered identical even though they operate
on disjoint lookahead sets. Consider:
#pred Alpha
rule1: <<Alpha>>? ID
| <<Alpha>>? Label
;
Because the comparison of predicates takes place without context
these will be considered identical. The reason for comparing
without context is that otherwise it would be necessary to re-evaluate
the entire predicate expression for each possible lookahead sequence.
This would require more code to be written and more CPU time during
grammar analysis, and it is not yet clear whether anyone will even make
use of the new #pred facility.
A temporary workaround might be to use different #pred statements
for predicates you know have different context. This would avoid
extraneous warnings.
The above example might be termed a "false positive". Comparison
without context will also lead to "false negatives". Consider the
following example:
#pred Alpha
#pred Beta
rule1: <<Alpha>>? A
| rule2
;
rule2: <<Alpha>>? A
| <<Beta>>? B
;
The predicate used for alt 2 of rule1 is (Alpha || Beta). This
appears to be different than the predicate Alpha used for alt1.
However, the context of Beta is B. Thus when the lookahead is A
Beta will have no resolving power and Alpha will be used for both
alternatives. Using the same predicate for both alternatives isn't
very helpful, but this will not be detected with 1.33MR11.
To properly handle this the predicate expression would have to be
evaluated for each distinct lookahead context.
To determine whether two predicate expressions are identical is
difficult. The routine may fail to identify identical predicates.
The #pred feature also compares predicates to see if a choice between
alternatives which is resolved by a predicate which makes the second
choice unreachable. Consider the following example:
#pred A <<A(LATEXT(1)>>?
#pred B <<B(LATEXT(1)>>?
#pred A_or_B A || B
r : s
| t
;
s : <<A_or_B>>? ID
;
t : <<A>>? ID
;
----------------------------------------------------------------------------
t11.g, line 5: warning: the predicate used to disambiguate the
first choice of rule r
(file t11.g alt 1 line 5 and alt 2 line 6)
appears to "cover" the second predicate when compared without context.
The second predicate may have no resolving power for some lookahead
sequences.
----------------------------------------------------------------------------
#132. (Changed in 1.33MR11) Recognition of identical predicates in alts
Prior to 1.33MR11, there would be no ambiguity warning when the
very same predicate was used to disambiguate both alternatives:
test: ref B
| ref C
;
ref : <<pred(LATEXT(1)>>? A
In 1.33MR11 this will cause the warning:
warning: the predicates used to disambiguate rule test
(file v98.g alt 1 line 1 and alt 2 line 2)
are identical and have no resolving power
----------------- Note -----------------
This is different than the following case
test: <<pred(LATEXT(1))>>? A B
| <<pred(LATEXT(1)>>? A C
;
In this case there are two distinct predicates
which have exactly the same text. In the first
example there are two references to the same
predicate. The problem represented by this
grammar will be addressed later.
#127. (Changed in 1.33MR11)
Count Syntax Errors Count DLG Errors
------------------- ----------------
C++ mode ANTLRParser:: DLGLexerBase::
syntaxErrCount lexErrCount
C mode zzSyntaxErrCount zzLexErrCount
The C mode variables are global and initialized to 0.
They are *not* reset to 0 automatically when antlr is
restarted.
The C++ mode variables are public. They are initialized
to 0 by the constructors. They are *not* reset to 0 by the
ANTLRParser::init() method.
Suggested by Reinier van den Born (reinier@vnet.ibm.com).
#126. (Changed in 1.33MR11) Addition of #first <<...>>
The #first <<...>> inserts the specified text in the output
files before any other #include statements required by pccts.
The only things before the #first text are comments and
a #define ANTLR_VERSION.
Requested by and Esa Pulkkinen (esap@cs.tut.fi) and Alexin
Zoltan (alexin@inf.u-szeged.hu).
#124. A Note on the New "&&" Style Guarded Predicates
I've been asked several times, "What is the difference between
the old "=>" style guard predicates and the new style "&&" guard
predicates, and how do you choose one over the other" ?
The main difference is that the "=>" does not apply the
predicate if the context guard doesn't match, whereas
the && form always does. What is the significance ?
If you have a predicate which is not on the "leading edge"
it is cannot be hoisted. Suppose you need a predicate that
looks at LA(2). You must introduce it manually. The
classic example is:
castExpr :
LP typeName RP
| ....
;
typeName : <<isTypeName(LATEXT(1))>>? ID
| STRUCT ID
;
The problem is that isTypeName() isn't on the leading edge
of typeName, so it won't be hoisted into castExpr to help
make a decision on which production to choose.
The *first* attempt to fix it is this:
castExpr :
<<isTypeName(LATEXT(2))>>?
LP typeName RP
| ....
;
Unfortunately, this won't work because it ignores
the problem of STRUCT. The solution is to apply
isTypeName() in castExpr if LA(2) is an ID and
don't apply it when LA(2) is STRUCT:
castExpr :
(LP ID)? => <<isTypeName(LATEXT(2))>>?
LP typeName RP
| ....
;
In conclusion, the "=>" style guarded predicate is
useful when:
a. the tokens required for the predicate
are not on the leading edge
b. there are alternatives in the expression
selected by the predicate for which the
predicate is inappropriate
If (b) were false, then one could use a simple
predicate (assuming "-prc on"):
castExpr :
<<isTypeName(LATEXT(2))>>?
LP typeName RP
| ....
;
typeName : <<isTypeName(LATEXT(1))>>? ID
;
So, when do you use the "&&" style guarded predicate ?
The new-style "&&" predicate should always be used with
predicate context. The context guard is in ADDITION to
the automatically computed context. Thus it useful for
predicates which depend on the token type for reasons
other than context.
The following example is contributed by Reinier van den Born
(reinier@vnet.ibm.com).
+-------------------------------------------------------------------------+
| This grammar has two ways to call functions: |
| |
| - a "standard" call syntax with parens and comma separated args |
| - a shell command like syntax (no parens and spacing separated args) |
| |
| The former also allows a variable to hold the name of the function, |
| the latter can also be used to call external commands. |
| |
| The grammar (simplified) looks like this: |
| |
| fun_call : ID "(" { expr ("," expr)* } ")" |
| /* ID is function name */ |
| | "@" ID "(" { expr ("," expr)* } ")" |
| /* ID is var containing fun name */ |
| ; |
| |
| command : ID expr* /* ID is function name */ |
| | path expr* /* path is external command name */ |
| ; |
| |
| path : ID /* left out slashes and such */ |
| | "@" ID /* ID is environment var */ |
| ; |
| |
| expr : .... |
| | "(" expr ")"; |
| |
| call : fun_call |
| | command |
| ; |
| |
| Obviously the call is wildly ambiguous. This is more or less how this |
| is to be resolved: |
| |
| A call begins with an ID or an @ followed by an ID. |
| |
| If it is an ID and if it is an ext. command name -> command |
| if followed by a paren -> fun_call |
| otherwise -> command |
| |
| If it is an @ and if the ID is a var name -> fun_call |
| otherwise -> command |
| |
| One can implement these rules quite neatly using && predicates: |
| |
| call : ("@" ID)? && <<isVarName(LT(2))>>? fun_call |
| | (ID)? && <<isExtCmdName>>? command |
| | (ID "(")? fun_call |
| | command |
| ; |
| |
| This can be done better, so it is not an ideal example, but it |
| conveys the principle. |
+-------------------------------------------------------------------------+
#122. (Changed in 1.33MR11) Member functions to reset DLG in C++ mode
void DLGFileReset(FILE *f) { input = f; found_eof = 0; }
void DLGStringReset(DLGChar *s) { input = s; p = &input[0]; }
Supplied by R.A. Nelson (cowboy@VNET.IBM.COM)
#119. (Changed in 1.33MR11) Ambiguity aid for grammars
The user can ask for additional information on ambiguities reported
by antlr to stdout. At the moment, only one ambiguity report can
be created in an antlr run.
This feature is enabled using the "-aa" (Ambiguity Aid) option.
The following options control the reporting of ambiguities:
-aa ruleName Selects reporting by name of rule
-aa lineNumber Selects reporting by line number
(file name not compared)
-aam Selects "multiple" reporting for a token
in the intersection set of the
alternatives.
For instance, the token ID may appear dozens
of times in various paths as the program
explores the rules which are reachable from
the point of an ambiguity. With option -aam
every possible path the search program
encounters is reported.
Without -aam only the first encounter is
reported. This may result in incomplete
information, but the information may be
sufficient and much shorter.
-aad depth Selects the depth of the search.
The default value is 1.
The number of paths to be searched, and the
size of the report can grow geometrically
with the -ck value if a full search for all
contributions to the source of the ambiguity
is explored.
The depth represents the number of tokens
in the lookahead set which are matched against
the set of ambiguous tokens. A depth of 1
means that the search stops when a lookahead
sequence of just one token is matched.
A k=1 ck=6 grammar might generate 5,000 items
in a report if a full depth 6 search is made
with the Ambiguity Aid. The source of the
problem may be in the first token and obscured
by the volume of data - I hesitate to call
it information.
When the user selects a depth > 1, the search
is first performed at depth=1 for both
alternatives, then depth=2 for both alternatives,
etc.
Sample output for rule grammar in antlr.g itself:
+---------------------------------------------------------------------+
| Ambiguity Aid |
| |
| Choice 1: grammar/70 line 632 file a.g |
| Choice 2: grammar/82 line 644 file a.g |
| |
| Intersection of lookahead[1] sets: |
| |
| "\}" "class" "#errclass" "#tokclass" |
| |
| Choice:1 Depth:1 Group:1 ("#errclass") |
| 1 in (...)* block grammar/70 line 632 a.g |
| 2 to error grammar/73 line 635 a.g |
| 3 error error/1 line 894 a.g |
| 4 #token "#errclass" error/2 line 895 a.g |
| |
| Choice:1 Depth:1 Group:2 ("#tokclass") |
| 2 to tclass grammar/74 line 636 a.g |
| 3 tclass tclass/1 line 937 a.g |
| 4 #token "#tokclass" tclass/2 line 938 a.g |
| |
| Choice:1 Depth:1 Group:3 ("class") |
| 2 to class_def grammar/75 line 637 a.g |
| 3 class_def class_def/1 line 669 a.g |
| 4 #token "class" class_def/3 line 671 a.g |
| |
| Choice:1 Depth:1 Group:4 ("\}") |
| 2 #token "\}" grammar/76 line 638 a.g |
| |
| Choice:2 Depth:1 Group:5 ("#errclass") |
| 1 in (...)* block grammar/83 line 645 a.g |
| 2 to error grammar/93 line 655 a.g |
| 3 error error/1 line 894 a.g |
| 4 #token "#errclass" error/2 line 895 a.g |
| |
| Choice:2 Depth:1 Group:6 ("#tokclass") |
| 2 to tclass grammar/94 line 656 a.g |
| 3 tclass tclass/1 line 937 a.g |
| 4 #token "#tokclass" tclass/2 line 938 a.g |
| |
| Choice:2 Depth:1 Group:7 ("class") |
| 2 to class_def grammar/95 line 657 a.g |
| 3 class_def class_def/1 line 669 a.g |
| 4 #token "class" class_def/3 line 671 a.g |
| |
| Choice:2 Depth:1 Group:8 ("\}") |
| 2 #token "\}" grammar/96 line 658 a.g |
+---------------------------------------------------------------------+
For a linear lookahead set ambiguity (where k=1 or for k>1 but
when all lookahead sets [i] with i<k all have degree one) the
reports appear in the following order:
for (depth=1 ; depth <= "-aad depth" ; depth++) {
for (alternative=1; alternative <=2 ; alternative++) {
while (matches-are-found) {
group++;
print-report
};
};
};
For reporting a k-tuple ambiguity, the reports appear in the
following order:
for (depth=1 ; depth <= "-aad depth" ; depth++) {
while (matches-are-found) {
for (alternative=1; alternative <=2 ; alternative++) {
group++;
print-report
};
};
};
This is because matches are generated in different ways for
linear lookahead and k-tuples.
#117. (Changed in 1.33MR10) new EXPERIMENTAL predicate hoisting code
The hoisting of predicates into rules to create prediction
expressions is a problem in antlr. Consider the following
example (k=1 with -prc on):
start : (a)* "@" ;
a : b | c ;
b : <<isUpper(LATEXT(1))>>? A ;
c : A ;
Prior to 1.33MR10 the code generated for "start" would resemble:
while {
if (LA(1)==A &&
(!LA(1)==A || isUpper())) {
a();
}
};
This code is wrong because it makes rule "c" unreachable from
"start". The essence of the problem is that antlr fails to
recognize that there can be a valid alternative within "a" even
when the predicate <<isUpper(LATEXT(1))>>? is false.
In 1.33MR10 with -mrhoist the hoisting of the predicate into
"start" is suppressed because it recognizes that "c" can
cover all the cases where the predicate is false:
while {
if (LA(1)==A) {
a();
}
};
With the antlr "-info p" switch the user will receive information
about the predicate suppression in the generated file:
--------------------------------------------------------------
#if 0
Hoisting of predicate suppressed by alternative without predicate.
The alt without the predicate includes all cases where
the predicate is false.
WITH predicate: line 7 v1.g
WITHOUT predicate: line 7 v1.g
The context set for the predicate:
A
The lookahead set for the alt WITHOUT the semantic predicate:
A
The predicate:
pred << isUpper(LATEXT(1))>>?
depth=k=1 rule b line 9 v1.g
set context:
A
tree context: null
Chain of referenced rules:
#0 in rule start (line 5 v1.g) to rule a
#1 in rule a (line 7 v1.g)
#endif
--------------------------------------------------------------
A predicate can be suppressed by a combination of alternatives
which, taken together, cover a predicate:
start : (a)* "@" ;
a : b | ca | cb | cc ;
b : <<isUpper(LATEXT(1))>>? ( A | B | C ) ;
ca : A ;
cb : B ;
cc : C ;
Consider a more complex example in which "c" covers only part of
a predicate:
start : (a)* "@" ;
a : b
| c
;
b : <<isUpper(LATEXT(1))>>?
( A
| X
);
c : A
;
Prior to 1.33MR10 the code generated for "start" would resemble:
while {
if ( (LA(1)==A || LA(1)==X) &&
(! (LA(1)==A || LA(1)==X) || isUpper()) {
a();
}
};
With 1.33MR10 and -mrhoist the predicate context is restricted to
the non-covered lookahead. The code resembles:
while {
if ( (LA(1)==A || LA(1)==X) &&
(! (LA(1)==X) || isUpper()) {
a();
}
};
With the antlr "-info p" switch the user will receive information
about the predicate restriction in the generated file:
--------------------------------------------------------------
#if 0
Restricting the context of a predicate because of overlap
in the lookahead set between the alternative with the
semantic predicate and one without
Without this restriction the alternative without the predicate
could not be reached when input matched the context of the
predicate and the predicate was false.
WITH predicate: line 11 v4.g
WITHOUT predicate: line 12 v4.g
The original context set for the predicate:
A X
The lookahead set for the alt WITHOUT the semantic predicate:
A
The intersection of the two sets
A
The original predicate:
pred << isUpper(LATEXT(1))>>?
depth=k=1 rule b line 15 v4.g
set context:
A X
tree context: null
The new (modified) form of the predicate:
pred << isUpper(LATEXT(1))>>?
depth=k=1 rule b line 15 v4.g
set context:
X
tree context: null
#endif
--------------------------------------------------------------
The bad news about -mrhoist:
(a) -mrhoist does not analyze predicates with lookahead
depth > 1.
(b) -mrhoist does not look past a guarded predicate to
find context which might cover other predicates.
For these cases you might want to use syntactic predicates.
When a semantic predicate fails during guess mode the guess
fails and the next alternative is tried.
Limitation (a) is illustrated by the following example:
start : (stmt)* EOF ;
stmt : cast
| expr
;
cast : <<isTypename(LATEXT(2))>>? LP ID RP ;
expr : LP ID RP ;
This is not much different from the first example, except that
it requires two tokens of lookahead context to determine what
to do. This predicate is NOT suppressed because the current version
is unable to handle predicates with depth > 1.
A predicate can be combined with other predicates during hoisting.
In those cases the depth=1 predicates are still handled. Thus,
in the following example the isUpper() predicate will be suppressed
by line #4 when hoisted from "bizarre" into "start", but will still
be present in "bizarre" in order to predict "stmt".
start : (bizarre)* EOF ; // #1
// #2
bizarre : stmt // #3
| A // #4
;
stmt : cast
| expr
;
cast : <<isTypename(LATEXT(2))>>? LP ID RP ;
expr : LP ID RP ;
| <<isUpper(LATEXT(1))>>? A
Limitation (b) is illustrated by the following example of a
context guarded predicate:
rule : (A)? <<p>>? // #1
(A // #2
|B // #3
) // #4
| <<q>> B // #5
;
Recall that this means that when the lookahead is NOT A then
the predicate "p" is ignored and it attempts to match "A|B".
Ideally, the "B" at line #3 should suppress predicate "q".
However, the current version does not attempt to look past
the guard predicate to find context which might suppress other
predicates.
In some cases -mrhoist will lead to the reporting of ambiguities
which were not visible before:
start : (a)* "@";
a : bc | d;
bc : b | c ;
b : <<isUpper(LATEXT(1))>>? A;
c : A ;
d : A ;
In this case there is a true ambiguity in "a" between "bc" and "d"
which can both match "A". Without -mrhoist the predicate in "b"
is hoisted into "a" and there is no ambiguity reported. However,
with -mrhoist, the predicate in "b" is suppressed by "c" (as it
should be) making the ambiguity in "a" apparent.
The motivations for these changes were hoisting problems reported
by Reinier van den Born (reinier@vnet.ibm.com) and several others.
#113. (Changed in 1.33MR10) new context guarded pred: (g)? && <<p>>? expr
The existing context guarded predicate:
rule : (guard)? => <<p>>? expr
| next_alternative
;
generates code which resembles:
if (lookahead(expr) && (!guard || pred)) {
expr()
} else ....
This is not suitable for some applications because it allows
expr() to be invoked when the predicate is false. This is
intentional because it is meant to mimic automatically computed
predicate context.
The new context guarded predicate uses the guard information
differently because it has a different goal. Consider:
rule : (guard)? && <<p>>? expr
| next_alternative
;
The new style of context guarded predicate is equivalent to:
rule : <<guard==true && pred>>? expr
| next_alternative
;
It generates code which resembles:
if (lookahead(expr) && guard && pred) {
expr();
} else ...
Both forms of guarded predicates severely restrict the form of
the context guard: it can contain no rule references, no
(...)*, no (...)+, and no {...}. It may contain token and
token class references, and alternation ("|").
Addition for 1.33MR11: in the token expression all tokens must
be at the same height of the token tree:
(A ( B | C))? && ... is ok (all height 2)
(A ( B | ))? && ... is not ok (some 1, some 2)
(A B C D | E F G H)? && ... is ok (all height 4)
(A B C D | E )? && ... is not ok (some 4, some 1)
This restriction is required in order to properly compute the lookahead
set for expressions like:
rule1 : (A B C)? && <<pred>>? rule2 ;
rule2 : (A|X) (B|Y) (C|Z);
This addition was suggested by Rienier van den Born (reinier@vnet.ibm.com)
#109. (Changed in 1.33MR10) improved trace information
The quality of the trace information provided by the "-gd"
switch has been improved significantly. Here is an example
of the output from a test program. It shows the rule name,
the first token of lookahead, the call depth, and the guess
status:
exit rule gusxx {"?"} depth 2
enter rule gusxx {"?"} depth 2
enter rule gus1 {"o"} depth 3 guessing
guess done - returning to rule gus1 {"o"} at depth 3
(guess mode continues - an enclosing guess is still active)
guess done - returning to rule gus1 {"Z"} at depth 3
(guess mode continues - an enclosing guess is still active)
exit rule gus1 {"Z"} depth 3 guessing
guess done - returning to rule gusxx {"o"} at depth 2 (guess mode ends)
enter rule gus1 {"o"} depth 3
guess done - returning to rule gus1 {"o"} at depth 3 (guess mode ends)
guess done - returning to rule gus1 {"Z"} at depth 3 (guess mode ends)
exit rule gus1 {"Z"} depth 3
line 1: syntax error at "Z" missing SC
...
Rule trace reporting is controlled by the value of the integer
[zz]traceOptionValue: when it is positive tracing is enabled,
otherwise it is disabled. Tracing during guess mode is controlled
by the value of the integer [zz]traceGuessOptionValue. When
it is positive AND [zz]traceOptionValue is positive rule trace
is reported in guess mode.
The values of [zz]traceOptionValue and [zz]traceGuessOptionValue
can be adjusted by subroutine calls listed below.
Depending on the presence or absence of the antlr -gd switch
the variable [zz]traceOptionValueDefault is set to 0 or 1. When
the parser is initialized or [zz]traceReset() is called the
value of [zz]traceOptionValueDefault is copied to [zz]traceOptionValue.
The value of [zz]traceGuessOptionValue is always initialzed to 1,
but, as noted earlier, nothing will be reported unless
[zz]traceOptionValue is also positive.
When the parser state is saved/restored the value of the trace
variables are also saved/restored. If a restore causes a change in
reporting behavior from on to off or vice versa this will be reported.
When the -gd option is selected, the macro "#define zzTRACE_RULES"
is added to appropriate output files.
C++ mode
--------
int traceOption(int delta)
int traceGuessOption(int delta)
void traceReset()
int traceOptionValueDefault
C mode
--------
int zzTraceOption(int delta)
int zzTraceGuessOption(int delta)
void zzTraceReset()
int zzTraceOptionValueDefault
The argument "delta" is added to the traceOptionValue. To
turn on trace when inside a particular rule one:
rule : <<traceOption(+1);>>
(
rest-of-rule
)
<<traceOption(-1);>>
; /* fail clause */ <<traceOption(-1);>>
One can use the same idea to turn *off* tracing within a
rule by using a delta of (-1).
An improvement in the rule trace was suggested by Sramji
Ramanathan (ps@kumaran.com).
#108. A Note on Deallocation of Variables Allocated in Guess Mode
NOTE
------------------------------------------------------
This mechanism only works for heap allocated variables
------------------------------------------------------
The rewrite of the trace provides the machinery necessary
to properly free variables or undo actions following a
failed guess.
The macro zzUSER_GUESS_HOOK(guessSeq,zzrv) is expanded
as part of the zzGUESS macro. When a guess is opened
the value of zzrv is 0. When a longjmp() is executed to
undo the guess, the value of zzrv will be 1.
The macro zzUSER_GUESS_DONE_HOOK(guessSeq) is expanded
as part of the zzGUESS_DONE macro. This is executed
whether the guess succeeds or fails as part of closing
the guess.
The guessSeq is a sequence number which is assigned to each
guess and is incremented by 1 for each guess which becomes
active. It is needed by the user to associate the start of
a guess with the failure and/or completion (closing) of a
guess.
Guesses are nested. They must be closed in the reverse
of the order that they are opened.
In order to free memory used by a variable during a guess
a user must write a routine which can be called to
register the variable along with the current guess sequence
number provided by the zzUSER_GUESS_HOOK macro. If the guess
fails, all variables tagged with the corresponding guess
sequence number should be released. This is ugly, but
it would require a major rewrite of antlr 1.33 to use
some mechanism other than setjmp()/longjmp().
The order of calls for a *successful* guess would be:
zzUSER_GUESS_HOOK(guessSeq,0);
zzUSER_GUESS_DONE_HOOK(guessSeq);
The order of calls for a *failed* guess would be:
zzUSER_GUESS_HOOK(guessSeq,0);
zzUSER_GUESS_HOOK(guessSeq,1);
zzUSER_GUESS_DONE_HOOK(guessSeq);
The default definitions of these macros are empty strings.
Here is an example in C++ mode. The zzUSER_GUESS_HOOK and
zzUSER_GUESS_DONE_HOOK macros and myGuessHook() routine
can be used without change in both C and C++ versions.
----------------------------------------------------------------------
<<
#include "AToken.h"
typedef ANTLRCommonToken ANTLRToken;
#include "DLGLexer.h"
int main() {
{
DLGFileInput in(stdin);
DLGLexer lexer(&in,2000);
ANTLRTokenBuffer pipe(&lexer,1);
ANTLRCommonToken aToken;
P parser(&pipe);
lexer.setToken(&aToken);
parser.init();
parser.start();
};
fclose(stdin);
fclose(stdout);
return 0;
}
>>
<<
char *s=NULL;
#undef zzUSER_GUESS_HOOK
#define zzUSER_GUESS_HOOK(guessSeq,zzrv) myGuessHook(guessSeq,zzrv);
#undef zzUSER_GUESS_DONE_HOOK
#define zzUSER_GUESS_DONE_HOOK(guessSeq) myGuessHook(guessSeq,2);
void myGuessHook(int guessSeq,int zzrv) {
if (zzrv == 0) {
fprintf(stderr,"User hook: starting guess #%d\n",guessSeq);
} else if (zzrv == 1) {
free (s);
s=NULL;
fprintf(stderr,"User hook: failed guess #%d\n",guessSeq);
} else if (zzrv == 2) {
free (s);
s=NULL;
fprintf(stderr,"User hook: ending guess #%d\n",guessSeq);
};
}
>>
#token A "a"
#token "[\t \ \n]" <<skip();>>
class P {
start : (top)+
;
top : (which) ? <<fprintf(stderr,"%s is a which\n",s); free(s); s=NULL; >>
| other <<fprintf(stderr,"%s is an other\n",s); free(s); s=NULL; >>
; <<if (s != NULL) free(s); s=NULL; >>
which : which2
;
which2 : which3
;
which3
: (label)? <<fprintf(stderr,"%s is a label\n",s);>>
| (global)? <<fprintf(stderr,"%s is a global\n",s);>>
| (exclamation)? <<fprintf(stderr,"%s is an exclamation\n",s);>>
;
label : <<s=strdup(LT(1)->getText());>> A ":" ;
global : <<s=strdup(LT(1)->getText());>> A "::" ;
exclamation : <<s=strdup(LT(1)->getText());>> A "!" ;
other : <<s=strdup(LT(1)->getText());>> "other" ;
}
----------------------------------------------------------------------
This is a silly example, but illustrates the idea. For the input
"a ::" with tracing enabled the output begins:
----------------------------------------------------------------------
enter rule "start" depth 1
enter rule "top" depth 2
User hook: starting guess #1
enter rule "which" depth 3 guessing
enter rule "which2" depth 4 guessing
enter rule "which3" depth 5 guessing
User hook: starting guess #2
enter rule "label" depth 6 guessing
guess failed
User hook: failed guess #2
guess done - returning to rule "which3" at depth 5 (guess mode continues
- an enclosing guess is still active)
User hook: ending guess #2
User hook: starting guess #3
enter rule "global" depth 6 guessing
exit rule "global" depth 6 guessing
guess done - returning to rule "which3" at depth 5 (guess mode continues
- an enclosing guess is still active)
User hook: ending guess #3
enter rule "global" depth 6 guessing
exit rule "global" depth 6 guessing
exit rule "which3" depth 5 guessing
exit rule "which2" depth 4 guessing
exit rule "which" depth 3 guessing
guess done - returning to rule "top" at depth 2 (guess mode ends)
User hook: ending guess #1
enter rule "which" depth 3
.....
----------------------------------------------------------------------
Remember:
(a) Only init-actions are executed during guess mode.
(b) A rule can be invoked multiple times during guess mode.
(c) If the guess succeeds the rule will be called once more
without guess mode so that normal actions will be executed.
This means that the init-action might need to distinguish
between guess mode and non-guess mode using the variable
[zz]guessing.
#101. (Changed in 1.33MR10) antlr -info command line switch
-info
p - extra predicate information in generated file
t - information about tnode use:
at the end of each rule in generated file
summary on stderr at end of program
m - monitor progress
prints name of each rule as it is started
flushes output at start of each rule
f - first/follow set information to stdout
0 - no operation (added in 1.33MR11)
The options may be combined and may appear in any order.
For example:
antlr -info ptm -CC -gt -mrhoist on mygrammar.g
#100a. (Changed in 1.33MR10) Predicate tree simplification
When the same predicates can be referenced in more than one
alternative of a block large predicate trees can be formed.
The difference that these optimizations make is so dramatic
that I have decided to use it even when -mrhoist is not selected.
Consider the following grammar:
start : ( all )* ;
all : a
| d
| e
| f
;
a : c A B
| c A C
;
c : <<AAA(LATEXT(2))>>?
;
d : <<BBB(LATEXT(2))>>? B C
;
e : <<CCC(LATEXT(2))>>? B C
;
f : e X Y
;
In rule "a" there is a reference to rule "c" in both alternatives.
The length of the predicate AAA is k=2 and it can be followed in
alternative 1 only by (A B) while in alternative 2 it can be
followed only by (A C). Thus they do not have identical context.
In rule "all" the alternatives which refer to rules "e" and "f" allow
elimination of the duplicate reference to predicate CCC.
The table below summarized the kind of simplification performed by
1.33MR10. In the table, X and Y stand for single predicates
(not trees).
(OR X (OR Y (OR Z))) => (OR X Y Z)
(AND X (AND Y (AND Z))) => (AND X Y Z)
(OR X (... (OR X Y) ... )) => (OR X (... Y ... ))
(AND X (... (AND X Y) ... )) => (AND X (... Y ... ))
(OR X (... (AND X Y) ... )) => (OR X (... ... ))
(AND X (... (OR X Y) ... )) => (AND X (... ... ))
(AND X) => X
(OR X) => X
In a test with a complex grammar for a real application, a predicate
tree with six OR nodes and 12 leaves was reduced to "(OR X Y Z)".
In 1.33MR10 there is a greater effort to release memory used
by predicates once they are no longer in use.
#100b. (Changed in 1.33MR10) Suppression of extra predicate tests
The following optimizations require that -mrhoist be selected.
It is relatively easy to optimize the code generated for predicate
gates when they are of the form:
(AND X Y Z ...)
or (OR X Y Z ...)
where X, Y, Z, and "..." represent individual predicates (leaves) not
predicate trees.
If the predicate is an AND the contexts of the X, Y, Z, etc. are
ANDed together to create a single Tree context for the group and
context tests for the individual predicates are suppressed:
--------------------------------------------------
Note: This was incorrect. The contexts should be
ORed together. This has been fixed. A more
complete description is available in item #152.
---------------------------------------------------
Optimization 1: (AND X Y Z ...)
Suppose the context for Xtest is LA(1)==LP and the context for
Ytest is LA(1)==LP && LA(2)==ID.
Without the optimization the code would resemble:
if (lookaheadContext &&
!(LA(1)==LP && LA(1)==LP && LA(2)==ID) ||
( (! LA(1)==LP || Xtest) &&
(! (LA(1)==LP || LA(2)==ID) || Xtest)
)) {...
With the -mrhoist optimization the code would resemble:
if (lookaheadContext &&
! (LA(1)==LP && LA(2)==ID) || (Xtest && Ytest) {...
Optimization 2: (OR X Y Z ...) with identical contexts
Suppose the context for Xtest is LA(1)==ID and for Ytest
the context is also LA(1)==ID.
Without the optimization the code would resemble:
if (lookaheadContext &&
! (LA(1)==ID || LA(1)==ID) ||
(LA(1)==ID && Xtest) ||
(LA(1)==ID && Ytest) {...
With the -mrhoist optimization the code would resemble:
if (lookaheadContext &&
(! LA(1)==ID) || (Xtest || Ytest) {...
Optimization 3: (OR X Y Z ...) with distinct contexts
Suppose the context for Xtest is LA(1)==ID and for Ytest
the context is LA(1)==LP.
Without the optimization the code would resemble:
if (lookaheadContext &&
! (LA(1)==ID || LA(1)==LP) ||
(LA(1)==ID && Xtest) ||
(LA(1)==LP && Ytest) {...
With the -mrhoist optimization the code would resemble:
if (lookaheadContext &&
(zzpf=0,
(LA(1)==ID && (zzpf=1) && Xtest) ||
(LA(1)==LP && (zzpf=1) && Ytest) ||
!zzpf) {
These may appear to be of similar complexity at first,
but the non-optimized version contains two tests of each
context while the optimized version contains only one
such test, as well as eliminating some of the inverted
logic (" !(...) || ").
Optimization 4: Computation of predicate gate trees
When generating code for the gates of predicate expressions
antlr 1.33 vanilla uses a recursive procedure to generate
"&&" and "||" expressions for testing the lookahead. As each
layer of the predicate tree is exposed a new set of "&&" and
"||" expressions on the lookahead are generated. In many
cases the lookahead being tested has already been tested.
With -mrhoist a lookahead tree is computed for the entire
lookahead expression. This means that predicates with identical
context or context which is a subset of another predicate's
context disappear.
This is especially important for predicates formed by rules
like the following:
uppperCaseVowel : <<isUpperCase(LATEXT(1))>>? vowel;
vowel: : <<isVowel(LATEXT(1))>>? LETTERS;
These predicates are combined using AND since both must be
satisfied for rule upperCaseVowel. They have identical
context which makes this optimization very effective.
The affect of Items #100a and #100b together can be dramatic. In
a very large (but real world) grammar one particular predicate
expression was reduced from an (unreadable) 50 predicate leaves,
195 LA(1) terms, and 5500 characters to an (easily comprehensible)
3 predicate leaves (all different) and a *single* LA(1) term.
#98. (Changed in 1.33MR10) Option "-info p"
When the user selects option "-info p" the program will generate
detailed information about predicates. If the user selects
"-mrhoist on" additional detail will be provided explaining
the promotion and suppression of predicates. The output is part
of the generated file and sandwiched between #if 0/#endif statements.
Consider the following k=1 grammar:
start : ( all ) * ;
all : ( a
| b
)
;
a : c B
;
c : <<LATEXT(1)>>?
| B
;
b : <<LATEXT(1)>>? X
;
Below is an excerpt of the output for rule "start" for the three
predicate options (off, on, and maintenance release style hoisting).
For those who do not wish to use the "-mrhoist on" option for code
generation the option can be used in a "diagnostic" mode to provide
valuable information:
a. where one should insert null actions to inhibit hoisting
b. a chain of rule references which shows where predicates are
being hoisted
======================================================================
Example of "-info p" with "-mrhoist on"
======================================================================
#if 0
Hoisting of predicate suppressed by alternative without predicate.
The alt without the predicate includes all cases where the
predicate is false.
WITH predicate: line 11 v36.g
WITHOUT predicate: line 12 v36.g
The context set for the predicate:
B
The lookahead set for alt WITHOUT the semantic predicate:
B
The predicate:
pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g
set context:
B
tree context: null
Chain of referenced rules:
#0 in rule start (line 1 v36.g) to rule all
#1 in rule all (line 3 v36.g) to rule a
#2 in rule a (line 8 v36.g) to rule c
#3 in rule c (line 11 v36.g)
#endif
&&
#if 0
pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g
set context:
X
tree context: null
#endif
======================================================================
Example of "-info p" with the default -prc setting ( "-prc off")
======================================================================
#if 0
OR
pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g
set context:
nil
tree context: null
pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g
set context:
nil
tree context: null
#endif
======================================================================
Example of "-info p" with "-prc on" and "-mrhoist off"
======================================================================
#if 0
OR
pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g
set context:
B
tree context: null
pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g
set context:
X
tree context: null
#endif
======================================================================
#60. (Changed in 1.33MR7) Major changes to exception handling
There were significant problems in the handling of exceptions
in 1.33 vanilla. The general problem is that it can only
process one level of exception handler. For example, a named
exception handler, an exception handler for an alternative, or
an exception for a subrule always went to the rule's exception
handler if there was no "catch" which matched the exception.
In 1.33MR7 the exception handlers properly "nest". If an
exception handler does not have a matching "catch" then the
nextmost outer exception handler is checked for an appropriate
"catch" clause, and so on until an exception handler with an
appropriate "catch" is found.
There are still undesirable features in the way exception
handlers are implemented, but I do not have time to fix them
at the moment:
The exception handlers for alternatives are outside the
block containing the alternative. This makes it impossible
to access variables declared in a block or to resume the
parse by "falling through". The parse can still be easily
resumed in other ways, but not in the most natural fashion.
This results in an inconsistentcy between named exception
handlers and exception handlers for alternatives. When
an exception handler for an alternative "falls through"
it goes to the nextmost outer handler - not the "normal
action".
A major difference between 1.33MR7 and 1.33 vanilla is
the default action after an exception is caught:
1.33 Vanilla
------------
In 1.33 vanilla the signal value is set to zero ("NoSignal")
and the code drops through to the code following the exception.
For named exception handlers this is the "normal action".
For alternative exception handlers this is the rule's handler.
1.33MR7
-------
In 1.33MR7 the signal value is NOT automatically set to zero.
There are two cases:
For named exception handlers: if the signal value has been
set to zero the code drops through to the "normal action".
For all other cases the code branches to the nextmost outer
exception handler until it reaches the handler for the rule.
The following macros have been defined for convenience:
C/C++ Mode Name
--------------------
(zz)suppressSignal
set signal & return signal arg to 0 ("NoSignal")
(zz)setSignal(intValue)
set signal & return signal arg to some value
(zz)exportSignal
copy the signal value to the return signal arg
I'm not sure why PCCTS make a distinction between the local
signal value and the return signal argument, but I'm loathe
to change the code. The burden of copying the local signal
value to the return signal argument can be given to the
default signal handler, I suppose.
#53. (Explanation for 1.33MR6) What happens after an exception is caught ?
The Book is silent about what happens after an exception
is caught.
The following code fragment prints "Error Action" followed
by "Normal Action".
test : Word ex:Number <<printf("Normal Action\n");>>
exception[ex]
catch NoViableAlt:
<<printf("Error Action\n");>>
;
The reason for "Normal Action" is that the normal flow of the
program after a user-written exception handler is to "drop through".
In the case of an exception handler for a rule this results in
the exection of a "return" statement. In the case of an
exception handler attached to an alternative, rule, or token
this is the code that would have executed had there been no
exception.
The user can achieve the desired result by using a "return"
statement.
test : Word ex:Number <<printf("Normal Action\n");>>
exception[ex]
catch NoViableAlt:
<<printf("Error Action\n"); return;>>
;
The most powerful mechanism for recovery from parse errors
in pccts is syntactic predicates because they provide
backtracking. Exceptions allow "return", "break",
"consumeUntil(...)", "goto _handler", "goto _fail", and
changing the _signal value.
#41. (Added in 1.33MR6) antlr -stdout
Using "antlr -stdout ..." forces the text that would
normally go to the grammar.c or grammar.cpp file to
stdout.
#40. (Added in 1.33MR6) antlr -tab to change tab stops
Using "antlr -tab number ..." changes the tab stops
for the grammar.c or grammar.cpp file. The number
must be between 0 and 8. Using 0 gives tab characters,
values between 1 and 8 give the appropriate number of
space characters.
#34. (Added to 1.33MR1) Add public DLGLexerBase::set_line(int newValue)
Previously there was no public function for changing the line
number maintained by the lexer.
#28. (Added to 1.33MR1) More control over DLG header
Version 1.33MR1 adds the following directives to PCCTS
for C++ mode:
#lexprefix <<source code>>
Adds source code to the DLGLexer.h file
after the #include "DLexerBase.h" but
before the start of the class definition.
#lexmember <<source code>>
Adds source code to the DLGLexer.h file
as part of the DLGLexer class body. It
appears immediately after the start of
the class and a "public: statement.