LT-TTT2 Tutorial Guide

Nearly all non-trivial transformations performed by lxtransduce are done on XML documents. Lxtransduce can process plain text but this is usually for the purpose of marshalling it into XML format, where the 'real' processing will take place. Take the following plain text example:

In July 1995 CEG Corp. posted net of $102 million, or 34 cents a share.

Late last night the company announced a growth of 20%.

The conversion from plain text to XML can be performed by the component preparetxt in the TTT2/scripts directory (this uses lxtransduce in conjunction with another utility, lxplain2xml). The example in TTT2/doc/tutorial/examples/eg01/source.xml is a representation of the above text which has been converted into XML and all words, numbers and punctuation have all been tokenised by wrapping with a <w> element. We will skip over the details of the conversion as this process is looked at in more detail in the pipelines documentation.

<TEXT>
  <p>
    <w>In</w>
    <w>July</w>
    <w>1995</w>
    <w>CEG</w>
    <w>Corp</w>
    <w>posted</w>
    <w>net</w>
    <w>of</w>
    <w>$</w>
    <w>102</w>
    <w>million</w>
    <w>,</w>
    <w>or</w>
    <w>34</w>
    <w>cents</w>
    <w>a</w>
    <w>share</w>
    <w>.</w>
  </p>
	
  <p>
    <w>Late</w>
    <w>last</w>
    <w>night</w>
    <w>the</w>
    <w>company</w>
    <w>announced</w>
    <w>a</w>
    <w>growth</w>
    <w>of</w>
    <w>20</w>
    <w>%</w>
    <w>.</w>
  </p>
</TEXT>

In the above example, the XML classifies all items, even the dollar symbol and punctuation, as word tokens. We can now show how grammars are used to identify patterns by marking up all numerical tokens in the XML above. Lxtransduce needs 'grammars', sets of rules stored as XML, to decide how to process input. We can see below TTT2/doc/tutorial/examples/eg01/grammar.gr, a very simple grammar used to identify numerical tokens:

<rules type="xml" apply="number">     <!-- TOP LEVEL ELEMENT, SPECIFIES TYPE OF INPUT AND FIRST RULE TO BE EXECUTED -->

  <rule name="number" wrap="num">     <!-- INDIVIDUAL RULE, REFERENCED AS FIRST RULE TO BE EXECUTED -->
    <query match="w[.~'^[0-9]+$']"/>
  </rule>

</rules>

The top-level element is always <rules>, which will appear exactly once in each grammar file. The <rules> element also has an attribute type, which indicates the nature of input the grammar will process. In this case, it is set to "xml" as we are dealing with XML input rather than plain text, in which case it would be set to "plain". Although there may be dozens of rules in a grammar, there is one parent-rule which is initiated first; this is indicated by the attribute apply in <rules>. To enable rules to be referenced in this way, each must have a unique name conforming with XML constraints: alphanumeric characters starting with a letter. The name of the parent-rule in this grammar is "number", which appears in the name attribute of the <rule> element. Each rule will attempt to match any sequence of input corresponding with its child elements.

The most basic kind of matching for XML grammars is <query>, which will always have a match attribute specifying the elements to match. The syntax is that of XPath, a query language for XML. XPath usage in lxtransduce will be discussed in greater detail later on. All we need to know now is that exact regular expression matches can be specified for particular elements of the tokenised input. The example above stipulates a match for the regular expression [0-9]+, i.e. a numeric string within the element <w>. We need not concern ourselves with the details of XPath syntax at this stage. All we need to know is that any valid regular expression can be placed in between the caret (^) and the dollar sign ($). Now might be a good time to take a look at the quick command reference, which specifies each element usable in lxtransduce grammars, along with valid attributes and children.

When the the grammar is executed on our input XML the output should resemble the XML shown below. As we can see below, the numerical expressions are now all contained within <num>. The choice of element in which a match is wrapped is determined by the optional wrap attribute of <rule>. A rule with no wrap attribute is still useful because it can match elements or text which can then be referenced by other rules.

<TEXT>
  <p>
    <w>In</w>
    <w>July</w>
    <num>
      <w>1995</w>    <!-- IDENTIFIED AS NUMBER -->
    </num>
    <w>CEG</w>
    <w>Corp</w>
    <w>posted</w>
    <w>net</w>
    <w>of</w>
    <w>$</w>
    <num>
      <w>102</w>     <!-- IDENTIFIED AS NUMBER -->
    </num>
    <w>million</w>
    <w>,</w>
    <w>or</w>
    <num>
      <w>34</w>      <!-- IDENTIFIED AS NUMBER -->
    </num>
    <w>cents</w>
    <w>a</w>
    <w>share</w>
    <w>.</w>
  </p>
	
  <p>
    <w>Late</w>
    <w>last</w>
    <w>night</w>
    <w>the</w>
    <w>company</w>
    <w>announced</w>
    <w>a</w>
    <w>growth</w>
    <w>of</w>
    <num>
      <w>20</w>      <!-- IDENTIFIED AS NUMBER -->
    </num>
    <w>%</w>
    <w>.</w>
  </p>
</TEXT>

You can test this for yourself by trying the example in TTT2/doc/tutorial/examples/eg01/.

You should now know enough to make useful changes to the grammar we examined above. Make a copy of the file and try altering it so that the "number" rule also matches capitalised roman numerals from 1 to 10. I.e. "I", "II", "III", "IV", "V", "VI", "VII", "VIII", "IX" and "X". The solution to this problem is given in the solutions section. HINT: this requires a single relatively simple alteration in one place.

We will now add a second rule to identify tokens referring to percentages, i.e. the character '%'. The first step is to add a new empty rule, specifying an appropriate name using the name attribute and text for the element in which matches will be wrapped using the wrap attribute:

<rules type="xml" apply="number">

  <rule name="number" wrap="num">
    <query match="w[.~'^[0-9]+$']"/>
  </rule>
  
  <rule name="percentage" wrap="per">     <!-- NEW RULE -->
  </rule>

</rules>

The regular expression '%' (without the quotes) will match the symbol we are interested in. We can put a <query> element inside our new rule with this regular expression:

<rules type="xml" apply="number">

  <rule name="number" wrap="num">
    <query match="w[.~'^[0-9]+$']"/>
  </rule>
  
  <rule name="percentage" wrap="per">
    <query match="w[.~'^%$']"/>        <!-- NEW QUERY ELEMENT -->
  </rule>

</rules>

We now face a problem that we want lxtransduce to apply both rules in the grammar, but only one can be specified as an attribute of the <rule> element specifying the first rule to be executed. This is solved by specifying a special rule with a disjunction. This rule is called "all" and contains two elements we have not used so far: <best> and <ref>. Because the "all" rule does not specify a wrap attribute, it will defer this function to the other rules which are specidied as children.

<rules type="xml" apply="all">         <!-- NOW APPLIES RULE "all" -->

  <rule name="number" wrap="num">
    <query match="w[.~'^[0-9]+$']"/>
  </rule>
  
  <rule name="percentage" wrap="per">
    <query match="w[.~'^%$']"/>
  </rule>
  
  <rule name="all">                    <!-- NEW DISJUNCTION REFERS TO OTHER RULES -->
    <best>                             <!-- MATCHES LONGEST INPUT OF ITS CHILDREN -->
      <ref name="number"/>             <!-- REFERENCES SPECIFIC RULE "number" -->
      <ref name="percentage"/>         <!-- REFERENCES SPECIFIC RULE "percentage" -->
    </best>
  </rule>

</rules>

A <ref> element refers to another rule by using the name attribute; only references to rules earlier in the grammar are allowed. The <ref> element allows complex rules to be built up from smaller, simpler ones. The two references to our existing rules are given as children inside the <best> element. The <best> element will match input if one of its child elements (in this case two <ref>) matches. If more than one child element matches then it will choose the "best" one - which will match the longest string of input. Imagine two rules, one matching instances of the string "abcd" and the other matching the string "abc". If these appeared as children of <best>, both would match a substring of "abcde" but "abcd" would be chosen as the match because it is longer.

We are now in a position to execute the grammar above which can be found in the TTT2/doc/tutorial/examples/eg02/ directory. As you will see, it correctly identifies both the percentage symbol and the numbers:

<TEXT>g
  <p>
    <w>In</w>
    <w>July</w>
    <num>
      <w>1995</w>
    </num>
    <w>CEG</w>
    <w>Corp</w>
    <w>posted</w>
    <w>net</w>
    <w>of</w>
    <w>$</w>
    <num>
      <w>102</w>
    </num>
    <w>million</w>
    <w>,</w>
    <w>or</w>
    <num>
      <w>34</w>
    </num>
    <w>cents</w>
    <w>a</w>
    <w>share</w>
    <w>.</w>
  </p>

  <p>
    <w>Late</w>
    <w>last</w>
    <w>night</w>
    <w>the</w>
    <w>company</w>
    <w>announced</w>
    <w>a</w>
    <w>growth</w>
    <w>of</w>
    <num>
      <w>20</w>
    </num>
    <per>
      <w>%</w>     <!-- IDENTIFIED AS PERCENTAGE SYMBOL -->
    </per>
    <w>.</w>
  </p>
</TEXT>

Think about adding a third rule to the grammar. What would you need to add add and where? You do not need to concern yourself with the specifics of the rule, only where it would be placed and other alterations required. The answer is given in the solutions section.

The <seq> element is one of the most commonly used and will match its child elements when they appear in sequence. For example, the following rule is part of a larger grammar; it is designed to find full names of people in text. It does this by referencing two other rules called "forename" and "surname". <seq> will only match sections which appear in the document in the order given in the grammar.

<rule name="name" wrap="nm">
  <seq>
    <ref name="forename"/>     <!-- A FORENAME MUST APPEAR FIRST -->
    <ref name="surname"/>      <!-- AND THEN A SURNAME -->
  </seq>
</rule>

<seq> can have other constructs nested as children. Suppose that the "surname" rule did not match hyphenated (so-called "double barreled") surnames. We could create another rule called "hyph-surname" which performs this function and then place all of the surname related rules in a <best> element; lxtransduce would simply pick the best match it could find in the list.

<rule name="name" wrap="nm">
  <seq>
    <ref name="forename"/>
    <best>
      <ref name="surname"/>      <!-- ORIGINAL SURNAME RULE -->
      <ref name="hyph-surname"/> <!-- HYPHENATED SURNAME RULE -->
    </best>
  </seq>
</rule>

As we will see, many other lxtransduce elements can accept other constructs as children. Sophisticated and powerful rules can be built up in this way. Take the grammar we were looking at earlier:

<rules type="xml" apply="all">

  <rule name="number" wrap="num">
    <query match="w[.~'^[0-9]+$']"/>
  </rule>
  
  <rule name="percentage" wrap="per">
    <query match="w[.~'^%$']"/>
  </rule>
  
  <rule name="all">
    <best>
      <ref name="number"/>
      <ref name="percentage"/>
    </best>
  </rule>
</rules>

Let's try and alter the grammar so that it marks up whole percentage expressions (like "34%"), rather than just numbers (like "34") and percentage symbols (like "%"). Firstly, we need to create a new rule called "per-expr" which will also wrap its matches with <per-expr>:

<rules type="xml" apply="all">

  <rule name="number" wrap="num">         <!-- RULE I -->
    <query match="w[.~'^[0-9]+$']"/>
  </rule>
  
  <rule name="percentage" wrap="per">     <!-- RULE II -->
    <query match="w[.~'^%$']"/>
  </rule>
  
  <rule name="per-expr" wrap="per-expr">  <!-- NEW SEQUENCE RULE, MATCHES RULE I FOLLOWED BY RULE II -->
    <seq>
    	<ref name="number"/>
    	<ref name="percentage"/>
    </seq>
  </rule>
  
  <rule name="all">
    <best>
      <ref name="number"/>
      <ref name="percentage"/>
    </best>
  </rule>
</rules>

We now have a new rule in place which will wrap matches for expressions such as "34%". We no longer need the "number" and "percentage" rules to wrap matches on their own and must therefore remove the wrap attribute from these rules. The "all" rule must also be altered to include only the "per-expr" rule:

<rules type="xml" apply="all">

  <rule name="number">                     <!-- WRAP ATTRIBUTE REMOVED -->
    <query match="w[.~'^[0-9]+$']"/>
  </rule>
  
  <rule name="percentage">                 <!-- WRAP ATTRIBUTE REMOVED -->
    <query match="w[.~'^%$']"/>
  </rule>
  
  <rule name="per-expr" wrap="per-expr"> 
    <seq>
    	<ref name="number"/>
    	<ref name="percentage"/>
    </seq>
  </rule>
  
  <rule name="all">
    <ref name="per-expr"/>                 <!-- SINGLE RULE WITHIN "all" -->
  </rule>
</rules>

If we run this grammar (in examples/eg03), it will no longer wrap using <num> and <per>; rather, it only wraps whole percentage expressions with <per-expr>:

  <p>
    <w>Late</w>
    <w>last</w>
    <w>night</w>
    <w>the</w>
    <w>company</w>
    <w>announced</w>
    <w>a</w>
    <w>growth</w>
    <w>of</w>
    <per-expr>
      <w>20</w>
      <w>%</w>
    </per-expr>
    <w>.</w>
  </p>

The example input we have been using contains a reference to a monetary expression: "34 cents" as in "34 cents a share". Alter the grammar we developed in this section by creating a new rule which will identify such references to cents and wrap them with <cent-expr>. We're only interested in matching whole cents here; don't worry about other units of currency such as whole dollars or fractions of cents. For the answer to this problem see the solutions section.

<first> works in precisely the same way as the <best> element we learned about earlier. The only difference is that first will look through its children in order, using the first valid match it finds rather than the longest one. Imagine a document which contains both plain numbers and also years expressed using numerals:

<p>
  <w>Simon</w>
  <w>Garner</w>
  <w>scored</w>
  <w>168</w>
  <w>goals</w>
  <w>for</w>
  <w>Blackburn</w>
  <w>Rovers</w>
  <w>between</w>
  <w>1978</w>
  <w>and</w>
  <w>1992</w>
  <w>.</w>
</p>

<first> is useful where we want to express exceptions to a rule. In the case above, the general rule is numbers and the exception is years. That is, years are a specific subset of general numbers which we may want to tag differently. We can achieve this easily using <first>, as shown in the grammar below:

<rules type="xml" apply="all">

  <rule name="number" wrap="num">     <!-- TYPICAL NUMBER RULE -->
    <query match="w[.~'^[0-9]+$']"/>
  </rule>
  
  <rule name="year" wrap="yr">        <!-- MATCHES YEARS 1900 TO 2019 -->
    <query match="w[.~'^(19[0-9][0-9])|(20[01][0-9])$']"/>
  </rule>
  
  <rule name="all">
    <first>                           <!-- WILL MATCH USING FIRST CHILD IT CAN USE -->
      <ref name="year"/>              <!-- TRIES TO MATCH YEAR EXPRESSION -->
      <ref name="number"/>            <!-- TRIES TO MATCH (POSSIBLY LONGER) NUMERICAL EXPRESSION -->
    </first>
  </rule>
</rules>

Both <best> and <first> are ways of presenting options to lxtransduce. We use the former when we want lxtransduce to choose the longest match of input, irrespective of the ordering of its children. We use the latter when we want the program to express a preference which isn't based on the length of the match and this is done through ordering the children. We can see below how lxtransduce marks up the input we gave it earlier:

<p>
  <w>Simon</w>
  <w>Garner</w>
  <w>scored</w>
  <num>
    <w>168</w>
  </num>
  <w>goals</w>
  <w>for</w>
  <w>Blackburn</w>
  <w>Rovers</w>
  <w>between</w>
  <yr>
    <w>1978</w>
  </yr>
  <w>and</w>
  <yr>
    <w>1992</w>
  </yr>
  <w>.</w>
</p>

Recall the grammar and the input XML we have seen in the previous section. What would be the effect on the marked up output if the ordering of the year and number rules were reversed in the "all" rule? The answer is given in the solutions section. HINT: This should be obvious by now. If it isn't, re-read the last section!

The following XML has been broken up into word tokens and also organised by sentence:

<TEXT>
  <sent>
    <w>What</w>
    <w>a</w>
    <w>sentence</w>
    <w>!</w>
  </sent>
  <sent>
    <w>And</w>
    <w>another</w>
    <w>,</w>
    <w>cool</w>
    <w>!</w>
  </sent>
</TEXT>

The <start>, <end>, <not-start> and <not-end> are typically used within a <seq> to stipulate positioning relative to the container element. In the case of the <w> tokens above, their container (or parent) element is <sent>. Imagine a situation where we want to identify the last token in a sentence and mark it with <last-token>. The following grammar will markup the above input in this way:

<rules type="xml" apply="all">

  <rule name="last" wrap="last-token">
    <seq>
      <query match="w"/>
      <end/>                           <!-- END ELEMENT -->
    </seq>
  </rule>

  <rule name="all">
    <best>
      <ref name="last"/>
    </best>
  </rule>
</rules>

Notice above that although there is only one functional rule ("last"), there is also an "all" rule which references it. At this stage, this is completely unnecessary but some users of lxtransduce do this with new grammars as it makes it easier to add more rules later on. Notice also that, unlike the other grammars we have worked with, the match specified in <query> contains only one letter ("w") and no regular expression matching. This indicates that we are interested in matching any word token, regardless of its contents. We can see below the processed output:

Expand the grammar above with two more rules: one should match word tokens at the start of sentences and wrap them with <first-token>; the other should match any word tokens in the middle (all the others) and wrap them with <middle-token>. The answer is given in the solutions section. HINT: you will need to use the other three grammar elements (not including suppress, which is an attribute) mentioned in the introduction to this section.

The <start> and <not-start>-style expressions can give grammar rules a limited awareness of context limited to the XML structure of container elements. More sophisticated left and right context can be added to a <seq> element by using the suppress="true" attribute on any match element at the beginning (left) or end (right) of the sequence. Elements with suppress="true" describe context which is required for the match to succeed but is not considered part of the match, meaning it is not part of the default rewrite for the <seq>, will not be included in any wrap if one is specified and will not be part of what is passed to other rules in references. Recall the example input we saw in the previous section. Imagine that the information about sentences given by the <sent> element was absent:

<TEXT>
  <w>What</w>
  <w>a</w>
  <w>sentence</w>
  <w>!</w>
  <w>And</w>
  <w>another</w>
  <w>,</w>
  <w>cool</w>
  <w>!</w>
</TEXT>

The absence of a container element for sentences means that, for example, we would be unable to match the first word in every sentence using <start>. If we use suppress, however, we can tell lxtransduce to look for indicators such as full stops, as well as the positioning within the container element:

<rules type="xml" apply="myrule">

  <rule name="myrule" wrap="first-word">
    <seq>
      <best suppress="true">                <!-- USE OF SUPPRESS -->
        <start/>                            <!-- FIRST OPTION: BEGINNING OF CONTAINER ELEMENT -->
        <query match="w[.~'^[.!]$']"/>       <!-- SECOND OPTION: FULL STOP OR EXCLAMATION -->
      </best>
      <query match="w"/>                    <!-- ONLY THIS PART WILL BE WRAPPED -->
    </seq>
  </rule>

</rules>

The grammar above uses suppress="true" on <best> to stipulate that a sentence boundary indicator (such as a full stop) or the beginning of a container element must be followed by a word token indicated by <w>. But it is only the second part, the word, which will be wrapped with "first-word" - everything else merely provides the context required and is contained within the suppressed <best>:

<TEXT>
  <first-word>
    <w>What</w>    <!-- WRAPPED - FOLLOWS START OF CONTAINER ELEMENT -->
  </first-word>
  <w>a</w>
  <w>sentence</w>
  <w>!</w>
  <first-word>
    <w>And</w>     <!-- WRAPPED - FOLLOWS FULL STOP -->
  </first-word>
  <w>another</w>
  <w>,</w>
  <w>cool</w>
  <w>!</w>
</TEXT>

How would the grammar be extended to markup words at the ends of sentences in a similar manner to exercise 5? The answer is given in the solutions section.

As we have seen, the <ref> element is used in grammars to refer to other rules. The mult attribute can be used with <ref> to specify multiplicities on the way that the rule matches. The mult attribute takes one of three characters which may be familiar from regular expressions:

We can see below the use of this attribute in an XML grammar:

<rules type="xml" apply="mult-test">

  <rule name="a">
    <query match="a"/>
  </rule>

  <rule name="b">
    <query match="b"/>
  </rule>

  <rule name="c">
    <query match="c"/>
  </rule>

  <rule name="mult-test" wrap="X">
    <best>
      <ref name="a" mult="?"/>
      <ref name="b" mult="+"/>
      <ref name="c" mult="*"/>
    </best>
  </rule>

</rules>

We will use the input below to demonstrate this grammar:

<TEXT>
  <a/>
  <a/>
  <a/>
  <b/>
  <b/>
  <b/>
  <c/>
  <c/>
  <c/>
</TEXT>

The following output is produced. We can see that mult="?" can only match zero or one reference at a time, but "+" and "*" can match numerous:

<TEXT>
  <X><a/></X>
  <X><a/></X>
  <X><a/></X>
  <X>
    <b/>
    <b/>
    <b/>
  </X>
  <X>
    <c/>
    <c/>
    <c/>
  </X>
</TEXT>

In the instance outlined above, mult="+" and mult="*" perform the same, matching as much input as they can. These constructs would behave differently, however, if they are used in the context of a sequence indicated by <seq>. Look at the rule excerpt from a grammar shown below. In the case where the items matched by "rule2" are absent, this rule will still match if it sees the "rule1" items followed by the "rule3" items:

<rule name="example1">
  <seq>
    <ref name="rule1"/>
    <ref name="rule2" mult="*"/>
    <ref name="rule3"/>
  </seq>
</rule>

In the rule below, however, the use of mult="+" means that the items by "rule2" are a requirement for the whole "example2" rule to match.

<rule name="example2">
  <seq>
    <ref name="rule1"/>
    <ref name="rule2" mult="+"/>
    <ref name="rule3"/>
  </seq>
</rule>

A <repeat-until> element refers to another rule by using a name attribute in the same manner as <ref>. A match element (such as <best> or <seq>) must be specified as a stop condition and placed as a child element of <repeat-until>. See the following example:

<repeat-until name="repeat-rule">  <!-- REPEATED RULE -->
  <ref name="stop-rule"/>          <!-- END CONDITION -->
</repeat-until>

The repeated rule is matched until either:

The rule can succeed without the end condition matching if there is no more input matching the repeated rule. If the end condition is matched, then this input is not considered part of the overall match. Take the following input:

<TEXT>
  <a/>
  <a/>
  <a/>
  <z/>
</TEXT>

If this is used with the following grammar...

<rules type="xml" apply="repeat-test">

  <rule name="a">
    <query match="a"/>
  </rule>

  <rule name="repeat-test" wrap="X" min-matches="1">
    <repeat-until name="a">
      <query match="z"/>
    </repeat-until>
  </rule>

</rules>

...then the output shows that the repeat rule is not included in the match:

<TEXT>
  <X>
    <a/>
    <a/>
    <a/>
  </X>
  <z/>    <!-- END CONDITION NOT INCLUDED IN MATCH -->
</TEXT>

By default, a <repeat-until> will succeed even if there are no matches of the repeated rule. This can sometimes lead to problems with looping which may cause lxtransduce to abort with an error:

Top-level rule matched without consuming any input;
This is not allowed because it would loop forever

The optional min-matches attribute can be used to specify a minimum number of matches. This is typically 1. If you want to require the end condition to match, use a <seq> containing the <repeat-until> followed by another copy of the end condition:

<seq>
  <repeat-until name="repeat-rule">  <!-- REPEATED RULE -->
    <ref name="stop-rule"/>          <!-- END CONDITION -->
  </repeat-until>
  <ref name="stop-rule"/>           <!-- END CONDITION REPEATED-->
</seq>

The min-matches attribute is not needed above to prevent looping, as the rule must always consume input due to a stop rule being specified.

This construct must have two child elements in sequence:

This construct is analogous to a backtracking <seq> where the sequence is given as ordered children. lxtransduce attempts progressively shorter multiples of <ref> with the mult attribute until it finds a match. If no match is found then the whole rule fails. See, for example, this rule taken from the noung.gr grammar, bundled with LT-TTT2:

<rule name="compound-noun">
  <seq>
    <ref name="restr-nom"/>
    <backtrack>                      <!-- BACKTRACK RULE -->
      <ref name="nomger" mult="*"/>
      <ref name="compoundlast"/>
    </backtrack>
  </seq>
</rule>

We can see <backtrack> above as part of a sequence. The "nomger" reference followed by the "compoundlast" reference are attempted for a match using progressively shorter multiples of "nomger".

A <ref> element may have one or more <with-param> children specifying values to be passed to the referenced rules. These values can then be used by the second rule, for example as constraints on matches. This can save time and space: instead of devising a rule for every particular instance of a situation, the user can write a smaller number of rules which handle precisely the same instances using the flexibility of parameter passing and variables.

The following rule is taken from a grammar dealing with natural language. It is designed to co-ordinate two verbs. For example, "John dances and sings" would be co-ordinated because "dances" and "sings" are both third person singular verbs. A reference is made to a rule called "coordinate-headverb". By using <with-param> this reference passes a parameter named "pos" with value "VBZ" to the "coordinate-headverb" rule:

<rule name="headverb-presact" attrs="headv='yes'">
    <first>
       <ref name="coordinate-headverb">               <!-- REFERENCE-->
         <with-param name="pos" value="'VBZ'"/>       <!-- PASSING A PARAMETER FOR USING WITH-PARAM -->
       </ref>
       <ref name="be-pres"/>
       <query match="w[@p='VBZ']"/>
    </first>
</rule>

Note that because the value is specified as an XPath, to pass a constant string you need to put quotes around the string as well as around the attribute, resulting in something like <with-param name="cat" value="'noun'"/>. We can see this above in value="'VBZ'". If lxtransduce didn't support parameter passing in this way, instances of the coordinate-headverb rule would have to be created for every single type of verb to be handled. We can see the coordinate-headverb below. The parameter p which was passed can be referred to using $p. This is done by a <constraint> element:

<rule name="coordinate-headverb">
  <seq>
    <query match="w[@p~'^V']" attrs="headv='yes'">
      <constraint test="@p = $pos"/>                     <!-- CONSTRAINT -->
      <var name="first_tag" value="@p"/>                 <!-- ASSIGNING A VARIABLE -->
    </query>
    <ref name="conj-headverb" mult="+">
      <with-param name="subs_tag" value="$first_tag"/>   <!-- USING VARIABLE VALUE -->
    </ref>
  </seq>
</rule>

<constraint> elements appear as the children of <query> elements, placing some restriction on the conditions under which the query will match. In the case above, it tests whether an attribute matches the parameter passed to it. As we will see later, constraints are also used to test lexicon entries for categories. The rule shown above is also an example of the use of <var> to declare a variable. In this case, the variable name is first_tag and the value is set to that of an attribute which is attached to the match. Note that <with-param/> and <var> constructs take the same attributes.

We have seen that rules may examine whether particular element(s) satisfy certain constraints defined by any single rule. The <and> construct takes two or more match elements as children and allows us to test if input satisfies all of those match elements. Imagine that we have two rules, one of which tests for elements which contain three characters and another which tests for elements containing numbers:

<rules type="xml" apply="all">
  
  <rule name="length-three">                <!-- MATCHES LENGTH = 3 -->
    <query match="w[.~'^...$']"/>
  </rule>

  <rule name="number">
    <query match="w[.~'^[0123456789]+$']"/>  <!-- MATCHES NUMBERS -->
  </rule>
  
  <rule name="all" wrap="num3">
    <and>
      <ref name="length-three"/>
      <ref name="number"/>
    </and>
  </rule>

</rules>

We can see the use of <and> above to create a rule which matches elements which are of length three and are numbers. Thus if we run the above grammar on the input below:

<TEXT>
  <w>1</w>
  <w>12</w>
  <w>123</w>
  <w>1234</w>
  <w>a</w>  
  <w>ab</w>
  <w>abc</w>
  <w>abcd</w>
</TEXT>

...we obtain the following output:

<TEXT>
  <w>1</w>
  <w>12</w>
  <num3>
    <w>123</w>
  </num3>
  <w>1234</w>
  <w>a</w>
  <w>ab</w>
  <w>abc</w>
  <w>abcd</w>
</TEXT>

If the rewrite for <and> is not specified lxtransduce defaults to the rewrite of the last child. <all> is a synonym for <and>.

A <not> element contains a single match element, and matches the input if the child element does not match. A successful <not> consumes no input; its match and rewrite are always empty. Recall the input used in the previous example:

<TEXT>
  <w>1</w>
  <w>12</w>
  <w>123</w>
  <w>1234</w>
  <w>a</w>  
  <w>ab</w>
  <w>abc</w>
  <w>abcd</w>
</TEXT>

Consider that we want to define a grammar which matches every element except those which are numbers of length three. How might we modify the grammar given above to achieve this? Remember that <not> can only take one child element. It also consumes no input, so we must be careful of looping issues similar to those found with <repeat-until>. The solution to this problem is given in the solutions section.

Lexicons, sometimes referred to as gazetteers, are lists of words stored as XML which are used by lxtransduce grammars. The top-level element of a lexicon is <lexicon> with <lex> elements as children. Each lex element represents a single word and has a required word attribute. We can see below an example lexicon storing colours:

<lexicon name="colour">
 <lex word="Red"/>
 <lex word="Green"/>
 <lex word="Blue"/>
 <lex word="Pink"/>
</lexicon>

We can use this with a grammar to identify colours in XML files. Notice in the grammar below the use of <lexicon> to declare the name and location of the lexicon we are using. Lexicons can also be stipulated on the command line. Within the only rule we can see the use of the simplest and most straightforward lexicon reference for rules, <lookup>, which combines a lexicon lookup and a query:

<rules type="xml" apply="all">
  <lexicon name="colours" href="colours.lex"/>   <!-- LEXICON DECLARATION -->
  
  <rule name="all" wrap="col">
    <lookup match="w" lexicon="colours"/>        <!-- CALL TO LEXICON -->
  </rule>
</rules>

Thus if we run the grammar on the following input:

<TEXT>
  <w>Alex</w>
  <w>wears</w>
  <w>pink</w>
  <w>trousers</w>
</TEXT>

...the colour in the XML is identified correctly using the lexicon:

<TEXT>
  <w>Alex</w>
  <w>wears</w>
  <col>
    <w>pink</w>
  </col>
  <w>trousers</w>
</TEXT>

In addition to the <lookup>, we can also specify a lexicon lookup by using the constraint attribute on <query>:

<rules type="xml" apply="all">
  <lexicon name="colours" href="colours.lex"/> 
  
  <rule name="all" wrap="col">
    <query match="w" constraint="colours()"/>        <!-- CALL TO LEXICON -->
  </rule>
</rules>

<rules type="xml" apply="all">
  <lexicon name="colours" href="colours.lex"/>

  <rule name="all" wrap="col">
    <query match="w">
      <constraint test="colours()"/>            <!-- CONSTRAINT AS CHILD ELEMENT -->
    </query>
  </rule>
</rules>

Succinctly, the following are all equivalent:

<lookup match="w" lexicon="colours"/>

<query match="w" constraint="colours()"/>

<query match="w">
  <constraint test="colours()"/>
</query>

In some cases, we might want to distinguish between two or more different types of string in a lexicon, or handle words which may be in some category "A", category "B" or both. Lxtransduce allows us to assign categories or classes to words in a lexicon using the <cat> element:

<lexicon name="words">
  <lex word="boot">
    <cat>verb</cat>
    <cat>noun</cat>
  </lex>
  
  <lex word="dog">
    <cat>noun</cat>
  </lex>
  
  <lex word="weep">
    <cat>verb</cat>
  </lex>
</lexicon>

These categories may then be stipulated in the grammar as follows. These four examples are all equivalent:

<query match="w" constraint="words()/cat='noun'"/> 

<query match="w">
 <constraint test="words()/cat='noun'"/>
</query> 

<query match="w[words()/cat='noun']"/> 

It may sometimes be helpful to list phrases in lexicons rather than individual words tokens. Take, for example, this list of universities:

<lexicon name="unis">
  <lex word="University of Edinburgh"/>
  <lex word="University of St Andrews"/>
  <lex word="University of Central Lancashire"/>
  <lex word="University of Life"/>
</lexicon>

By using the phrase="true" attribute on <lookup> lxtransduce can match the lexicon even with sequences of individual word tokens. Observe the following grammar:

<rules type="xml" apply="all">

  <lexicon name="unis" href="unis.lex"/>
  
  <rule name="all" wrap="uni">
    <lookup match="w" lexicon="unis" phrase="true"/> <!-- USE OF PHRASE ATTRIBUTE -->
  </rule>

</rules>

We can see the output here from running this grammar on a series of plain <w> tokens. Notice that the match "University of Edinburgh" is a phrase split across three tokens:

<TEXT>
  <w>I</w>
  <w>graduated</w>
  <w>from</w>
  <w>the</w>
  <uni>
    <w>University</w>
    <w>of</w>
    <w>Edinburgh</w>
  </uni>
  <w>in</w>
  <w>2006</w>
</TEXT>

By default, lexicon lookups are case insensitive. There are instances, however, where we must use case sensitive implementations with the case attribute on either <lexicon> or <lex>. If used on the former we can control the case sensitivity of the entire lexicon, while on the latter we can control individual entries:

<lexicon name="dates">
  <lex word="May" case="yes">
    <cat>month</cat>
    <cat>verb</cat>
  </lex>

  <lex word="may">
    <cat>verb</cat>
  </lex>
</lexicon>

We can see above that the capitalized word "May" matches the first entry; uncapitalized it matches the second. In the first case it has both the lexical categories "month" and "verb", in the second only "verb".