Lex is a computer program that generates lexical analyzers ('scanners' or 'lexers').[1][2]
Lex is commonly used with the yaccparser generator. Lex, originally written by Mike Lesk and Eric Schmidt[3] and described in 1975,[4][5] is the standard lexical analyzer generator on many Unix systems, and an equivalent tool is specified as part of the POSIX standard.[citation needed]
Lex reads an input stream specifying the lexical analyzer and outputs source code implementing the lexer in the C programming language.In addition to C, some old versions of Lex could also generate a lexer in Ratfor.[6]
Open source[edit]
Lexical Analyzer Source Code. Following are the contents of the calc.lex file. This file contains include statements for standard input and output, as well as for the y.tab.h file. The yacc program generates that file from the yacc grammar file information if you use the -d flag with the yacc command. Chapter 3: Lexical Analysis - Sharif PPT. Presentation Summary: source program get next token lexical. Transition diagram for whitespace Lexical Analyzer Generator - Lex Lexical Compiler Lex Source program lex.l.c. Lex is a lexical analysis tool that can be used to identify specific text strings in a structured way from source text. Yacc is a grammar parser; it reads text and can be used to turn a sequence of words into a structured format for processing.
Though originally distributed as proprietary software, some versions of Lex are now open source. Open source versions of Lex, based on the original AT&T code are now distributed as a part of open source operating systems such as OpenSolaris and Plan 9 from Bell Labs.[clarification needed] One popular open source version of Lex, called flex, or the 'fast lexical analyzer', is not derived from proprietary coding.
Structure of a Lex file[edit]
The structure of a Lex file is intentionally similar to that of a yacc file; files are divided into three sections, separated by lines that contain only two percent signs, as follows
Example of a Lex file[edit]
The following is an example Lex file for the flex version of Lex. It recognizes strings of numbers (positive integers) in the input, and simply prints them out.
If this input is given to
flex , it will be converted into a C file, lex.yy.c . This can be compiled into an executable which matches and outputs strings of integers. For example, given the input:
the program will print: Second rounds on me obie trice zip.
Using Lex with other programming tools[edit]Using Lex with parser generators[edit]
Lex and parser generators, such as Yacc or Bison, are commonly used together. Parser generators use a formal grammar to parse an input stream, something which Lex cannot do using simple regular expressions (Lex is limited to simple finite state automata).[clarification needed]
It is typically preferable to have a (Yacc-generated, say) parser be fed a token-stream as input, rather than having it consume the input character-stream directly. Lex is often used to produce such a token-stream.
Scannerless parsing refers to parsing the input character-stream directly, without a distinct lexer.
What Is Lex Source ProgrammingLex and make[edit]
make is a utility that can be used to maintain programs involving Lex. Make assumes that a file that has an extension of
.l is a Lex source file. The make internal macro LFLAGS can be used to specify Lex options to be invoked automatically by make.[7]See also[edit]References[edit]
External links[edit]
Retrieved from 'https://en.wikipedia.org/w/index.php?title=Lex_(software)&oldid=907636875'
On Unix-like operating systems, the lex command generates programs to perform lexical tasks.
Description![]()
The lexutility generates C programs to be used in lexical processing of character input, and that can be used as an interface to yacc. The C programs are generated from lex source code and conform to the ISO C standard. Usually, the lex utility writes the program it generates to the file lex.yy.c. The state of this file is unspecified if lex exits with a non-zero exit status. See EXTENDED DESCRIPTION for a complete description of the lex input language.
SyntaxOptions
Syntax
Each input file contains lexsource code, which is a table of regular expressions with corresponding actions in the form of C program fragments.
When lex.yy.c is compiled and linked with the lex library (using the -ll operand with c89 or cc), the resulting program reads character input from the standard input and partitions it into strings that match the given expressions.
What Is Lex Source Programs
When an expression is matched, these actions will occur:
During pattern matching, lex searches the set of patterns for the single longest possible match. Among rules that match the same number of characters, the rule given first will be chosen.
The general format of lex source is:
The first %% is required to mark the beginning of the rules (regular expressions and actions); the second %% is required only if user subroutines follow.
Any line in the 'Definitions in lex' section beginning with a blank character will be assumed to be a C program fragment and will be copied to the external definition area of the lex.yy.c file. Similarly, anything in the 'Definitions in lex' section included between delimiter lines containing only %{ and %} will also be copied unchanged to the external definition area of the lex.yy.c file.
Any such input (beginning with a blank character or within %{ and %} delimiter lines) appearing at the beginning of the 'Rules' section before any rules are specified will be written to lex.yy.c after the declarations of variables for the yylex function and before the first line of code in yylex. Thus, user variables local to yylex can be declared here, as well as application code to execute upon entry to yylex.
The action taken by lex when encountering any input beginning with a blank character or within %{ and %} delimiter lines appearing in the 'Rules' section but coming after one or more rules is undefined. The presence of such input may result in an erroneous definition of the yylex function.
Definitions in lex
Definitions in lex appear before the first %% delimiter. Any line in this section not contained between %{ and %} lines and not beginning with a blank character is assumed to define a lex substitution string. The format of these lines is:
If a name does not meet the requirements for identifiers in the ISO C standard, the result is undefined. The string substitute will replace the string {name} when it is used in a rule. The name string is recognized in this context only when the braces are provided and when it does not appear within a bracket expression or within double-quotes.
In this section, any line beginning with a % (percent sign) character and followed by an alphanumeric word beginning with either s or S defines a set of start conditions. Any line beginning with a % followed by a word beginning with either x or X defines a set of exclusive start conditions. When the generated scanner is in a %s state, patterns with no state specified will be also active; in a %x state, such patterns will not be active. The rest of the line, after the first word, is considered to be one or more blank-character-separated names of start conditions. Start condition names are constructed in the same way as definition names. Start conditions can be used to restrict the matching of regular expressions to one or more states as described in Regular expressions in lex.
Implementations accept either of the following two mutually exclusive declarations in the Definitions in lex section:
Note: When using the %pointer option, you may not also use the yyless function to alter yytext.
%array is the default. If %array is specified (or neither %array nor %pointer is specified), then the correct way to make an external reference to yyext is with a declaration of the form:
If %pointer is specified, then the correct external reference is of the form:
lex will accept declarations for setting certain internal table sizes. The declarations are shown in the following table.
Programs generated by lex need either the -e or -w option to handle input that contains EUC characters from supplementary codesets. If neither of these options is specified, yytext is of the type char[], and the generated program can handle only ASCII characters.
When the -e option is used, yytext is of the type unsigned char[] and yyleng gives the total number of bytes in the matched string. With this option, the macros input(), unput(c), and output(c) should do a byte-based I/O in the same way as with the regular ASCII lex. Two more variables are available with the -e option, yywtext and yywleng, which behave the same as yytext and yyleng would under the -w option.
When the -w option is used, yytext is of the type wchar_t[] and yyleng gives the total number of characters in the matched string. If you supply input(), unput(c), or output(c) macros with this option, they must return or accept EUC characters in the form of wide character (wchar_t). This allows a different interface between your program and the lex internals, to expedite some programs.
Rules in lex
The Rules in lex source files are a table in which the left column contains regular expressions and the right column contains actions (C program fragments) to be executed when the expressions are recognized.
The extended regular expression (ERE) portion of a row will be separated from action by one or more blank characters. A regular expression containing blank characters is recognized under one of the following conditions:
![]() User Subroutines in lex
Anything in the user subroutines section will be copied to lex.yy.c following yylex.
Regular Expressions in lex
The lex utility supports the set of Extended Regular Expressions (EREs), with the following exceptions:
Within an ERE, a backslash character (, a, b, f, n, r, t, v) is considered to begin an escape sequence. Also, the escape sequences in the following table will be recognized.
A literal newline character cannot occur within an ERE; the escape sequence n can be used to represent a newline character. A newline character cannot be matched by a period operator.
Escape Sequences in lex
The order of precedence given to extended regular expressions for lex is as shown in the following table, from high to low.
Note: The escaped characters entry is not meant to imply that these are operators, but they are included in the table to show their relationships to the true operators. The start condition, trailing context and anchoring notations have been omitted from the table because of the placement restrictions described in this section; they can only appear at the beginning or ending of an ERE.
The ERE anchoring operators (^ and $) do not appear in the table. With lex regular expressions, these operators are restricted in their use: the ^ operator can only be used at the beginning of an entire regular expression, and the $ operator only at the end. The operators apply to the entire regular expression. Thus, for example, the pattern (^abc)|(def$) is undefined; it can instead be written as two separate rules, one with the regular expression ^abc and one with def$, which share a common action via the special | action (see below). If the pattern were written ^abc|def$, it would match either of abc or def on a line by itself.
Unlike the general ERE rules, embedded anchoring is not allowed by most historical lex implementations. An example of embedded anchoring would be for patterns such as (^)foo($) to match foo when it exists as a complete word. This functionality can be obtained using existing lex features:
Notice also that $ is a form of trailing context (it is equivalent to /n and as such cannot be used with regular expressions containing another instance of the operator (see the preceding discussion of trailing context).
The additional regular expressions trailing-context operator / (slash) can be used as an ordinary character if presented within double-quotes, '/'; preceded by a backslash, /; or within a bracket expression, [/]. The start-condition operators < and > are special only in a start condition at the beginning of a regular expression; elsewhere in the regular expression they are treated as ordinary characters.
The following examples clarify the differences between lex regular expressions and regular expressions appearing elsewhere in this document. For regular expressions of the form r/x, the string matching r is always returned; confusion may arise when the beginning of x matches the trailing portion of r. For example, given the regular expression a*b/cc and the input aaabcc, yytext would contain the string aaab on this match. But given the regular expression x*/xy and the input xxxy, the token xxx, not xx, is returned by some implementations because xxx matches x*.
In the rule ab*/bc, the b* at the end of r will extend r's match into the beginning of the trailing context, so the result is unspecified. If this rule were ab/bc, however, the rule matches the text ab when it is followed by the text bc. In this latter case, the matching of r cannot extend into the beginning of x, so the result is specified.
Lex Program DownloadActions in lex
The action to be taken when an ERE is matched can be a C program fragment or the special actions described below; the program fragment can contain one or more C statements, and can also include special actions. The empty C statement ; is a valid action; any string in the lex.yy.c input that matches the pattern portion of such a rule is effectively ignored or skipped. However, the absence of an action is not valid, and the action lex takes in such a condition is undefined.
The specification for an action, including C statements and special actions, can extend across several lines if enclosed in braces:
The default action when a string in the input to a lex.yy.c program is not matched by any expression is to copy the string to the output. Because the default behavior of a program generated by lex is to read the input and copy it to the output, a minimal lex source program that has just %% generates a C program that copies the input to the output unchanged.
Four special actions are available:
The functions or macros described below are accessible to user code included in the lex input. It is unspecified whether they appear in the C code output of lex, or are accessible only through the -ll operand to c89 or cc (the lex library).
The following functions appear only in the lex library accessible through the -ll operand; they can therefore be redefined by a portable application:
The reason for breaking these functions into two lists is that only those functions in libl.a can be reliably redefined by a portable application.
Except for input, unput and main, all external and static names generated by lex begin with the prefix yy or YY.
Usage
Portable applications are warned that in the Rules in lex section, an ERE without an action is not acceptable, but need not be detected as erroneous by lex. This may result in compilation or run-time errors.
The purpose of input is to take characters off the input stream and discard them as far as the lexical analysis is concerned. A common use is to discard the body of a comment once the beginning of a comment is recognized.
Lex Source
The lex utility is not fully internationalized in its treatment of regular expressions in the lex source code or generated lexical analyzer. It would seem desirable to have the lexical analyzer interpret the regular expressions given in the lex source according to the environment specified when the lexical analyzer is executed, but this is not possible with the current lex technology. Furthermore, the very nature of the lexical analyzers produced by lex must be closely tied to the lexical requirements of the input language being described, which will frequently be locale-specific anyway. For example, writing an analyzer that is used for French text will not automatically be useful for processing other languages.
Examples
The following is an example of a lex program that implements a rudimentary scanner for a Pascal-like syntax:
Lex Source CodeRelated commands
yacc â 'Yet another compiler-compiler.'
Comments are closed.
|
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |