Snippets, SPCC

Implementing Lexical Analyser in C++

Leave a comment

Lexical analysis (also called scanning or tokenisation) is the very first phase of a compiler. The lexical analyser reads the raw source text character by character and groups characters into meaningful units called tokens. Each token belongs to a category: an identifier (variable or function name), a literal (numeric constant), or a terminal (operator or punctuation symbol found in a predefined database file).

This C++ program reads an expression string (terminated by $) from the user, classifies each character, and builds a Uniform Symbol Table (UST) that records every token together with its type (LIT, IDN, or TER) and a pointer (sequential index within its type). The results are written to four files and then displayed on the console.

Required file: Before running the program you must create D:andb.txt containing all operator/punctuation characters that should be recognised as terminals (e.g. =+-*/()/), one or more per line.

C++ Code

#include <iostream.h>
#include <conio.h>
#include <stdio.h>
#include <ctype.h>

void main() {
    int i, j;
    int max_length = 40;
    int token_count;
    int ptr_value;
    
    char current_char, db_char;
    char token_value[10];
    char input_string[40];

    // File pointers for Lexical Analysis stages
    FILE *fp_literal, *fp_identifier, *fp_terminal, *fp_db, *fp_ust;

    clrscr();

    // Opening files with descriptive names and double backslashes
    fp_literal   = fopen("D:\\an\\literal.txt",    "w");
    fp_identifier = fopen("D:\\an\\identifier.txt", "w");
    fp_terminal   = fopen("D:\\an\\terminal.txt",   "w");
    fp_ust        = fopen("D:\\an\\ust.txt",        "w");

    if (fp_literal == NULL || fp_identifier == NULL || fp_terminal == NULL) {
        printf("Error: File system access denied. Ensure D:\\an exists.");
        getch();
        return;
    }

    // --- Input Stage ---
    cout << "\n Enter expression (terminate with $): ";
    for (i = 0; i < max_length; i++) {
        cin >> input_string[i];
        if (input_string[i] == '$') break;
    }

    // --- Terminal Identification (Operators/Symbols) ---
    token_count = 0;
    for (i = 0; i < max_length; i++) {
        current_char = input_string[i];
        if (current_char == '$') break;

        fp_db = fopen("D:\\an\\db.txt", "r");
        if (fp_db != NULL) {
            while (fscanf(fp_db, " %c", &db_char) != EOF) {
                if (db_char == current_char) {
                    fprintf(fp_terminal, "%d %c\n", token_count, db_char);
                    token_count++;
                    break;
                }
            }
            fclose(fp_db);
        }
    }
    fclose(fp_terminal);

    // --- Identifier Identification (Variables) ---
    token_count = 0;
    for (i = 0; i < max_length; i++) {
        current_char = input_string[i];
        if (current_char == '$') break;
        if (isalpha(current_char)) {
            fprintf(fp_identifier, "%d %c\n", token_count, current_char);
            token_count++;
        }
    }
    fclose(fp_identifier);

    // --- Literal Identification (Numbers) ---
    token_count = 0;
    for (i = 0; i < max_length; i++) {
        current_char = input_string[i];
        if (current_char == '$') break;
        
        if (isdigit(current_char)) {
            fprintf(fp_literal, "%d %c", token_count, current_char);
            
            // Handle multi-digit literals (up to 3 digits)
            if ((i + 1) < max_length && isdigit(input_string[i + 1])) {
                fprintf(fp_literal, "%c", input_string[++i]);
                if ((i + 1) < max_length && isdigit(input_string[i + 1])) {
                    fprintf(fp_literal, "%c", input_string[++i]);
                }
            }
            fprintf(fp_literal, "\n");
            token_count++;
        }
    }
    fclose(fp_literal);

    // --- Final Table Generation (UST) ---
    cout << "\n--- Lexical Analysis Results ---\n";
    cout << "Token\tType\tPointer\n";
    cout << "-------------------------------\n";

    fp_literal    = fopen("D:\\an\\literal.txt",    "r");
    fp_identifier = fopen("D:\\an\\identifier.txt", "r");
    fp_terminal   = fopen("D:\\an\\terminal.txt",   "r");

    // Process Literals
    while (fscanf(fp_literal, "%d %s", &ptr_value, token_value) != EOF) {
        fprintf(fp_ust, "%s\tLIT\t%d\n", token_value, ptr_value);
        printf("%s\tLIT\t%d\n", token_value, ptr_value);
    }
    
    // Process Identifiers
    while (fscanf(fp_identifier, "%d %s", &ptr_value, token_value) != EOF) {
        fprintf(fp_ust, "%s\tIDN\t%d\n", token_value, ptr_value);
        printf("%s\tIDN\t%d\n", token_value, ptr_value);
    }
    
    // Process Terminals
    while (fscanf(fp_terminal, "%d %s", &ptr_value, token_value) != EOF) {
        fprintf(fp_ust, "%s\tTER\t%d\n", token_value, ptr_value);
        printf("%s\tTER\t%d\n", token_value, ptr_value);
    }

    fclose(fp_ust);
    fclose(fp_identifier);
    fclose(fp_terminal);
    fclose(fp_literal);

    cout << "\nUST construction complete.";
    getch();
}

How the Code Works

  1. Input collection: The user types an expression character by character (each cin >> reads one non-whitespace character). The loop stops when $ is entered, which acts as the end-of-input sentinel.
  2. Terminal detection (Pass A): For each character in the string, the program re-opens db.txt and scans it looking for a match. If found, the character and its index are written to terminal.txt. Reopening the file every iteration is inefficient but guarantees scanning from the start each time.
  3. Identifier detection (Pass B): A second scan uses the standard isalpha() function to find letters and writes them to identifier.txt.
  4. Literal detection (Pass C): A third scan uses isdigit() to find numeric characters. Consecutive digits are grouped into a single multi-digit literal (up to 3 digits) by peeking ahead.
  5. UST construction: The program re-opens the three token files and reads them in round-robin order (one literal, one identifier, one terminal per iteration) to interleave the token types in the uniform symbol table, which is written to ust.txt and printed to the console.

Sample Input / Output

 Enter a string:a=b+c*(60/j)+k*80$

 Entered String:a=b+c*(60/j)+k*80
 Tokens Generated

 Uniform symbol table
Token   Type    Pointer
60      LIT     0
a       IDN     0
=       TER     0
80      LIT     1
b       IDN     1
+       TER     1
c       IDN     2
*       TER     2
j       IDN     3
(       TER     3
k       IDN     4
/       TER     4
)       TER     5
+       TER     6
*       TER     7

 Uniform Symbol table constructed

Output Explanation

  1. Input expression: a=b+c*(60/j)+k*80 is tokenised. The three passes independently build the literal, identifier, and terminal lists.
  2. Literals (LIT): The numeric tokens 60 and 80 are grouped correctly as two-digit literals and assigned pointers 0 and 1.
  3. Identifiers (IDN): Single-letter variables a, b, c, j, k are identified by isalpha() and assigned sequential pointers 0–4.
  4. Terminals (TER): Operator symbols =, +, *, (, /, ), +, * are matched against db.txt and listed with their occurrence order.
  5. UST layout: Tokens are interleaved in the table (LIT → IDN → TER → LIT → …) because the output loop reads one entry from each file per iteration. This is the standard uniform symbol table format used by subsequent compiler phases.

See Also

Conclusion

This lexical analyser demonstrates the three fundamental token categories every compiler front-end must handle: literals, identifiers, and operators. The three-pass design (one pass per category) makes the logic easy to follow, though a production scanner would combine all three into a single DFA-driven pass for efficiency. The uniform symbol table produced here feeds directly into the next compiler phase — the parser — which uses token types and values to verify syntactic correctness and drive code generation.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.