Henry Shakeshaft's Website

Reviving the Site

Sun, 14 Jun 2026 19:53:54 +0100

After a (lengthy) hiatus I have decided to revive this site, it’s fun writing, and a good creative outlet; even if I am only yelling into the void!

So stay tuned, more articles will be posted soon, perhaps, maybe :D

Acknowledgements

Tue, 26 May 2026 13:21:51 +0100

This page exists to acknowledge various technologies, software, and other works, which went in to creating this website.

The Website - Hugo

This site is built using hugo, a fast static site generator, which I have grown to love for the simplicity and high customisability. The site, sitemap, and RSS feed are all generated by hugo; if you haven’t tried out Hugo, I would recommend trying it for your next project which requires some form of content-management system, without the overhead that usually entails (esp. if you were considering a WordPress blog).

Colour Scheme - Compline

This website utilises colours from the compline theme, created by Joshua Blais. The light theme view of the site utilises the sister theme, Lauds.

The code blocks are always in a dark theme - I couldn’t find a light theme I could stand (nor did I try investigating how to get hugo to switch this out at run time).

Font - Iosevka

The font used on this website if Iosevka, which is distributed under the OFL-1.1 (SIL Open Font License) licence.

About Me

Mon, 25 May 2026 17:30:47 +0100

I currently work as a Test Engineer. I enjoy programming, particularly low-level C programming, and generally writing about topics which interest me. These include (but are not limited to):

programming/software
technology
philosophy

You can read my writings here, or check out my projects here. This website is also syndicated as an RSS feed if you prefer.

Tyger 3 Evaluation

Tue, 18 Nov 2025 17:12:20 +0000

Finally, we have arrived at the stage where we can actually get something out of our program; out of any program which we write in the language - evaluation.

Evaluation is where the interpreter actually applies meaning to the series of statements which compose our AST; this is how an expression like 5 + 10 actually affect “reality”, i.e. by producing a value from computation.

Now the primary divergence here from the book is that since I am using C, I don’t have the “luxury” of garbage collection; so my plans for initial implementation of the interpreter is to simply malloc new objects, and let it leak until the interpreter exits. I will revisit this when I implement “proper” garbage collection, for now leaking is fine as it allows me to get up and running, and the lifespan of the interpreter will only be short anyway (it is also hard to test garbage collection is properly functional in absence of the evaluation as it requires the presence of objects to free).

Background And Asides

I’m going to cover a few things before I start on the meat of actual evaluation; jump here for the actual process of evaluation.

An Aside on Objects

Throughout the rest of this article - and likely the series - I will make reference to objects; these are not objects in the Object-Oriented-Programming (OOP) sense, merely this is the most effective way of communicating and referring to a type, a data structure, to the reader. I will note however that there are, in some ways, similarities to objects in a language, such as Java, where “primitives” (e.g. integers) are encapsulated within the object’s memory, whereas other objects are more like “Objects” (classes) where there is an additional heap allocation for the stored data (e.g. strings).

Objects are defined as:

typedef struct tyinteger
{
  int64_t value;
} TyInteger;

typedef struct tystring
{
  const char *value;
  size_t len;
} TyString;

typedef union utyobj
{
  TyInteger integer;
  TyString string;
} uTyObj;

typedef struct tyobj
{
  TyObject_Kind kind;
  uTyObj o;
} TyObj;

Again the pattern of a tagged union arises, simply because it is useful; but to reinforce the point raised earlier, we see the TyInteger type encapsulates the value in the structure (i.e. dereferencing a pointer to an instance of this object in the VM will not require a second pointer hop to get the value), whereas the TyString type contains the length but requires a second “pointer hop” to access the underlying data (and hence comes with a second heap allocation). Whilst this isn’t super important now, it is worth covering the memory allocation, and the “management strategy” because, even though the current strategy is just “leak”, when I implement garbage collection, this will have to be covered and dealt with again (and possibly changed once more).

All objects are, however, allocated on the heap, I think the first example of arbitrary heap allocation we’ve had in this project (at least the non-test code). To perform all necessary creation/destruction of objects there are just 2 functions:

TyObj *tyobj_new(TyObject_Kind kind, void *datum);
void tyobj_delete(TyObj *obj)

I won’t dwell on the details as you can read the code to see implementation, but these exist to abstract repeated logic required to construct objects from given data; or in this case datum. I sense that this will be subject to revisiting if/when I add classes to the language but the simple interface of datum may suffice. These are also functions that I will possibly want to rework some more once garbage collection is implemented, which is another argument for centralising the logic here as it makes it so there is less surface area of allocation code.

Results of Evaluation

All evaluations must have some result, the question is what should this be on a fundamental type level? Take for example, the 2 statement types within the language, var statements, and expression statements; what should the result of each be? This is a design decision, as it affects some architecture, and process decisions for the internals of the “virtual machine” (a term I use with respect to this interpreter rather loosely).

Take, for instance, the process of binding (read assigning) a variable, var x = 10; - should this return a value? I would argue no, there is no result of said statement other than to mutate internal state of the VM, there is no type. This is where I differ, in aesthetic standards, from say JavaScript where the equivalent - let x = 10; - would yield a undefined. This isn’t to imply that there is no effect here, obviously an Object needs to be allocated somewhere, and this value associated with the ident. Expression statements however do return values, they may also mutate some internal state of the VM, but these do produce values. For example, evaluating an ident - x; yielding 10 in this case - or the result of an arithmetic expression - such as 10 + 5; yielding 15 - should produce a value, and return these either for other variables to be bound to, or for human interpretation of a result. It is worth noting that this is the reason for the TYOBJ_NONE enumerant, for cases where the VM performs an operation which doesn’t “produce” a value.

Environments

The first major component of the VM is the concept of “Environments”, defined as a collection of variables(/constants - when implemented) with a defined scope; the actual scoping of environments will be done later, but at time of writing there is no support for nested blocks, nor functions, there is only the “global” scope (i.e. top level to the interpreter).

Environments are implemented as a hash map, where the hash of a given key (the ident of a variable) is the result of a 32-bit FNV-1a hash function modulo the [defined] capacity of the hash map; moduloing the hash clamps the result to a defined range of entries/slots, i.e. assume the hash result of "foo" is 11736239235, if we apply %16 (the capacity of an example environment) to said hash, then we are gaurenteed the hashed key always is in the range 0 <= x <= 15, in this case the value is 3; this constrains our objects to a set of slots which can be allocated up front. In general, our algorithm for generating hashes looks like the following:

key := "foo"
hash := fnv1a(key)           -- generate initial value
hash := hash % env.capacity  -- clamp to defined range of valid slots

Evironments are defined as follows:

typedef struct tyenv_var TyEnv_Var;
struct tyenv_var
{
  const char *ident;
  size_t ident_len;
  TyObj *object;
  TyEnv_Var *next;
};

typedef struct tyenv
{
  TyEnv_Var *vars;
  size_t capacity;
} TyEnv;

I.e. each environment is an array of TyEnv_Vars, which are themselves defined as a singly-linked list node; the reason for the linked list nature of each hashmap entry is this allows for collision resolution later, i.e. the case where 2 keys (idents) have the same hash generated - asssuming the hash is generated according to the earlier defined algorithm. Capacity is a defined member here for a few reasons:

it allows for preallocation of a defined number of slots which can be written to (indicating our variables)
it allows for consistent clamping of the results of hashing to a defined range between all functions that accept the environment as an argument
it allows for a user (or me the developer) to specify different amounts of slots on a per instance basis

The last of these points was particularly useful for testing collisions, but I forsee if as potentially useful for when implementing functions - it seems sensible to me that a function’s environment should be smaller than that of the global environment, for instance. Due to the fact that each environment is backed by a dynamically defined array, then each environment needs to be explicitly created and destroyed:

// snippet taken from the REPL
{
  TyEnv global;
  tyenv_init(&global, 16);
  while (true)
  {
    // ... parse code and do things with the environment
  }
  tyenv_free(&global);
}

What options should we be able to perform on our environment (referred to as hash maps from here on)? We need to be able to:

insert new key/value pairs,
update existing keys,
and delete keys

A pretty simple set of operations.

Inserting new key/value pairs is a fairly simple process, all we need to do is initially generate our hash value, perform a lookup to see if our key conflicts with existing entries, and if so resolve. In pseudocode this looks something like¹:

def env_do_insert(var: *TyEnv_Var, key: str, obj: TyObj) -> bool:
    if var.key == key:
        # handle case where key exists
    else:
        if var.next:
            return env_do_insert(var.next, key, obj)
        else:
            var.next = env_new_var(key, obj)
            return True

def env_insert(env: *TyEnv, key: str, obj: TyObj) -> bool:
    key_hash: u32 = fnv1a(key) % env.capacity
    exists: *TyObj = env_get(env, key_hash)
    if exists:
        if key == exists.key:
            # handle case where key exists
        else:
            return env_do_insert(exists, key, obj)
    else:
        env.elems[key_hash].key = key
        env.elems[key_hash].obj = obj
    return True

Here we see that key insertion consists of 2 functions, a pattern I implement across all hash map related functions:

env_insert
1. hashes the key
2. checks whether the map already contains an entry at the given [hash] slot
3. if not then insert the object here
4. if it does exist, check if the keys match
5. if keys match, handle this*
6. else call the function to perform the insert
env_do_insert
1. compare if keys match
2. if they match, handle this*
3. else check if there is a “next” value in the linked list
4. if there is a “next” value then recurse into this function, passing next
5. otherwise create a new node in the list

*R.e. the question of what to do in cases where insertion hits a key which already exists, I decided to handle this as an “in-place” update and not throw an error; I decided that throwing errors in relation to this should be handled at a higher level, i.e. in actual evaluation rather than here; although, as I write, I may reconsider this in a planned refactor that’s coming for this whole system¹.

The pattern for updating keys is very similar to this pattern, the main difference here is if no stored object is found in the derived slot for the ident, or at in the linked list of entries, then we return an error:

def env_do_update(var: *TyEnv_Var, key: str, obj: TyObj) -> bool:
    if var.key == key:
        var.obj = obj
    else:
        if var.next:
            return env_do_update(var.next, key, obj)
        else:
            return False

def env_update(env: *TyEnv, key: str, obj: TyObj) -> bool:
    key_hash: u32 = fnv1a(key) % env.capacity
    exists: *TyObj = env_get(env, key)
    if exists:
        if key == exists.key:
            exists.obj = obj
            return True
        else:
            return env_do_update(exists, key, obj)
    else:
        return False;

In this case, “updates” where the key is not found should be considered a failure, as this is a failure in the operation on the type rather than a semantic thing in the language VM, albeit the same behaviour we would expect [from the language]. One thing I should make a callout to here as well is that there are no language semantics within Tyger to facilitate updates, however I figured I should implement it in preparation of that eventual feature. The same holds true for deletion of objects from the map, i.e. no language level support for it (and this doesn’t happen during garbage collection as there is none), but implementing this now will be useful later (even if I plan to refactor this all anyway).

The only other operation I think which is interesting to cover is retrieving (“getting”) values from the map:

Retrieving a value from the map is simple, again:

def env_do_get(var: *TyEnv_Var, key: str) -> *TyEnv_Var:
    if var.key == key:
        return var
    else:
        if var.next:
            return env_do_get(var.next, key)
        else:
            return nil

def env_get(env: *TyEnv, key: str) -> *TyEnv_Var:
    key_hash: u32 = fnv1a(key) % env.capacity
    exists: *TyEnv_Var = nil
    if env.elems[key_hash].key:
        if key == env.elems[key_hash].key:
            exists = env.elems[key_hash]
        else:
            exists = env_do_get(env.elems[key_hash], key)
        return exists
    else:
        return exists

generate the hash
check if the var exists in any of the slots
if so return
else, recursively call do_get until either the code reaches the end of the linked list, or finds a matching key

In this case, our “invalid” lookup case returns a null pointer, as this is simple to check for, and given how much weird out-parameter stuff I’ve already had to do in this project, I would rather reframe from returning a success parameter - and correctly overwriting the out parameter - and take the “risk” of null pointer dereference (which will at least crash my program, is obvious, and will not potentially yield weird, alternative errors later)².

Deleting values is a little more tricky as, like insertions, I decided to make it require 2 distinct code paths to implement, depending on whether the key occupies a slot, or is in the linked list, and also some careful re-jigging of the next node pointers needs to be performed.

def env_do_delete(var: *TyEnv_Var, key: str) -> bool:
    next_node: *TyEnv_Var = nil
    if var.key == key:
        next_node = var.next
        tyenv_var_delete(var)
        if next_node:
            var = next_node
        return True
    else:
        if var.next:
            return env_do_delete(var.next, key)
        else:
            return False

def env_delete(env: *TyEnv, key: str) -> bool:
    key_hash: u32 = fnv1a(key) % env.capacity
    exists: *TyEnv_Var = env_get(env, key)
    if exists:
        if exists.key == key:
            next_node: *TyEnv_Var = exists.next
            # NOTE: assume this function exists and will handle all cleanup "in place"
            tyenv_var_inplace_delete(exists)
            if next_node:
                exists = next_node
            return True
        else:
            return env_do_delete(exists, key)
    else:
        return False

In essence, if we need to delete a node which contains a next pointer, then we need to take the reference to this, delete the current object (TyEnv_Var), then reassign the current instance to point to the previous reference; obviously if next was null then no update needs to happen.

Evaluation And Object Creation

Firstly let’s cover evaluation of expressions. The previous installment in the series covered the creation of the AST, so you should hopefully remember that expressions are simply things which produce values; whilst there are several types of expression I will zoom in on the 2 easiest to process, which are literals. As shown earlier there are 2 builtin types currently supported, those being integers and strings.

How Eval Works

Before we touch that, I should state that the API for interfacing with the actual evaluation engine in the VM is defined as a single function:

TyObj eval(TyEnv *global_env, const Program *prog);

A single function, which will always yield some kind of object (even if its type is NONE, i.e. not an object). Currently there is a question over whether I should return a pointer to the object, rather than dereference it within eval, if I change my opinion here - which is not strongly held, or informed - then I will write some notes on why. This however, isn’t the main workhorse, this immediately calls another function which will iterate over all statements in the parsed program and repeatedly calls another eval function on each:

static TyObj *eval_program_statements(TyEnv *env, const Program *prog)
{
  TyObj *res = NULL;

  for (size_t i = 0; i < prog->statements.len; ++i)
  {
    const Statement *stmt = &(prog->statements.elems[i]);
    const Parser_Context *ctx = &(prog->context);
    res = eval_statement(env, stmt, ctx);
  }

  return res;
}

As mentioned, there are only 2 statements, those being var and expression; we will focus first on evaluating expressions (since this is the part which I think is coolest as we actually get to see things happen).

Expression Evaluation

All expression evaluation starts first, by retrieving a pointer to the expression we want to evaluate, by converting a handle (see the previous entry to refresh why this is needed):

static TyObj *eval_expression_statement(TyEnv *env, Expression_Handle handle, const Parser_Context *ctx)
{
  TyObj *res;
  const Expression *expr = &(ctx->expressions.elems[handle]);
  res = eval_expression(env, expr, ctx);
  return res;
}

Integer Evaluation

Within eval_expression, the relevant function is called depending on the type of expression parsed; for our first check we’ll look at integers (EXPR_INT):

static TyObj *eval_int(const Int_Expression *expr)
{
  TyObj *res = tyobj_new(TYOBJ_INT, (void*) &expr->value);
  return res;
}

This is quite simple, a call to tyobj_new is made where we specify that we’re constructing an integer, and just pass the underlying value straight from the AST.

String Evaluation

Strings are a little more involved in the requisite code:

static TyObj *eval_string(const String_Expression *expr, const Parser_Context *ctx)
{
  char *buffer = malloc(sizeof(char) * (expr->len + 1));
  assert(buffer && "failed to malloc string memory");

  const char *string = &(ctx->strings.elems[expr->string_handle]);
  strncpy(buffer, string, expr->len);
  buffer[expr->len] = '\0';

  TyObj *res = malloc(sizeof(TyObj));
  assert(res && "failed to allocate space for [TyObj::String]");
  res->kind = TYOBJ_STRING;
  res->o.string = (TyString) {
    .value = buffer,
    .len = expr->len,
  };

  return res;
}

You will notice my flagrant disregard for my own new/delete functions here; I honestly don’t know why it was implemented this way, and is on the list of things to be fixed in a cleanup pass after this entry in the series. To summarise, I think that I need to utilise some “string view” type (as I have already defined) to pass in as the datum argument to tyobj_new, which can then perform the allocation, and copy from a defined, sized string.

Evaluating Infix Expressions

With these basics covered, what we could consider “leaves” to our interpreter (because they are terminal points, i.e. no further evaluation can be done), we can move to evaluating infix epxressions (and hopefully make the mind-melting data processing in recursive descent parsing worth it!) All calls to eval_infix perform indirect recursion as they themselves call eval_expression on the left, and right hand side of tree, which may call eval_infix and repeat arbitrarily many times:

static TyObj *eval_infix(TyEnv *env, const Infix_Expression *expr, const Parser_Context *ctx)
{
  TyObj *res = NULL;

  const Expression *lhs = &(ctx->expressions.elems[expr->lhs]);
  const Expression *rhs = &(ctx->expressions.elems[expr->rhs]);

  TyObj *lhs_res = eval_expression(env, lhs, ctx);
  TyObj *rhs_res = eval_expression(env, rhs, ctx);
  assert(eval__do_op(&res, expr->op, lhs_res, rhs_res));

  return res;
}

So eventually the left, and right hand sides will reach some “leaf” node, and terminate (i.e. they will encounter a string or a integer), and subsequently this will return up the call stack. Take for example, 1 + 2 - the left hand side of this evaluates to 1, and the right to 2, subsequently the add opp is performed on these 2 operands (and will eventually yield 3). This will also correctly handle more complex expressions such as 1 + 2 + 3 (a very simple but illustrative example) as:

the following sub-expressions are defined
1. left hand side: 1 + 2
2. right hand side: $LHS + 3
the left hand side is evaluated first:
1. this calls eval_expression and encounters an infix
2. the LHS and RHS are both integers
3. after evaluation these are passed to the op function
4. the 2 values are summed to 3 and this new object is returned
the right hand side is then evaluated
1. since this is an integer only this will be returned
another call to do_op is performed on the result of left hand side evaluation and the right hand side
a new object with integer value 6 is generated
this is returned, evaluation has finished

Now I know I mentioned not wanting to deal with out parameters earlier, this however seems like a better option for handling operator evaluation since I have to allocate a new object somewhere anywhere, and rather than require all do_op functions have to allocate an object, I decided it would be better to allocate once and pass this to be later modified. Again, string concatenation isn’t supported and the only other type, integers, are stored “inplace” in the object, so I can just update these values later. This may change again once I start implementing garbage collection, and add other types, as my hacky work around may no longer be applicable.

Speaking of how operators are handled:

static int eval__do_op(TyObj **out, Operator op, const TyObj *lhs, const TyObj *rhs)
{
  int success = 0;

  (*out) = tyobj_new(TYOBJ_NONE, NULL);
  assert((*out) && "failed to create new object");

  if (lhs->kind == rhs->kind)
  {
    switch (op)
    {
    case OP_PLUS:     { success = eval__do_add((*out), lhs, rhs); } break;
    case OP_MINUS:    { success = eval__do_sub((*out), lhs, rhs); } break;
    case OP_ASTERISK: { success = eval__do_mul((*out), lhs, rhs); } break;
    case OP_SLASH:    { success = eval__do_div((*out), lhs, rhs); } break;
    default: { assert(0 && "unreachable"); }
    }
  }
  else
  {
    return success;
  }
  return success;
}

This VM requires that in all cases, the types of each operand must match, i.e. if one operand is an integer, both must be integers - this is subject to change as logic requirements become more complex, but this is serviceable for now. Each supported operator calls an interpreter specific function to perform the neccessary logic; one thing should be noted which is arithmetic operator calls on non-integer types will yield an error (i.e. there is no string concatenation for "foo" + "|" + "bar").

NOTE: The gif shows the evaluation result is 7, rather than 6.2, this is because the language only supportes integer division; this is the way Python 2 worked, and will be changed once support for floats is added (where the result of integer division will be a float if there is a modulo is non-zero)

Call Expressions

I will lead by noting that my implementation is the most scuffed one could make it for such a simple case, and will need revisiting as soon as I expand call expression support beyond the builtin println:

static void eval_function_builtin(TyEnv *env, const Argument_List *args, const Parser_Context *ctx)
{
  // NOTE(HS): all println handling
  {
    TyObj *evaluated_args = malloc(sizeof(TyObj) * args->len);
    for (size_t i = 0; i < args->len; ++i)
    {
      const Expression *expr = &(args->elems[i]);
      TyObj *o = eval_expression(env, expr, ctx);
      memcpy(&(evaluated_args[i]), o, sizeof(TyObj));
    }

    for (size_t i = 0; i < args->len; ++i)
    {
      TyObj *obj = &(evaluated_args[i]);
      switch (obj->kind)
      {
      case TYOBJ_INT: { printf("%" PRIi64 "", obj->o.integer.value); } break;
      case TYOBJ_STRING: { printf("%s", obj->o.string.value); } break;
      case TYOBJ_NONE: { printf("<NONE>"); } break;
      }
      tyobj_delete(obj);
      if (i + 1 < args->len) { printf(", "); }
    }
    free(evaluated_args);
    printf("\n");
  }
}

static TyObj *eval_call_expression(TyEnv *env, const Call_Expression *expr, const Parser_Context *ctx)
{
  TyObj *res = tyobj_new(TYOBJ_NONE, NULL);

  const Expression *function = &(ctx->expressions.elems[expr->function]);
  assert(function->kind == EXPR_IDENT);
  const Ident_Expression *ie = &(function->expression.ident_expression);
  const char *ident = &(ctx->evaluated_identifiers.elems[ie->ident_handle]);

  assert(compare_function_names_eq("println", ident));
  eval_function_builtin(env, &expr->args, ctx);

  return res;
}

Println is analagous to the print function from Python except I do not allow things like overriding the line ending (so you only get newline termination).

How this works is the current function is checked to be println (and if not, the interpreter crashes - elegantly… not!), then call eval_function_builtin on the function, which:

preallocates a buffer of objects which hold the evaluation result of each arg the function is called with
loop over all the args the function was called with
evaluate each argument, and copy the resultant object into the buffer
loop over the buffer of objects
serialise the object to a string
print the string, then print a comma (assuming the arg was not the last one)
delete the object from the buffer
free the buffer
print the final newline

I’ve separated builtins out for now, as these are things which are builtin to the interpreter and require some different handling compared to a user defined function.

Among the obvious (like the very much hard coded to only support println) that I could improve on, I think that:

I need to find some alternative way of handling object evaluation for each arg
- I don’t like the preallocation of some object buffer which then has separate tyobj_delete calls for each entry in the buffer
- I’m sure something else must be better - maybe a candidate for object pooling where some permanent set of objects is always present to be overridden
- only problem there is a related clear function probably needs to exist to ensure no leaks
create an object to_string function to handle to serialisation of strings to be printed
it would be nice to serialise 1 string, then print rather than printf each arg and delimiter
- could be a case for my string_builder from the parer trace code

With that point about leaking, it is worth noting that this current implementation leaks anyway, as the call to tyobj_new calls malloc and is not freed; see, it gets hard to manage the lifetime soup beyond the basics (there’s even a leak present in expression evaluation)!

I guess now would also be a good time to clarify that the convention I will work with with respect to order of argument evaluation, if it wasn’t apparent from the code this is left to right, i.e. if you pass 5 args to a function, arg 1 (index 0) will always be evaluated first, then 2, 3, 4, and finally 5.

Wiring It All Together

Now we’ve covered most of what’s important, so the only thing is to “complete” the program and get this all working as part of the interpreter REPL so I can start “using” it. Fortunately, like pretty much all operations involved in extending the REPL it’s a fairly small change:

void repl_run(void)
{
  TyEnv global;
  tyenv_init(&global, 16);

  while (true)
  {
    // ...
    Program program = parser_parse_program(&parser);
    TyObj res = eval(&global, &program);
    tyobj_inspect(&res);
    // ...
  }

  tyenv_free(&global);
}

Improvements And Next Steps

I’ve not quite finished even this “first pass”, and there’s a lot more that needs doing before I’m comfortable with “finishing” this project; in no particular order here’s a list of improvements and next steps in the project.

Refactor My Hash Map Implementation

As noted¹, I am not happy with my current implementation, however I think that I can refactor this to be significantly better, and more feature complete now I’ve walked through what the actual algorithms for each operation should look like. One thing for example, that wasn’t any good was my deletion code. I attempted to be “lazy” and only implement only a single memory free path, rather than do as I did for insertion. This ended up biting me on my initial attempt as I passed stack memory to the nested free call and this blew up my program in a very non-helpful way.

Refactor Tagged Union Member

There are a few differet ways the union member of the project’s tagged unions are labelled, and I think from a semantic view this could be improved across the code base; the improvement is to renanme things according to the “as pattern” (for lack of a better description). This “pattern” is just a naming convention for naming the union member of a tagged union such that the code reads <tagged union>.as.<unioned type>, for example, the TyObj would look like:

typedef struct tyobj
{
  TyObj_Kind kind;
  union {
    TyInteger integer;
    TyString  string;
  } as;
} TyObj

In this case all usage of this in code will look something like:

TyObj *obj = env_get(env, "foo");
switch (obj->kind)
{
  // ...
  case TYOBJ_INT:
  {
    printf("object is integer with value " PRIi64 "\n", obj->as.integer);
  } break;
  // ...
}

I personally think this is a nicer, and clearer way of semantically describing what is being done in code; I took advice for this after reading through this Git repo and seeing the naming scheme implemented.

Actual Error Handling

Now whilst I don’t have plans for language level ways of handling errors, what I want is to actually implement a good system for reporting errors, both those generated from parsing, and also those which arise from runtime errors. The infrastructure is mostly there, I just need to actually do something with it.

One big thing that would be an improvement is to actually use the location struct generated from lexing, and include this in all parser node info, then print out debug info like the “file” (which will be more useful once this evolves beyond a REPL), the line, and column position of the error. For example, assume the following:

var x = 10 + "foo";

I would expect that the error states something about invalid operands to call to + operator, then tell me the fact the types are mismatched, what the types of each operand are, and the solution (in this case, arithmetic operators can only be called on 2 numeric types).

Something like the following:

NULL:1 TypeError: mismatched types of operands for operator(+)
    var x = 10 + "foo";
    10: integer
    "foo": string
    operator(+): strings cannot be concatenated to integers
    Remedy: both operands must be of the same type

And more aspirational:

NULL:1 TypeError: mismatched types of operands for operator(+)
var x = 10 + "foo"
        ^  ^ ^
        |  | |_ type: string
        |  |
        |  |_ operator: +
        |
        |_ type: integer
    
    Cannot apply operator(+) to operands <integer> and <string>.
    Both operands must be of the same type.

More General Evaluation of Call Expressions

I wonder what a better way of handling functions would be; I have been playing with the idea of creating some wrapper function for environments, which would set up the gloval environment, then register the builtin functions to said environment. This means that if I decide to reimplement environments such that reinserting an existing key fails, then I can also prevent users from overriding these functions (and I that if I make the “global”, then these will be available from all function and other scopes).

Richer Built-In Types And More Language Constructs

So far there are 2 builtin types, string and integer, and only 2 statement types;

The set of additional types I want to add support for include:

floating point numbers (like integers I will probably only provide _double_s)
nil type - i.e. null pointers (they’re just useful)
functions - i.e. user defined functions
booleans

Additional language constructs:

return statements - return a value, when done at “top level” then this should exit the program
if/else control flow
loops - for and while
boolean logic
prefix operators (! and -)
improved type operations
- I think that a + b should be successful where a & b are numbers (int or float)
- I think that a + b should be successful where a & b are strings

This should make the langauge much more useful, and, for the most part, consist of adding a few extra cases in most places in the interpreter.

VM & Garbage Collection

So I should probably actually create some concept of a VM where all state for the interpreter can be stored, and also I need to get to working on garbage collection, and more generally clean up the various memory leak issues (besides the intended ones). I think I’ll implement the mark and sweep algorithm, mainly as this will take care of circular references (which are a problem in simpler reference counting methods).

Evaluation of Files

As it stands, the language and project are REPL only, and don’t support user ability to run files; support for this will be relatively trivial to implement but I left it as the REPL is a good enough general solution, and allows for quick testing of new features.

General Asides

Some general points which don’t fit well elsewhere, but I think are worth calling out.

Discovering Old Bugs: Why Tooling Matters

I found a very fun bug in my parsing whilst playing around with my interpreter, namely that the following statement would always evaluate (“successfully”) to 1, rather than the expected 2:

var x = 1 + 1;

As it turns out there was a bug in my parsing code, specifically the order in which I get a new expression handle to copy the parsed epxressions into after parsing the value being assigned in a var statement:

Tyger_Error parse_var_statement(Parser *p, Parser_Context *ctx, Statement *stmt)
{
  //..
  parser_next_token(p);

  Expression_Handle expression_handle = va_array_next_handle(ctx->expressions);
  Expression expr;
  err = parse_expression(p, ctx, &expr, PRECIDENCE_LOWEST);
  if (err.kind != TYERR_NONE)
  {
    return err;
  }

  va_array_append(ctx->expressions, expr);
  // ...
}

The fix for this was as simple as moving the call to va_array_next_handle to just before the call to append. Simply this is because the va_array_next_handle macro doesn’t increment the length of the array, because of this, the handle I “generated” was overriden during the subsequent call to parse_expression.

This should serve as a good demonstration as to why, even with well thought out tests, onse should engage in as much exploratory testing of a system as possible, simply because you have almost certainly missed stuff. It should also be worth reiterating that you should never assume your code is correct just because the tests are all green - this should increase your confidence, but don’t let it blind you. It’s also a good idea to have some form of immediate, more easily digestable form of feedback whenever you’re doing testing, it would have been alot of work stepping over, and re-stepping over many lines of code in the debugger if I didn’t have my “trace” system for parse trees as previously implemented. Yes I could have spotted this in the debugger, but since the “trace” module just shows you the parsed AST this helped narrow down that it was a problem in how expressions were parsed, and probably confined to var statement parsing (as things like this in lone infix expression tests were correct, and also 1 + 1 on its own just worked in the REPL).

There are new tests in the parser which cover this case, at least to a basic level, now so this shouldn’t rear its head again (and will be caught if it ever does).

GTest Fixtures - Actually Really Useful

I don’t know why I haven’t been using these the whole time! Seriously, these are a useful construct within the test framework and I don’t know why I didn’t start using them, especially during my parser tests. One of the many things I did, which may not be the best use, is just to set up some protected members within the class, and some util methods for setting up a new program to evaluate, and all the other associated boilerplate; also a few associated helper methods for running some assertions.

This is the loose scaffold of what I came up with:

class TestEval : public ::testing::Test
{
protected:
  Lexer l;
  Parser parser;
  Program prog;
  TyEnv env;

  void setup_test(const char *input)
  {
    lexer_init(&this->l, input);
    parser_init(&this->parser, &this->l);
    this->prog = parser_parse_program(&this->parser);
    tyenv_init(&this->env, 16);
  }

  void teardown_test()
  {
    program_free(&this->prog);
    tyenv_free(&this->env);
  }

  // ...
};

tests/test_eval.cpp

To utilise these effectively, I would recommend checking out the docs, but to summarise you need to extend the GTest testing::Test class, then create whatever shared data/methods you want. My implementation makes it very simple to run a series of tests without needing to do all the setup and teardown code shown here:

TEST_F(TestEval, Test_Eval_Int_Expression)
{
  // ... setting up test cases
  for (auto& tc : test_cases)
  {
    setup_test(tc.input);
    DEFER({ teardown_test(); });
    TyObj act = eval(&this->env, &this->prog);
    assert_object_value_eq(act, tc.exp);
  }
}

Much cleaner, and I only need to implement the same code once, I may have to go refactor my parser tests to do something similar now I’ve had a chance to experiment.

Test Your Data Structures/Algorithms

I mentioned that I build the hash map twice; the first time was a disaster as my tests for it were all done at the language level, rather than being sensible and writing separate tests for the map. The first attempt was essentially vibe coded without the AI - so maybe that’s where I went wrong! - but really, I think this has been a lesson in how to actually develop things like this, i.e. data structures; I “understood” how hash maps worked in theory, but clearly I either didn’t or didn’t translate that well to the inital code. Anyway, even though I’ll be refactoring again, the current code at least works, and isn’t nightmarish to integrate.

The Curse Of Your Prior Self

Yourself 2 weeks ago really isn’t a bad guy, even though he seems it when you’re dealing with his code and terrible decisions. Once again, as mentioned in the previous post, try not to leave projects to rot, it’s more painful to come back and pick it up even if it’s painful in the now. Also I have come to appreciate a take I’ve seen echoed by a few people, just don’t leave todos in your code (there is a great irony in this project here but bear with me), normally it’s something simple and you’re just being lazy, and also future you probably won’t know what you meant. He may gleam it, but he may also fail, either way it’s a lot of work and you’re probably in a better position to take 10 minutes to deal with it than he is.

The example I will raise from my “finishing” this project is just issue #11, where simply I stumbled across an old commend in my handle converter functions stating

// TODO(HS): need to update functions here to accept the raw dynamic array which
// can then be operated on (otherwise we get problems).

I have no idea what I was thinking here, I think I meant that I should just accept the minimum possible amount of data to work on, so as to minimise potential side effects, but like I say this is past me communicating something he actually knew with no real context because “I’m sure he’ll understand”. We do complain about third party code in programming, but unfortunately the worst provider of third party code is your past self, because he never suspects you will not just know what he meant.

Epilogue

That’s it, the base of the interpreter is done, we can actually evaluate (basic) programs! I’m reasonably happy so far with this, but a lot more work is required. I’ll be posting less frequent updates, mainly covering things as I actually implement stuff I think is worth covering, such as garbage collection, or control flow (particularly loops as this might be worth showing).

But until then I hope you’ve enjoyed reading this, and maybe this has helped inspire your own journey. You can check out the repo as it stands at time of writing here, download it, build and try it out, and remember to have fun!

I’m not totally happy with how this was implemented anyway; it was my first time implementing a linked list, and second attempt at a hash map, and I think I can do a better job rebuilding now I’ve (a) done it once, and (b) have some examples of usage. A read of the code at this stage should demonstrate why I’m not thrilled with my own work here. ↩︎ ↩︎ ↩︎
The out parameter manipulation performed whilst parsing was primarily due to other constraints I had placed on myself and the system, rather than it being an objectively good choice, hence why I reframe from reaching for said hammer again. ↩︎

Tyger 2 Parsing

Thu, 12 Jun 2025 19:22:41 +0100

We left off last time with a (mostly) functioning lexer; as developments laid out here revealed I left a few things unimplemented which should have been, for example the lexer didn’t support commas, which made for a fun bug discovery later (when parsing function calls). But that’s us getting ahead of ourselves, what we’re concerned with now is the act of parsing the lexed tokens into something with meaning; i.e. generating an AST (Abstract Syntax Tree).

If you’re like me and have taken a few stabs at implementing a language, then you may have found the topic of parsing complex; some of this is due to the diverse set of methods you can approach this with. Many examples I’ve seen all “default” to using parser generators, which I’m sure are fine and have their place in the space, I however think implementing things yourself is quite useful (and I’ve never liked the hand-waving response of “just do this and don’t worry about the details”). So hopefully this works as a good explanation of Pratt Parsing, and also shows how one may want to design a solution to the problem in C.

The Structure of A Tyger Program

Programs are a collection of 1 or more statements. Each statement is one of either a variable declaration statement, or an expression statement. Expression statements are, to be pithy, statements wrapping expressions; these only really exist as the following examples are valid statements within the language:

"Hellope!";
1015;
5 + 4 - 2 * 3 / 1;

With that said, what is an expression in this language? In short, anything which produces a value. To be more explicit, integers, strings, function calls, and operations containing some infix operator (+ - * /) are all expressions. Now as noted in part 1, this is a radically pared down version of what I want the final language to be, so undoubtedly the range of available kinds of statements and expressions will grow to meet the changing scope of the language.

This however is good enough to begin the process of parsing our lexical output into an AST, which itself will later be evaluated.

Setting Up The Parser

In order to parse programs, there needs to be some data structure, much like for the lexer. Our parser, again, is derived from the book, and as such looks (near) identical:

typedef struct parser
{
  Lexer *lexer;      // an instance of the lexer
  Token cur_token;   // the token the parser is currently inspecting
  Token peek_token;  // the next token the parser will look at
} Parser;

The extent of the “public” API to use the parser is then the following 2 functions:

void parser_init(Parser *p, Lexer *lx);    // performs correct initialisation of the parser
Program parser_parse_program(Parser *p);   // generates our "program" struct as a result of parsing

These should be self-explanatory, and only the internals obscured (they aren’t that interesting at the moment anyway). The Program struct returned by our function is defined as

typedef struct program
{
  Statement_VaArray statements;
  Error_VaArray errors;
  Parser_Context context;
} Program;

This struct comprises the following members:

2 dynamic arrays, 1 for the program statements and another for the errors
some Context object; for now this can be treated as entirely opaque as it’s not yet important.

A fairly simple series of data structures, now how the parser transforms the input into the final program is through the following steps:

Check if Parser::cur_token is EOF
If so, we have generated our program
Else, until EOF is reached repeatedly call parse_statement
1. Once statement is parsed, if any errors encountered, add them to add errors array
2. Else add the statement to the array of statements

And parsing a statement comprises:

Match Parser::cur_token
If kind is TK_VAR (i.e. the var keyword) parse_var_statement
Else, for all other token kinds, attempt to parse_expression_statement

All relatively straightforward, at least from a high level. But before covering Pratt Parsing, I will quickly detour to talk about some useful testing and debugging features I implemented for myself.

Testing and Debugging

Printing The AST

It’s all well and good saying that after parser_parse_program is called there’s now a program there; it is however much harder (again without running it) to be able to verify the correctness of the parse result (i.e. the shape of the AST) in absence of just inspecting memory addresses in a debugger. This is impractical as soon as the program become more complex than 10;, so a better solution is required.

As a result, I decided to take a(nother) idea from the book and write a little internal utility for me to transform any given (parsed) program into a string form, containing the nuances of the parsed result (such as operator precedence) and then use this generated string in my comparison functions. This in particular proved useful for rigorous operator precedence testing, and for comparisons of items such as idents.

The API I came up with can be found under the trace.h file; this is relatively small, consisting of a single function call and an enum representing the output format:

typedef enum trace_format
{
  TRACE_YAML,
  TRACE_SEXPR,
} Trace_Format;

const char *program_to_string(const Program *p, Trace_Format format);

The 2 formats supported are YAML and SEXPR, the former should be obvious, but the latter is meant to be like an S Expression. For example, the following table shows how some different programs may be converted into the following S-expression syntax

Program	S-Expression
`var x = 10;`	`(var x 10)`
`"Hello, World!";`	`("Hello, World!")`
`10;`	`(10)`
`1 + 4;`	`((+ 1 4))`
`10 + 5 - 4 * 3 / 2;`	`((- (+ 10 5) (/ (* 4 3) 2)))`

The last 2 rows look a little strange with the extra parentheses, this is mainly an artifact of not being able to find a more elegant solution to guaranteeing that representations of s-expression style infix expressions were always surrounded by parentheses (particularly when either nested or appearing as a function call arg).

It should be evident that, particularly in the case of the last, that this is better than just writing the individual, unrolled assertions (particularly in a generic way which can be “table driven” for testing). The way this would look, for the 4th option would be

TEST(Parser_Tests, Parse_Infix_Expression)
{
    // input: "1 + 4;"
    // setup for generating the program here
    Expression *expr = prog.statement.expression_statement.expression;
    ASSERT_EQ(expr->kind, EXPR_INFIX);
    ASSERT_EQ(expr->lhs->kind, EXPR_INT);
    ASSERT_EQ(expr->lhs->int_expression.value, 1);
    ASSERT_EQ(expr->rhs->kind, EXPR_INT);
    ASSERT_EQ(expr->lhs->int_expression.value, 4);
}

It should probably be noted, that the expanded version of the final table exaple, when represented as a tree, is

Imagine writing all the asserts for that!

The YAML output format is a more structured format, and also includes some additional debug information per node in the AST. Take the simple example of

var x = 10;

This is transformed into

---
Errors: 0
Statements: 1
---
program:
    - kind: VAR
      ident: x
      expression:
          - kind: INT_LITERAL
            value: 10

And the statement var foo = 1 + 2 - 3 * 4 / 5; becomes

---
Errors: 0
Statements: 1
---
program:
    - kind: VAR
      ident: foo
      expression:
          - kind: INFIX
            op: "-"
            lhs:
                - kind: INFIX
                  op: "+"
                  lhs:
                      - kind: INT_LITERAL
                        value: 1
                  rhs:
                      - kind: INT_LITERAL
                        value: 2
            rhs:
                - kind: INFIX
                  op: "/"
                  lhs:
                      - kind: INFIX
                        op: "*"
                        lhs:
                            - kind: INT_LITERAL
                              value: 3
                        rhs:
                            - kind: INT_LITERAL
                              value: 4
                  rhs:
                      - kind: INT_LITERAL
                        value: 5

Testing Utilities

One thing I thought would be useful in my C++ code, as it is in other languages I have used, is a defer keyword. In languages such as Go, there exists a keyword defer which basically means “run this statement/block later”; this doesn’t exist in C++, however this can be implemented using some “more modern” (C++11) features.

Specifically, defer can be implemented using lambdas, and embracing C++’s deterministic object lifetimes (i.e. RAII). Now I should note that the implementation I used, is cloned from a blog post¹ by one of the developers from the game The Witness.

To start, C++11 added support for lambdas, which allows you to define local functions within larger functions (for example), or even pass them to as parameters to functions. To leverage this in testing, it is possible to create a (template) class, have this take in a lambda as a constructor parameter, then call said lambda in the destructor:

template <typename F>
struct ScopeExit
{
  ScopeExit(F t_f) : f(t_f) {}
  ~ScopeExit() { f(); }
  F f;
};

So this works something like

void log(int level, const char *message)
{
  auto buffer = new char[256];
  ScopeExit([=]() { delete[] buffer; });  // object constructed here, with lambda
  if (level < 30) {
    return;
  }  // if `level < 30` then ~ScopeExit() runs, and the lambda
  int bytes = std::snprintf(NULL, 0, "[INFO] %s\n", message);
  std::snprintf(buffer, bytes + 1, "[INFO] %s\n", message);
  buffer[bytes] = '\0';
  std::cout << buffer;
} // ~ScopeExit() runs here, and the lambda

As can be seen in this example, there are 2 different exit points, after a conditional fires and at the end of the “happy path”. Without ScopeExit, you have to remember to put the corresponding delete[] call in 2 different places, and the more places you need to cover, the more likely you miss something. You can also get more complex “teardown” processes, this is partly why RAII became desired as a language feature; it provides deterministic lifetimes for objects, and in this case, this temporary buffer is (now) guaranteed to be deleted whenever the function exits.

The rest of the code is designed to offer a prettier wrapper around the object:

template <typename F>
ScopeExit<F> MakeScopeExit(F f)
{
  return ScopeExit<F>(f);
}

#define DO_STRING_JOIN(S1, S2) S1 ## S2
#define STRING_JOIN_2(S1, S2) DO_STRING_JOIN(S1, S2)
#define DEFER(...) \
  auto STRING_JOIN_2(defer_scope_exit_, __LINE__) = MakeScopeExit([=] () { __VA_ARGS__ })

MakeScopeExit is a wrapper which constructs the object with the correct type, and DEFER is the actual macro in use.

Pratt Parsing

Pratt parsing, described by Vaughan Pratt in a 1973 paper Top Down Operator Precedence², is a parser technique based on recursive descent parsing. These are parsers which operate in a top-down fashion, meaning they define the start symbol as the root and move down to define terminal symbols at the bottom; the recursive part means that until terminal nodes are found in the tree the parser will call other procedures, possibly recursively; for example, when parsing expressions, one continually calls parse_expression until a leaf is encountered, such as a value expression, like an integer.

This can be a little abstract and hard to see what it means; don’t worry, I definitely felt that when first going over this - and the next 2 times as well! (yes I have started and stopped this a few times). As a result, I think it’s best that I show some examples and embrace the notion of a picture says 1000 words; but before that, I’m going to discuss the high level mechanism of parsing before the actual recursive descent part.

Parsing Var Assignment Statements

At time of writing, there are 2 kinds of statements present in Tyger, Var statements, and Expression statements. We’ll cover Var statements only for now. In Tyger, a Var statement is defined as a statement beginning with the keyword var, followed by any ident (e.g. foo), then an assignment (to an expression). In other words, this is defined in the following struct:

typedef struct var_statement
{
  const char *ident;
  Expression *expression;
} Var_Statement;

And Expression is defined as

// Forward declaration as size unknown and need to ref with pointers in other expressions
typedef struct expression Expression;

// ... concrete expression union kinds ...

struct expression
{
  Expression_Kind kind;    // An enumerant representing the kind of expression (like Token_Kind)
  uExpression expression;  // A union of all the different kinds of expressions, opaque for now
};

Note that if you look at the definitions within parser.h these structs differ (in part) from what is actually in the repo. This is simply because the internals for how storing and accessing expressions (in particular) were drastically refactored from when I first wrote the parser code; this is all stuff which will be explained later on.

But taking a step back, how is this actually handled at the level of parser_parse_program?

Well, that function looks like:

Program parser_parse_program(Parser *p)
{
  Parser_Context ctx;
  // initialisation of structs ...
  Program program = {0};
  // initialisation of structs ...

  while (p->cur_token.kind != TK_EOF)
  {
    Statement stmt = {0};
    Tyger_Error err = parser_parse_statement(p, &ctx, &stmt);
    // if error, store the error, else store the statement
    parser_next_token(p);
  }

  // misc post-processing
  return program;
}

This function just runs until EOF is encountered, and repeatedly calls parser_parse_statement. I will omit an explanation of what Tyger_Error is for now, and cover it later The parser_parse_statement function looks like:

Tyger_Error parser_parse_statement(Parser *p, Parser_Context *ctx, Statement *stmt)
{
  Tyger_Error err = {0};

  switch (p->cur_token.kind)
  {
  case TK_VAR:
  {
    err = parse_var_statement(p, ctx, stmt);
  } break;

  default:
  {
    err = parse_expression_statement(p, ctx, stmt);
  } break;
  }

  return err;
}

I.e. if TK_VAR (the var keyword) is encountered whilst parsing, the program will attempt to parse a variable assignment statement. Rather than just dump the code for parsing var statements, I will lay out the steps, and then show the code with some omissions for brevity (especially as a lot of the code is error handling, or “ceremony” for the internals):

enter parse_var_statement
expect_peek that the next token is an IDENT (i.e. parser::peek_token.kind == TK_IDENT)
store the value of TK_IDENT
expect_peek that the next token is TK_ASSIGN
if peek_token is TK_SEMICOLON return syntax error
call parse_expression
if peek_token_is that the next token is TK_SEMICLON read over the semicolon
write statement members to out parameter

Before dumping the annotated function here, I should explain what expect_peek and peek_token_is are; they are utility functions for testing the kinds of tokens so that I don’t have to keep writing if (token.kind == <token kind>) { ... }; again, another idea from the book.

static inline bool peek_token_is(Parser *p, Token_Kind kind)
{
  return p->peek_token.kind == kind;
}

static inline bool expect_peek(Parser *p, Token_Kind kind)
{
  if (peek_token_is(p, kind))
  {
    parser_next_token(p);
    return true;
  }
  else
  {
    return false;
  }
}

Meanwhile, the (annotated) parse_var_statement function is defined as:

Tyger_Error parse_var_statement(Parser *p, Parser_Context *ctx, Statement *stmt)
{
  // 1. enter function
  Tyger_Error err = {0};

  // 2. expect the next token is an ident
  if (!expect_peek(p, TK_IDENT)) { /* return error */ }

  // 3. store the ident
  // ...

  // 4. expect the next token is `TK_ASSIGN` (`=`)
  if (!expect_peek(p, TK_ASSIGN)) { /* return error */ }

  // 5. if `peek_token` is `TK_SEMICOLON` return error
  if (expect_peek(p, TK_SEMICOLON)) { /* return error */ }

  // 6. parse expresion
  // ...
  err = parse_expression(p, ctx, &expr, PRECIDENCE_LOWEST);
  // ...

  // 7. if peek_token is `TK_SEMICOLON` read it
  if (peek_token_is(p, TK_SEMICOLON))
  {
    parser_next_token(p);
  }

  // 8. write statement out parameter
  stmt->kind = STMT_VAR;
  stmt->statement.var_statement = (Var_Statement) {
    .ident = ident,
    .expression = expression,
  };

  return err;
}

Again, the current iteration of the code is different to that present in the actual parser, because of some internal refactoring that came later. The difference is not important for now.

Parsing Expressions

The central part of Pratt parsing, parsing expressions. The kinds of expressions available in the language (at time of writing) are:

Integer Expressions - producing integer values (1, 5, 100)
String Expressions - producing string values (“Hellope!”, “the quick brown fox jumps over the lazy dog”)
Infix Expressions - all binary arithmetic operations (1 + 5 and 1 * 7 - 3)
Ident Expressions - calls to existing, bound, identifiers, (var x = 10; x;)
Call Expressions - function calls (println("Hello, World!");)
- albeit this only really supports the println builtin (for now)

So, the expression parsing code is defined as this:

static Tyger_Error parse_expression(Parser *p, Parser_Context *ctx, Expression *expr, int precidence)
{
  Tyger_Error err = {0};

  switch (p->cur_token.kind)
  {
  case TK_INTEGER:
  {
    err = parse_int_expression(p, expr);
  } break;

  case TK_STRING:
  {
    err = parse_string_expression(p, ctx, expr);
  } break;

  case TK_IDENT:
  {
    err = parse_ident_expression(p, ctx, expr);
  } break;

  case TK_PRINTLN:
  {
    err = parse_call_expression(p, ctx, expr);
  } break;

  default:
  {
    Token_Kind k = p->cur_token.kind;
    fprintf(
      stderr, "[ERROR] Unexpected token kind encountererd whilst parsing: %i (%s)\n",
      k, token_kind_to_string(k)
    );
    assert(0);
  } break;
  }

  if (err.kind == TYERR_NONE)
  {
    while (!peek_token_is(p, TK_SEMICOLON) && (precidence < peek_precidence(p)))
    {
      parser_next_token(p);
      err = parse_infix_expression(p, ctx, expr);
      if (err.kind != TYERR_NONE)
      {
        return err;
      }
    }
  }

  return err;
}

All called functions within parse_expression are defined as returning an error object, and all write their results (on success) to the expr out parameter. There is also just an assert included under the default to crash in case some unsupported token is found; this is a source for future improvement in error reporting.

But to explain the function, if either a “value producing” token is encountered - defined as ints, strings, or idents - then the parser functions for those nodes are called, and I will leave their explanation as they are fairly simple to understand. Once the function parses these leaf nodes, a check is done where if, and only if, the current “precedence” of the expression is lower than the next one, parse_infix_expression is called, which itself will make a call to parse_expression (enter the recursion).

Precedence

You are probably familiar with operator precedence, the idea that certain operations should be carried out in a certain, defined order; you probably also learned some acronym to go along with this in your maths classes (in my case this was BIDMAS - brackets, indices, divide, multiply, add, subtract). Whilst precedence in this case is not exactly the same, we can still encode the order of operations in code, this is also how we encode how evaluation will be carried out later.

The precedence in the parser is defined, simply, as an enum:

enum {
  PRECIDENCE_LOWEST      = 0,
  PRECIDENCE_EQUALS      = 10, // ==
  PRECIDENCE_LESSGREATER = 20, // > or <
  PRECIDENCE_SUM         = 30, // +
  PRECIDENCE_PRODUCT     = 40, // *
  PRECIDENCE_PREFIX      = 50, // -X or !X
  PRECIDENCE_CALL        = 60, // myFunction(X)
};

So in this case, the expression 5 + 4 * 3 should be parsed as 5 + (4 * 3).

Now, with this we can unpick a worked example (of 5 + 4 * 3), when parse_expression_statement executes, it calls parse_epxression with precedence of PRECIDENCE_LOWEST. That means that, when we call parse expression, can think of the following being the case when we enter the loop:

// ...
// expr = { .kind = EXPR_INT, .expression = { .int_expression = { .value = 5 } } }
//...
while (!peek_token_is(p, TK_SEMICOLON) && (precidence /* PRECIDENCE_LOWEST */ < peek_precidence(p /* PRECIDENCE_SUM */)))
  {
    parser_next_token(p);
    err = parse_infix_expression(p, ctx, expr);
    if (err.kind != TYERR_NONE)
    {
      return err;
    }
}
// ...

What this means is that until either a semicolon is encountered, or until the current precedence exceeds or is equal to that of the precedence of the next token, repeatedly call parse_infix_expression (which itself will call parse_expression); this guarantees that we can parse arbitrarily long expressions, such as 1 + 2 + ... + 98 + 99 + 100.

The strangest looking part of this is why is expr, the out-param we just overwrote in the preceding lines, being passed into parse_infix_expression as an inout-param itself, therefore overwriting its value? Looking at the code:

Tyger_Error parse_infix_expression(Parser *p, Parser_Context *ctx, Expression *expr)
{
  // ... Explanatory Comment ...
  Tyger_Error err = {0};

  int precidence = precidence_of(p->cur_token.kind);
  Operator op = token_kind_to_operator(p->cur_token.kind);
  parser_next_token(p);

  Expression_Handle lhs_handle = va_array_next_handle(ctx->expressions);
  va_array_append(ctx->expressions, *expr);

  Expression rhs;
  err = parse_expression(p, ctx, &rhs, precidence);
  if (err.kind != TYERR_NONE)
  {
    return err;
  }
  Expression_Handle rhs_handle = va_array_next_handle(ctx->expressions);
  va_array_append(ctx->expressions, rhs);

  *expr = (Expression) {
    .kind = EXPR_INFIX,
    .expression.infix_expression = (Infix_Expression) {
      .op = op,
      .lhs = lhs_handle,
      .rhs = rhs_handle,
    }
  };

  return err;
}

The strangest looking part of this may be why we pass expr, the out param we just overrode into parse_infix_expression. We pass this in because this is the left-hand side (LHS) of our infix expression, and let parse_infix_expression handle the copying of the data into our Parser_Context::expressions dynamic array. This param is then later overridden to become an infix expression which is our return value (on success).

When parse_infix_expression is called, the following happens:

the precedence of the current operator is taken
a handle to the left-hand side (LHS) of our infix expression is generated
the value of expr is copied into the Parser_Context::expressions dynamic array
a new expression is created on the stack
this is passed to another call to parse_expression, using the precedence derived in 1
when this returns, a handle to the right-hand side (RHS) of our infix expression is generated
the value of rhs is copied into the Parser_Context::expressions dynamic array
the in param, expr, is then overridden, and becomes an out param, defining our new infix expression

The magic of this is, if we carry on until a lower precedence operator, or a semicolon is met, it will result in successive rewrites of the underlying AST.

Worked Example 1

Take the example 5 + 4 * 3; - this will first generate the following infix expression:

pass 1:
  (+)
 /   \
5     4

Then, because the * operator hash higher precedence than +, we pass 4 into another call to parse_infix_expression where we generate the following tree:

pass 2:
  (+)
 /   \
5    (*)
    /   \
   4     3

The RHS of the AST from pass 1 is rewritten to be another infix expression, containing our multiplication, which will be evaluated before our + 5.

Worked Example 2

Take however another example, 3 * 4 + 5; (the mirror of the former); after the first pass we have this as our expr:

pass 1:
  (*)
 /   \
3     4

Hopefully this isn’t shocking, but when we call parse_expression on the RHS of this tree (i.e. on 4), since * hash higher precedence than +, we return this expression as our result, then continue the loop in the initial call to parse_infix_expression, this time passing in the result of pass 1 as the left-hand side of our expression. This then generates:

pass 2:
     (+)
    /   \
  (*)    5
 /   \
3     4

Which again is correct; the deeper nesting of 3 * 4 will be evaluated before summing with 5.

Worked Example 3

Take 1 * 5 + 4 * 2 / 1;:

First, we parse 1, then encounter *, subsequently we generate this subtree:

  (*)
 /   \
1     5

Precedence of * is greater than +, so we return and start parsing the right-hand side of +, since precedence of * is the same as that of /, the statement returns after parsing 4:

      (+)
     /   \
    /     \
  (*)     (*)
 /   \    /  \
1     5  4

Finally, the expression 2 / 1 is parsed, and we end with:

       (+)
      /   \
    (*)    \
   /   \    \
  1     5   (*)
           /   \
          4    (/)
              /   \
             2     1

An Aside

I hope that makes sense - spent quite a bit trying to figure out some good explanations, and visual aids, to help demonstrate the point - maybe I’ll revisit this idea in another post with some animations (if I ever learn how to use available tools). This is also something I wanted to touch on, there was a gap between writing this post and the actual completion of work - mainly because I got stuck on this explanation and I think used it - initially - as an excuse to not finish the article. This did lead to some extra fun when picking it back up as I had to unpick this memory model again of what was going on. (I will also be honest - I got sidetracked on other projects and left this alone for a few months.)

Once again, I would like to stress the importance of having a decent debugging tool, and in this case developing the S-Expression based debug strings was a life-saver; imaging trying to watch all this correctly shuffling data via pointers in a debugger’s watch window! Even though their development (seemingly) saps from the main goal, and the fun stuff, it is time well invested if it is a useful tool, hence why I also wanted a more structured AST representation in YAML, YAML for me, S-Expressions for the test framework.

But that’t it really, Pratt parsing, and handling operator precedence; not actually too bad, and I hope that this has helped cement it in your mind, if not go play with the code or draw out your own diagrams and build your AST from these nodes!

Parser Context

Now, on to some more technical details of the parser, more the “nuts & bolts” of how things work - first up, context args.

Other languages, such as Go, Odin, and JAI (not publically available, and not something I’ve had a change to play with) have an implicit function arg, called context. Depending on the language, this is either a map or a struct, which can be overridden or extended by the programmer. The point of this parameter is a way to bundle extra parameters into the scope of the function without having to explicitly pass them - in Odin for example, the context contains the default (heap) allocator of the language, which is used by all “new” and “delete” style operations (and can be overridden to provide custom allocator functionality).

I decided I wanted a similar concept for C, for at least this problem, i.e. requiring somewhere to store all and expression data which was not just in arbitrary many desperate heap locations (which becomes a nightmare to untangle and ensure correct resource cleanup later). I did think about two approaches as for how the context could be passed to functions; on one hand I could pass a separate parameter, which means adding yet another parameter to all function calls, and on the other hand bundling context alongside the Parser struct. Ultimately, I opted for supplying a second argument, primarily because this is simpler than having to handle the handover of ownership at the end of parsing from the parser to the program, or (potentially) worse, copying all data - which is just wasteful if it’s all there!

I should also explain why I saw need for this context struct in the first place. Given some of my comments in the previous entry in this series about making lifetime management of objects in C as easy and simple as possible, I foresaw quite early on the fact that I would need some backing memory buffer where I could store all data generating during parsing such that I would not have to handle the, rather hairy, lifetime soup that comes from performing multiple, arbitrary heap allocations, not to mention any cleanup efforts that come later when deallocating all resources.

To facilitate this I chose to implement a series of dynamic array, referred to throughout the code as va_arrays (for variable arrays, as opposed to static arrays). I will address what those are next, but in short, for those more familiar with C++ think std::vector, or in Java think ArrayList, i.e. some kind of resizable array. The Parser_Context struct is defined as:

typedef struct parser_context
{
  String_VaArray identifiers;            // Identifiers that appeared in `var` statements
  String_VaArray evaluated_identifiers;  // Identifiers which are to be evaluated for a value
  Expression_VaArray expressions;        // Contains all expressions generated during parsing
  String_VaArray strings;                // contains all string data, the sereis of bytes
} Parser_Context;

The reason for separating the identifiers is superfluous, and reflecting on why it’s this way, I can’t think of a reason other than lower mental overhead whilst writing. This is something else to revisit at a later date, there may be something I’m forgetting whilst writing as to why they need to be separate, but would otherwise be a simple refactor. What exactly each VaArray is defined as is not worth going into on a per-case basis, so I will leave the reader to read the source if he wants more “insight” (but this is all about to be covered anyway).

Dynamic Arrays (In C)

Yes, dynamic arrays can be implemented in C, they aren’t exclusive to C++, Java, or Python! They are phenomenally simple, at least in my implementation, now what you may have noticed is that there are several seemingly desperate VaArray types. C unfortunately does not have generics, and if you attempt to cajole them into the language though use of void* compilers tend to hate you, so I opted for an easier but less simple technique, leverage the preprocessor to define interfaces over which a series of operations can work.

Any dynamic array can be expressed as a struct with at least the following fields:

struct Va_Array
{
  T *elems;         // pointer to the buffer containing elements of type T
  size_t capacity;  // the capacity of the array, i.e. how many elements it **can** hold
  size_t len;       // the number of elements the array **does** hold
};

Since we know that all dynamic arrays conform to this interface, we can define a macro which performs our operations, such as initialisation:

#define va_array_init(T, DA)                        \
  do {                                              \
    (DA).capacity = 32;                             \
    (DA).len = 0;                                   \
    (DA).elems = malloc(sizeof(T) * (DA).capacity); \
  } while (0)

So for example, to initialise a va_array of expressions we would write:

Expression_VaArray expressions = {};
va_array_init(Expression, expressions);

This is an imperfect system, mainly as it requires working with macros which just get annoying to work with over longer, more complex functions; there’s also the whole lack of typing inherent in the preprocessor but eh, an inconvenience worth living with for convenience (also it seems you can just debug once and then it more or less works without issue).

This is all well and good, but what happens when we append elements?

#define va_array_append(DA, ELEM)                               \
  do {                                                          \
    if ( (DA).len + 1 > (DA).capacity ) {                       \
      size_t new_capacity = (DA).capacity * 2;                  \
      void *new_buffer = realloc((DA).elems, new_capacity);     \
      if (new_buffer != (DA).elems) {                           \
        (DA).elems = new_buffer;                                \
      }                                                         \
      (DA).capacity = new_capacity;                             \
    }                                                           \
    memcpy( &((DA).elems[(DA).len]), &(ELEM), sizeof((ELEM)) ); \
    (DA).len += 1;                                              \
  } while (0)

Firstly, we must check if the addition of a new element will overflow the capacity. If yes, we need to allocate a new buffer with enough space for all previous elements, plus the one, ideally with enough headroom for us to be able to allocate many more entities before we need to allocate space again. Then after allocating the new space, we must copy all elements from the old buffer to the new, and free the old. I chose to just use realloc under the hood as it means I (personally) have to care less about manually copying the bytes into a new buffer (and freeing the old). The full suite of “functions” I implemented for dynamic arrays can be found in the repo.

You do also need to pick a “sensible” scaling factor to regrow your array by each time you exceed the capacity, the reason for this is it’s slow to go out to the OS and have it provision memory for you to use (via malloc), so we ideally want this minimised. I chose to just double the capacity every time it gets exceeded; this may be a more wasteful strategy, but I think the reduction in allocations is a worth the cost.

The major benefit here, as with arenas[^20] is I now have a single block of objects (of shared type), all bundled under a single pointer, which makes cleanup as easy as free(<va-array>->elems)! Now I should come on to where this attempt to simplify my life actually caused some headaches…

Handles Vs Pointers

Earlier on I made reference to taking handles to the Parser_Context::expressions array; this may have seemed hand wavy and that was intentional. Initially, the struct for a String_expression, for example, was defined as such:

typedef struct string_expression
{
  const char *value;
  size_t len;
} String_Expression;

And the function parse_string_expression looked something like this:

Tyger_Error parse_string_expression(Parser *p, Parser_Context *ctx, Expression *expr)
{
  Tyger_Error err = {0};

  const char *str_value = &(ctx->strings.elems[ctx->strings.len]);
  va_array_append_n(ctx->strings, p->cur_token.literal.str, p->cur_token.literal.len);
  va_array_append_n(ctx->strings, PARSER_NULL_TERMINATOR, 1);

  *expr = (Expression) {
    .kind = EXPR_STRING,
    .expression.string_expression = (String_Expression) {
      .value = str_value,
      .len = p->cur_token.literal.len
    }
  };

  return err;
}

Don’t be put off by va_array_append_n - this is just a way to copy N elements into the dynamic array (only really exists for copying strings). What is important is can you spot the bug in this code? Nor could I when I wrote this. Worse, it didn’t even rear its head until I started fleshing out string parsing test cases later on in development! The bug I saw was that, when I tried parsing some strings, seemingly those where the length of was greater than 32 chars, I would start encountering “heap corruption” errors and garbled strings, sometimes containing the actual string, and other times containing junk (like Unicode chars).

Again, given what I have laid out in this article about my code architecture, functions, and data models, can you spot the bug? Seriosuly, I encourage you to take a minute before reading further to think about what the bug is…

/*














 ██████   █████               █████████  █████                          █████     ███                      ███
░░██████ ░░███               ███░░░░░███░░███                          ░░███     ░░░                      ░███
 ░███░███ ░███   ██████     ███     ░░░  ░███████    ██████   ██████   ███████   ████  ████████    ███████░███
 ░███░░███░███  ███░░███   ░███          ░███░░███  ███░░███ ░░░░░███ ░░░███░   ░░███ ░░███░░███  ███░░███░███
 ░███ ░░██████ ░███ ░███   ░███          ░███ ░███ ░███████   ███████   ░███     ░███  ░███ ░███ ░███ ░███░███
 ░███  ░░█████ ░███ ░███   ░░███     ███ ░███ ░███ ░███░░░   ███░░███   ░███ ███ ░███  ░███ ░███ ░███ ░███░░░ 
 █████  ░░█████░░██████     ░░█████████  ████ █████░░██████ ░░████████  ░░█████  █████ ████ █████░░███████ ███
░░░░░    ░░░░░  ░░░░░░       ░░░░░░░░░  ░░░░ ░░░░░  ░░░░░░   ░░░░░░░░    ░░░░░  ░░░░░ ░░░░ ░░░░░  ░░░░░███░░░ 
                                                                                                  ███ ░███    
                                                                                                 ░░██████     
                                                                                                  ░░░░░░      
          













*/

va_array_append(_n) may call realloc at any point, and thus invalidate all pointers previously stored anywhere, including the context va_arrays.

AGHHHHH!!!

This was such a stupid bug, and stupid oversight on my part, and the worst part is the solution is so simple; I’ll leave a link for an article called Handles are the better pointers³ for the reader’s curiosity, it’s a good article, and embarasingly something I had read a while before embarking on this project, so I already knew the correct approach. Anyway, after realising this, I refactored all my parsing code to use handles instead of pointer⁴.

The fixed code, for string expression handling at least, looks like this:

// parser.h
typedef struct string_expression
{
  String_Handle handle;
  size_t len;
} String_Expression;

// parser.c
Tyger_Error parse_string_expression(Parser *p, Parser_Context *ctx, Expression *expr)
{
  Tyger_Error err = {0};

  size_t str_value_handle = va_array_next_handle(ctx->strings);
  va_array_append_n(ctx->strings, p->cur_token.literal.str, p->cur_token.literal.len);
  va_array_append_n(ctx->strings, PARSER_NULL_TERMINATOR, 1);
  const char *str_value = va_array_address_of(ctx->strings, str_value_handle)

  *expr = (Expression) {
    .kind = EXPR_STRING,
    .expression.string_expression = (String_Expression) {
      .value = str_value,
      .len = p->cur_token.literal.len
    }
  };

  return err;
}

Where String_Handle is just an aliased size_t for an index in the array, and va_array_next_handle just returns the current length of the dynamic array, as that’s just where the next element will be inserted (i.e. the next free space).

I should also note that even though I only spotted this during string parsing tests, it was obvious, once I figured out that my issues were down to pointer invalidation, that this affected every single parsing function which interacted with those dynamic arrays⁴. Interestingly it’s a good thing that this showed up because of me being lazy and wanting to run some integration tests on the simplest data type I was tracking (strings). Why? Well imaging trying to debug this reallocation issue when you have some program comprising 32+ value producing statements, I don’t think I would have found it as quickly as I did by just watching my strings since only “a single object” - in reality 32+ objects because each char is considered an object for purposes of my functions - and that is easy to trace in a debugger because it’s the result of stepping over 1 line of code. This is also probably a demo as to why just stress testing/fuzzing systems and throwing junk at them to see what happens is a good idea, again I just threw some random strings which were crafted for force reallocation. Granted, this could have been avoided had I written some tests elsewhere, so that’s a lesson learned for next time.

I think this bug alone has helped solidify a certain mental model for memory allocation, and ensuring you think about the flow of data through your application’s lifetime, exactly the thing I wanted to get out of doing a “real”, and more complicated project - you don’t really see stuff when you just look over the simple examples that you see littering the internet for C programming tutorials/explainers. Also better to spend a few hours banging your head against a wall on a “toy” project when you encounter these issues, so you know how not to encounter them again when you work on production code.

// An example of the overly simplistic examples of memory mamngement you see littering
// the internet in reference to C/C++
char *foo() {
  char *buffer = malloc(100);
  return buffer;
}

void bar() {
  char *buffer = foo();
  // ... do something with buffer ...
  free(buffer);
}

Error Handling

Error handling is a bit of a pain in C (and C++, but if you’re a fan of exceptions you have those at least), and I wanted to do something a little better than just returning either NULL pointers, or integer error codes. Now whilst in practice what I have initially scaffolded is just an error code, I do aim to extend this after I get program evaluation functioning, but what I do in my parser is return structs which represent the error, Tyger_Error:

typedef struct tyger_error
{
  Tyger_Error_Kind kind;
} Tyger_Error;

The kinds of errors tracked in the language, at this level, include syntax errors (e.g. declaring a variable with no ident: var = 10;), and integer parsing errors (returned when a call to atoll returns 0, and the parsed string was not "0").

The main downside of choosing to handle errors like this, rather than return pointers to parsed expressions/statements, is that I now get into more complex data flows (as previously laid out) due to the need to pass round multiple out/inout params, i.e. parameters to which a result is written, and parameters which both provide data and where the result is written to, respectively. If I were just returning pointers, which may be null in case of an error, then all my functions simply would accept the parser state, and just return the null pointer and check this. Again, for reasons laid out earlier, namely I wanted to simplify the memory management of my data, I chose to trade off a more complex data flow for improving the mental load required for wide scale memory management. This approach also does allow for expansion later into a better error reporting system without later, more major refactors, for example:

typedef struct tyger_error
{
  const char *what;       // A formatted error message, as a heap allocated string
  Location where;         // A struct showing where the error was found, containing line and col
  Tyger_Error_kind kind;  // A simple marker of what class of error this was
} Tyger_Error;

Misc Issues From Development

This is mainly a round up of a few things which don’t really fit anywhere, but were big enough headaches or just weird minor issues that I think need to be noted here, if for no other reason so somebody else doesn’t tear his hair out trying to figure it out.

Forgetting How println Was Defined

When I initially tried to parse call expressions, how function calls are defined, I forgot that TK_PRINTLN was how the println keyword was lexed, i.e. as its own keyword, not an ident, which would actually be more sensible as I even noted in code⁵. This something I will be changing later, particularly once I have a working eval loop going; it will also require me to probably work a peek_token_is into the relevant case for parsing idents to check for this.

Forgetting The Existence of Commas

As alluded to earlier, the parser just wasn’t aware commas were a valid token, as the lexer didn’t support them⁶. This was a headache when parsing functions, namely println as this is the only supported function, as I couldn’t figure out why multi-arg parsing was just failing for some syntax error reason. This is something that was missed from initial lexer test cases and will need to be added later to check things (which will require a minor addition to the code gen script).

Cross-Compiler Compatibility

Most of my development is done on Windows, although in future this will change as I plan to move back to Linux (for the most part) thanks to Microsoft’s efforts to ruin Windows. Now, I do run my tests in CI, which uses GCC on Linux, but I because I don’t usually run these locally (I do have act running on WSL, but I don’t always run them) so it can take a while for it to become obvious that these are problems.

Warning Levels

In my experience, attempting to use /Wall warning levels with MSVC is just a pain, and nothing will ever compile because it complains about things in the standard library. This however does lead to problems, as you compile on /W4 level, and since this doesn’t catch certain problems, you only learn in CI that these are issues. One example of this is the snprintf function, which is used by my string builders for debug tracing⁷. Initially, the wrapping functions which append strings all call this internally, I did however encounter a fun compiler discrepancy between MSVC and GCC, namely that with my Windows config this code does not error:

void string_builder_append(String_Builder *sb, const char *str)
{
  // ...
  int bytes_written = snprintf(&(sb->buffer[sb->len]), bytes_to_write + 1, str); 
  // ...
}

The problem here is that you can’t just use snprintf to append a c string to another buffer, the fix for this was simple, it was simply to change the call to be snprintf(&(sb->buffer[sb->len]), bytes_to_write + 1, "%s", str);. But this is one of the weird differences in compilers and platform development you encounter; I’m not saying this is exclusively a GCC/Clang problem, what I am saying is that were development with /Wall not miserable on Windows, I would probably have seen this.

A Small Rant About Comments

I swear I find the dumbest bugs in everything - over the last few months for example, I can semi-reliably cause a rendering bug in Slack where, if I drag the window from one monitor to another quickly and have it resize in quick succession, it just renders the chat content at like 3/4 height. But anyway, my hate for chat clients will come another day, I’m here to vent my hatred for compilers!

Who would like to guess the problem with this test case code? Hint, it’s specifically a problem with GCC/Clang, and think about macro rules.

TEST(ParserTestSuite, Test_Operator_Precidence)
{
  struct Precidence_Test
  {
    const char *input;
    const char *ast;
  };

  std::vector<Precidence_Test> test_cases{
    // ... 
    { "1 + 2 - 3 * 4 / 5;", "(- (+ 1 2) (/ (* 3 4) 5))" },
                                                     //       (-)
                                                     //      /   \
                                                     //    (+)    \
                                                     //   /   \    \
                                                     // (1)   (2)  (/)
                                                     //           /   \
                                                     //         (*)   (5)
                                                     //        /   \
                                                     //      (3)  (4)

    // ...
  };
};

Well here’s (a recreation of) the compiler error message (from godbolt):

<source>:7:54: error: multi-line comment [-Werror=comment]
    7 |                                                      //      /   \

That’s right, the backslashes in my ASCII art representation of the parse trees of infix expressions escape the proceeding line’s backslash denoting a new comment! Yes I’m serious, yes there is a commit for this⁸, yes I did yell into cyberspace fixing it, and for posterity I will just paste the commit message here as I think it summarises this best:

    fix -Werror-comment compiler error on GCC/Clang

    Apparently GCC/Clang don't like handling the \ char in comments
    and assume that this means you want to escape the new line and
    join this line with the next. Because this is ObViOuSlY sEnSiBlE
    bEhAvIoUr DuMmY ¯\_(ツ)_/¯

I have many other words I could say about this, but I will just choose not, I will simply say I am so glad that programming generally does not come up against this utter nonsense often because I would have quit. The fix was simple, just use /* */ multi-line comments, hardly a Sisyphean task, but this is just stuff you shouldn’t have to get involved with or waste neurons on! Maybe there’s good reason for this, I don’t know, hopefully this saves someone 5 minutes.

Closing Remarks

Parsing has been done! This now lays the foundations for later evaluation, i.e. the thing which I think any language designer wants to get to; it is to be seen whether my architecture here is any good for implementing evaluation. I also think that doing this project and having to explain certain things - like the memory manipulation flow for parsing expressions, and Pratt Parsing - has helped me better understand some of these rather abstract concepts. I’ll leave a few extra resources here I used to help understand Pratt Parsing, and of course, buy book!

I will also close with this, I think writing is proving the point of “don’t make the perfect the enemy of the good”, a misquote, but the sentiment of don’t make inability to achieve “perfection” [in the moment] a reason not to make something which is “just” good. I could have finished this article, and the first pass of this project a few months sooner had I just started trying to paper a bad explanation of both Pratt Parsing, and the flow of memory through my expression parsing; I’m not even convinced that my explanation now is phenomenal, however I am actually happy with it, even with several rewrites in the background.

Additional Resources

There are other books in the world that cover this topic (parsing, and language development more generally), and one which is on my reading, I will list a few here:

Compilers: Principles, Techniques, and Tools
- I have not read this one, but am aware that it’s a well cited book for compiler developers, and is on my reading list
Crafting Interpreters
- this is freely available, and does feature the “from scratch” approach to development
Build Your Own Lisp
- this is a different kind of book, also freely available
- from my reading of it, there’s a bit too much “use this, and it’ll work” for my liking, but maybe this is more your thing

I may have to do a review of these all one day, comparing them to the Interpreter Book, and go over what they seem to do well as learning resources.

There are also a few videos I’ll link, which cover some practical aspects of compiler development, parsing, and operator precedence, from somebody developing a C++ replacement (Jonathan Blow, creator of JAI):

Finally, a couple of links on Pratt Parsing which were of varying usefulness, and are collated here for completeness:

Tyger 1: Prelude and Lexing

Sun, 01 Jun 2025 23:05:31 +0100

Tyger is a dynamically typed, interpreted, scripting language, developed as an extension of, and inspired by Thorsten Ball’s book, Writing An Interpreter In Go. I wanted a more complex project to get my teeth into that would improve my both my skills as a programmer generally, and with C programming (specifically). The aforementioned book is a fantastic resource, and I would recommend people check it out, even if language development isn’t something you’re super interested in as it’s just a good resource to work through (try translating it into your language of choice for a challenge!)

As mentioned, I wanted to improve my C skills and my general programming ability. To me, it seems C provides the best opportunity for this as

it is low level (defined as giving you fine control over memory)
it features no garbage collection
- garbage collection is a feature I have used, as I have used several languages which implement it (including Go, Python, and JavaScript)
- I would like to try implementing it, to get a better appreciation of how it works
many convieniences present in Go are non-existent in C:
- count-based strings, and arrays
- UTF-8 support (well it is there as part of the standard library, but I don’t like working with null terminated strings)
- automatic serialization of types (kind of, the most useful example is being able to stringify a struct implicitly with %v/%+v format specifiers)

The interpreter is written specifically using the C99 standard; I chose this, primarily, for the following reasons:

“C++” style comments, until C99 all comments had to be defined between /**/
stdint.h which provides a nice series of fixed witdth int definitions
- no more checking whether an int is 2 or 4 bytes (nice when required)!
designated initialisers for structs and unions
no forced forward declaration of variables
- in C89 you needed to declare all variables used in a scope at the beginning of the scope
- C99 relaxes this and allows you to declare only when required

The Basic Outline

What I am going to implement, at least initially, is an interpreter capable of running the following program:

var x = 10;
var y = x + 5 - 4 * 3 / 2;
println(x, y);
println("Hello, World!");

To do this, I need to implement a lexer capable of lexing

identifiers
- e.g. x, and y
- builtin functions (i.e. println)
keywords
- i.e. var
string and integer literals
binary infix operators

I will also need to parse these lexed tokens into an Abstract Syntax Tree (AST), the program will be interpreted using a tree-walk interpreter, finally, I will need to implement a garbage collector (although until more complex constructs are implemented this may be hard to properly validate the function of).

I will not be implementing ability to handle comments, functions, or control flow yet (loops, if/else, etc); I also will not implement things such as block statements (a series of statements bounded by { ... }), return statements, constant vs variable binding, macros, classes, packages/modules, or a C FFI (foreign function interface). This is a brief list, and not exhaustive.

All of these are things I would like to implement, however I am trying to keep the scope quite radically small to ensure I can, and will finish it. Even so, the breadth of what needs doing is quite large, mainly due to several technical challenges that will come out of this, such as:

memory management, I’m discussing the management of the interpreter memory here, not the process of garbage collection
- i.e. allocating the AST, where to store variables and idents until evaluation, etc
error handling (“traditionally” C programs will use int return codes, or set errno, I would like to trial something different)
how to actually handle the process of constructing an AST
type resolution
- i.e. whem a variable encountered, derive the type so evaluating operations can be handled appropriately

Perparing The Ground

Build Systems

I decided to go with CMake for this project, not out of a real like for it it, mainly because

it’s fine
I’ve used it before
it has easy setup to allow me to get started with GTest

Normally, CMake is not my main choice, I tend to just dump things in batch/shell scripts, and use those to build my projects. I may add these later, mainly for end user convienience if they should choose to compile it themselves. As a result of my… ambivalance… towards CMake, I choose to keep the scripts as minimal as possible, and as a result the extent of the build system is:

set C99 and C++14 as the target standard used in compilation
- C99 is for the “library” build and the main interpreter exe
- C++14 is required for GTest
compile a “library” binary, containing all files which require testing
compile the interpreter, linking it against the “library” generated
setup, and compile the test runner executable

I also make sure I set a handful of compiler flags:

if (MSVC)
    add_compile_options(/W4 /w14640 /WX /permissive- /Zi /Od)
else()
    add_compile_options(-Wall -Wextra -Werror -Wshadow -pedantic -g -O0)
endif()

This is hardly extensive, and probably requires some tweaking, but for now it’s good enough. In essence, the aim is to have

the maximum number of warnings enabled possible
treat all warnings as errors
enable debug info
disable optimisations (no real reason for this, besides making inspection of any generated code easier)

You can inspect the full build file if you wish, but further delving into the build system hardly seems interesting, or germane to the topic at hand.

Test Driven Development

I think Test Driven Development (TDD) has a time and a place; namely, where you are dealing with well defined input and output requirements. For example, assume you were to design a library containing some basic maths and linear algebra functions, this would be a good case for TDD as, for example, you can define that

function vector2_scale should scale the magnitude of the vector appropriately
some example inputs/outputs:
- vector2_scale(Vector2{ 1, 1 }, 5) == Vector2{ 5, 5 }
- vector2_scale(Vector2{ 10, 10 }, 0.2) == Vector2{ 2, 2 }
- vector2_scale(Vector2{ 1, 1 }, -1) == Vector2{ -1, -1 }

Since I can quite easily define the expected inputs, and outputs for each stage of program interpretation, I have decided to employ this as a test strategy in the development of my interpreter, at least initially.

Now to actually perform the testing, I will be using the C++ testing library GTest to run tests, mainly because it’s one I had seen recommended and it’s also one I have some experience with from other (dead) side projects I’ve worked on. GTest has some interesting features, one of which I only found documented in a Stack Overflow thread, this being the ability to supply extra diagnostic messages when assertions/expectations fail via streams:

...
int res = 2 + 2;
ASSERT_EQ(res, 5) 
	<< "... Sometimes they are five. Sometimes they are three. Sometimes they are all of them at once..."
...

This is particularly useful when coupled with running comparisons on enums, since, when these are output to console, they are (implicitly) converted to integers (at least in C), and it’s hard to translate that to what you’ve written.

Generating Test Case Code

Part of what I want to do in this project is include good error diagnostic messages for users, as a result, even though it won’t be implemented fully, I am scaffolding out a lot of this early. As a result, this will require me to write a lot of finicky tests involving counting column and line numbers to verify reporting is correct. This is tedious, and potentially error prone, I have decided that a certain degree of duplication of work is acceptable. In this case, I am duplicating the code that will be used to create the Lexer, to automatically generate:

Test cases (the expected tokens)
The suite of Enumerants of what kind of tokens are supported
A dump of the source code used to generate the tests

Since the code is mostly a duplicate of that implemented in the actual lexer, I will omit explanations of the code, and defer you to my next section; however, for those keen to purose - or those more confident with Python than C - I will link the script here.

One thing I will touch on as an interesting issue, which caused some confusion initially, was line endings; oh yes, they’re still causing problems in <current year>!

The main issue that I found, was because I chose to load a separate file from disk (rather than just handling the string as a raw literal within the script) I immediately encountered issues in my CI workflow to do with different offsets of all tokens within the test cases compared to what was actually generated. This is because \r\n (what was present on the Windows machine I used) were assumed to be present in the file, and the CI ran using Linux, which uses \n line endings (legacy versions of MacOS used \r but this is less common now).

This is simple to solve, but I will tackle this problem later; the solution, I think, is to only do tests for the location of the token in the file as a measure of line and column position (and omit checking the absolute position in the file).

Now enough preamble. The code proper!

Lexical Analysis

The first step in parsing a language is to convert the text into a stream of tokens. This is done as it is easier to work assign meaning when parsing when you operate on a higher level than individual characters; for example, operating on a token of type Var_Keyword is easier than deciding what needs to be done with the char v as this could be a keyword, an identifier, or a keyword wrongly being used in place of an identifer (e.g. as an identifier).

Defining The Process

I am essentially copying the model defined in the book, but before we get to that, I want to outline the end usage code, and then define what we need to happen within the code. Every time we want to get the next token, we need to call lexer_next_token(&lexer), passing in a pointer to a Lexer struct. This function then needs to perform the following:

skip whitespace, since in this language it is not significant (unlike Python)
match the current character the lexer is “looking at”
decide, based on the current char, how to transform this into a token
advance the lexer
return the generated token

Defining The Lexer

As mentioned, this comes from the book, but is modified to (a) add debug information in the form of column and line numbers (although this is unsused for now), and (b) track the length of the input string. The later of these is required since Go tracks this information implicitly by means of the string type; in C strings are null terminated (by default), for now I will be deriving the length of the string once during initialisation, rather than running strlen each time a call to lexer_read_char or lexer_peek_char is made.

typedef struct lexer
{
  const char *program; // The input string to lex
  size_t program_len;  // The length of the input string (in bytes)
  size_t pos;          // The current offset from the start of the string (in bytes)
  size_t read_pos;     // The next offset from which the lexer will read the char (in bytes)
  size_t col;          // The current column in the file the lexer is reading (unused)
  size_t line;         // The current line in the file the lexer is reading   (unused)
  char ch;             // The current character the lexer is reading
} Lexer;

Initialisation of the lexer looks like this:

void lexer_init(Lexer *lx, const char *program)
{
  lx->program = program;
  lx->pos = 0;
  lx->read_pos = 0;
  lx->col = 0;
  lx->line = 0;
  lx->ch = '\0';

  size_t prog_len = strlen(program);
  lx->program_len = prog_len;

  lexer_read_char(lx);
}

There is an annoyance I have here, nameley that we are reliant on strlen and the fact our input will be properly null terminated. Fow this is okay, but when hooking this code into the actual program (and not just passing char* to the function) I will need to take special care to ensure the input string is properly formatted (whether this be from REPL input, or from reading a file).

Tokenisation

Onto the process of generating Tokens from source code…

Token lexer_next_token(Lexer *lx)
{
  // 1. skip whitespace
  lexer_skip_whitespace(lx);

  // 1.5. default construct the Token
  Token token = {
    .location = make_location(lx->pos, lx->col, lx->line),
    .literal = make_string_view_ex(lx->program, lx->pos, 1),
  };

  // 2. Match the current character being inspected by the lexer
  switch (lx->ch)
  {
	// 3. based on what character is matched, generate the appropriate token
	...
  }

  // advance the lexer to read the next char
  lexer_read_char(lx);
  return token;
}

Default token construction contains 2 macro calls, these are present to make the process of creating the nested strcuts a little more pleasant, particularly as the make_string_view_ex macro performs some pointer tricks to generate a pointer which is offset from the initial pointer, which is quite noisy. Expansion of the macro yields (String_View) { &(lx->program[lx->pos]), 1 }.

The reasons for setting the default state of the Token are:

all tokens, after skipping whitespace, will be found at the location the lexer currently points at (i.e. the absolute offset, col, and line number)
many tokens will be of small length (1), so it makes sense to just pre-populate that and worry about updates to it later
- The tokens which are of length 1 include: ( ) { } [ ] + - * / = ! < >

Perhaps, as more tokens are added, I will change my mind on setting the default representation of the literal to not be of length 1, but for now I think it’s worth setting the default.

Defining Our Tokens

Now we’ve seen the way the lexer is defined, and how new tokens are generate, it makes sense to dive in to how these tokens are structured:

typedef struct token
{
  Location location;   // A representation of where in the source code the token was found
  String_View literal; // A representation of the underlying string the token represents
  Token_Kind kind;     // A tag representing the kind of token this is
} Token;

Where Location is defined as

typedef struct location
{
  size_t pos;
  size_t col;
  size_t line;
} Location;

and String_View as

typedef struct stirng_view
{
  char *str;
  size_t len;
} String_View;

We will touch on how the Token_Kind is defined later, for now it is sufficient to say it is an enum.

String_Views

A Note About Memory Allocation

Dynamic memory allocation is one of those things before I got started with C programming I never really understood; neither the actual process, nor the reasons why you would want/need to use this method. It is important to note that for a while the only languages I ever really used for the first 4-5 years or me programming were Python, Java, and JavaScript, all garbage collected languages.

At some stage I got annoyed at not understanding this whole process and wanted to dive deeper (even if I thought I’d never use it), so I started writing a CLI program which would take a DNA/RNA string as its input and convert it the corresponding RNA/DNA sequence. The code for this has been lost, but it did begin the journey of understanding memory management at a lower level than that afforded by garbage collection.

The first thing I did, is what I beleive most, if not all, budding C developers do when they first reach for dynamic memory allocation strategies, which to to emulate RAII. What I mean by this, is every <object>_alloc(...) call is followed by a corresponding <object>_free(...) call. Whilst this is fine, there are still some problems with this style of allocation:

it is important to note that every allocation takes time, as does each free.
every allocation can fail, now on modern operating systems and hardware this may indicate a much more serious problem, but it is still an extra point of failure that you need to check, such as you don’t write/access some null pointer.
this is less robust than RAII, mainly because you the programmer are now responsible for correctly handling the ending of each object lifetime at every scope/function exit. That means all happy and unhappy paths!
(what I consider most) importantly it is incredibly difficult to track object lifetimes, not only within a function call, but across function calls, and decide who is responsible for handling cleanup.
- this bleeds a little out from the prior point on RAII.

The last point is particularly true once the number of objects (of potentially differing lifetimes and types) increases beyond a [small] handful! Ownership is also increasingly problematic to keep track of. This situation is what I have seen be referred to as the lifetime soup.

A much better approach, if possible, is to group lifetimes together and handle the deallocation of backing resources in batches. Take for example, a program which reads some input file and splits the text into its constituent words by whitespace, let us for now assume that the programmer wishes to echo these words to stdout, and possibly retain a count of the number of words found.

One solution could look something like this, where each word is allocated as a small string, then freed after printing:

int main(int argc, const char *argv[])
{
 	// function is assumed to exist
	const char *file = read_entire_file_into_memory(argv[1]);

	size_t index = 0;
	int word_count = 0;
	while (file[index] != '\0')
	{
		size_t buffer_start = index;
		size_t buffer_offset = index;

		// NOTE: this a function assumed to exist which checks for " ", \n, \r, \t
		while (!is_whitespace(file[buffer_offset++])) {}

		size_t bytes_read = buffer_offset - buffer_start;

		// remember space for null terminator
		char *word = malloc(sizeof(char) * (bytes_read + 1));
		strncpy(word, &(file[index]), bytes_read);
		word[bytes_read] = '\0';
		printf("%s\n", word);
		free(word);
		word_count += 1;

		index += bytes_read;
	}

	free(file);
}

Every time through this loop, we allocate a string, then print, then free. Now whilst the allocation, usage, and deallocation of the resource (our string) are colocated, so the lifetime of the object (the string) is obvious, this is very often not the case in reality (the colocation of allocation, usage, and deallocation). We may, in this case and more generally, be better served by a different strategy.

int main(int argc, const char *argv[])
{
 	// allocate a backing buffer capable of storing 256 chars
	size_t word_buffer_len = 0;
	size_t word_buffer_capacity = 256;
	char *word_buffer = malloc(sizeof(char) + word_buffer_capacity);

 	// function is assumed to exist
	const char *file = read_entire_file_into_memory(argv[1]);

	size_t index = 0;
	int word_count = 0;
	while (file[index] != '\0')
	{
		size_t word_start = index;
		size_t word_next = index;
		while (!is_whitespace(file[word_next++])) {}
		size_t word_len = word_next - word_start;

		if (word_buffer_len + word_len >= word_buffer_capacity)
		{
			size_t new_capacity = word_buffer_capactiy * 2;
			word_buffer = realloc(word_buffer, sizeof(char) * new_capacity);
			word_buffer_capacity = new_capacity;
		}
		strncpy(&(word_buffer[word_buffer_len]), &(file[index]), word_len);
		word_buffer[word_buffer_len + word_len] = '\0';

		word_buffer_len += (word_len + 1);
		word_count += 1;

		index += word_len;
	}

	char *next_word = word_buffer;
	for (size_t i = 0; i < word_count; ++i)
	{
		printf("%s\n", next_word);
		while (*next_word++ != '\0') {}
	}

	free(word_buffer);
	free(file);
}

The later of these 2 requires some more code to be present, however we are not in a state where every word registerd requires us to allocate a new buffer, then delete. Instead we allocate one backing buffer, then resize this as needed. Most of this added complexity can be abstracted away through function calls should we desire to clean it up, but it should demonstrate the principle (more so than be a good example of why it’s preferable to do this here). This is most similar to a technique known as Arena allocation, and whilst I won’t cover/use that here (at least, not yet), I will leave some links for the reader to enjoy¹ ² ³.

It should also be noted that these programs are trivial, and in reality we could probably refuse to deallocate any allocations, and just let the OS handle that on exit (the ultimate garbage collector); but again, this is to demonstrate how you can choose to tackle a problem in a different way.

How The Lexer Handles Lifetimes

So in my lexer, I have not opted for the previously outlined method, at least in this case; instead, I opted to allocate a single buffer once, then create a struct which points into this buffer and gives us our view of a smaller subsection of the string.

In this program, that structure is called a String_View. As previously noted, this type is composed of a pointer, and a length. The pointer points to some offset in Lexer.program, for example, assume that program is defined as

var x = 10;

The first token, representing var, generated from here will have its String_View defined as:

token.literal = (String_View) {
	.str = &(lexer->program[0]),
	.len = 3
};

And the token representing the integer literal 10 will have its literal defined as:

token.literal = (String_View) {
	.str = &(lexer->program[8]),
	.len = 2
};

To better demonstrate what some of this pointer trickery is doing, here is an image showing the string as a char array, with each char assigned an index:

Encoding The Kind of Token

Token_Kind as previously mentioned is an eunm, which is defined as

typedef enum token_kind
{
#define X(NAME) TK_##NAME,
  #include "defs/token-kind.def"
#undef X
} Token_Kind;

Now this probably looks like utter nonsense to most of you…

Where are the enumerants?
Why am I defining some obtuse macro which isn’t used?
Why is immediatly #undefed?
What on earth is the include doing there?
What is a .def file?

Simply put, this is a convienience I took to make my life easier, as the trick used here is also used to help me write a to_string function for this enum. But to discuss what is going on, and to appreciate the problem this solves, we need to discuss the C Preprocessor.

The Preprocessor

For those unfamiliar with C/C++, particularly the compilation model let’s just cover that before we get to the details. The steps in compilation are as follows:

Run the prepocessor - handle #include, #define, and other such directives.
Compile a translation unit into an object file.
Link all compiled object files, and any libraries, into the final executable.

A translation unit in this case, roughly, is a single C/C++ source file. This isn’t quite accurate, but it’s good enough working definition for our purposes.

But, we’re here to discuss the preprocessor, not the full compilation model.

A Crash Course

As mentioned, the preprocessor is the first step in compilation. What happens during this is that a series of directives - defined as #<directive> - are enountered whilst the translation unit is processed, and a textual replacement operation is performed.

Take for example the following macro, part of a familiy called function-like macros:

#define MIN(A, B) ((A) < (B) ? (A) : (B))

Now assume that this is used in a statement int res = MIN(10, 5);, what happens when the preprocessor runs is that MIN(10, 5) will be expanded to ((10) < (5) ? (10) : (5)). This style of macro was more commonly used when compilers didn’t support function inlining, but does come with the double edged sword of lacking type information; this is good as now you have a “function” which works for all integral types and floats - no need to define different functions for each type! - this does however mean that you can end up with… less than helpful errors should you, for example, attempt to compare some struct with an unsigned long.

With that, let’s move on to a cool technique, the one used to define the Token_Kind enumerants…

X-Macros

X-Macros are a technique I only found out about quite recently, from listening to (and reading the comments on) an interview with Billy Basso (creator of Animal Well), which references a subsection of a GDC talk, The Simplest AI Trick in the Book.

They are a sanity saver! The technique leverages this property of C macros, and the preprocessor, namely that it is a textual substitution. Let us take a classic example of this, paired down for brevity, of describing the days of the week as an enum, and a function which generates a debug string:

enum week_day {
	Monday = 0,
	Tuesday,
};

const char *week_day_to_string(enum week_day d) {
	const char *str;
	switch (d) {
		case Monday:
			str = "Monday";
			break;
		case Tuesday:
			str = "Tuesday";
			break;
		default:
			break;
	}
	return str;
}

Now whilst this may not seem too bad, we have 2 places where the same list needs to be maintained, and as an artifact of how C is normally written, these will be in 2 different files; as a result, it is quite easy (without strict compiler warning enabled) to forget to update the enum cases in these 2 places, a problem which only gets worse the more locations

This is a bit of a headache, so let’s rewrite this using the X-Macros technique, and then explain what’s happening:

#define WEEK_DAY_LIST \
	X(Monday)         \
	X(Tuesday)

enum week_day {
	#define X(NAME) NAME,
	WEEK_DAY_LIST
	#undef X
};

const char *week_day_to_string(enum week_day d) {
	const char *str;
	switch (d) {
		#define X(NAME) case NAME: str = #NAME ;  break;
		WEEK_DAY_LIST
		#undef X
		default:
			break;
	}
	return str;
}

The code looks a little more concise, however it’s not apparent what is going on; lets dig into the enum and walk through the evaluation of this dense block:

Starting with

enum week_day {
	#define X(NAME) NAME,
	WEEK_DAY_LIST
	#undef X
};

If we replace WEEK_DAY_LIST with the content it encodes:

enum week_day {
	#define X(NAME) NAME,
		X(Monday)
		X(Tuesday)
	#undef X
};

Now finally let’s substitute the X:

enum week_day {
	#define X(NAME) NAME,
		Monday,
		Tuesday,
	#undef X
};

What we have done is encode a way of describing large scale substitution of a pattern, not too disimilar from our MIN macro from earlier. The key difference here, is that we immediately undefine the macro (X) after invoking it; this means we can “reuse” the macro for a different subsitution later.

The power of this becomes apparent as we apply the same substitutions to our to_string function:

const char *week_day_to_string(enum week_day d) {
	const char *str;
	switch (d) {
		#define X(NAME) case NAME: str = #NAME ;  break;
		WEEK_DAY_LIST
		#undef X
		default:
			break;
	}
	return str;
}

Substituting WEEK_DAY_LIST

const char *week_day_to_string(enum week_day d) {
	const char *str;
	switch (d) {
		#define X(NAME) case NAME: str = #NAME ;  break;
			X(Monday)
			X(Tuesday)
		#undef X
		default:
			break;
	}
	return str;
}

And substituting X:

const char *week_day_to_string(enum week_day d) {
	const char *str;
	switch (d) {
		#define X(NAME) case NAME: str = #NAME ;  break;
			case Monday: str = "Monday" ; break;
			case Tuesday: str = "Tuesday" ; break;
		#undef X
		default:
			break;
	}
	return str;
}

The final transformation may look a little magic to those not seasoned with the preprocessor weridness, the #NAME directive translates as “make this a string”.

From these 2 examples, the power and utility of this technique should be evident. To dig a little deeper, one way in which we can make things cleaner is to pull the WEEK_DAY_LIST out into a separate .def file (week-day.def) which looks something like:

X(Monday) \
X(Tuesday)

And would mean we refactor our code accordingly:

enum week_day {
	#define X(NAME) NAME,
	#include "week-day.def"
	#undef X
};

const char *week_day_to_str(enum week_day d) {
	const char *str;
	switch (d) {
		#define X(NAME) case NAME: #NAME ; break;
		#include "week-day.def"
		#undef X
		default: 
			break;
	}
	return str;
}

So in the case of the Token_Kind enum, if we take the definition:

typedef enum token_kind
{
#define X(NAME) TK_##NAME,
  #include "defs/token-kind.def"
#undef X
} Token_Kind;

and peek inside the token-kind.def file (with some lines omitted for brevity):

X(EOF) \
X(ILLEGAL) \
X(LPAREN) \
...
X(VAR) \
X(PRINTLN)

The expansion of our enum results in

typedef enum token_kind
{
	TK_EOF,
	TK_ILLEGAL,
	TK_LPAREN,
	...
	TK_VAR,
	TK_PRINTLN,
} Token_Kind;

It should be noted that the whitespace separation won’t actually be there when the file is subject to preprocessing, it is added for readability, and you may encounter problems with your compiler if you have long expansions - something to do with a max line length amount being exceeded (not something I have encounterd, but is something I have heard and thought I’d pass it on).

Hopefully you can see why this is quite a powerful, and useful technique; I have linked some more resources on X-Macros in the footnotes, if you want some extra reading.⁴⁵⁶⁷⁸

Lexing symbols

The set of symbols currently supported in the language include:

Brackets/parens: ( ) [ ] { }
Punctuation: ;
Operators
- Assignment: =
- Arithmetic: + - * /
- Logical: ! < > <= >= == !=

This is the most boring part to talk about as it’s as simple, in most cases as:

case '(':
{
	token.kind = TK_LPAREN;
} break;

In some cases, such as logical comparison operators (== != <= >=) you have to do some extra checks, take for example the case of != vs !:

case '!':
{
  if (lexer_peek_char(lx) == '=')
  {
    token.kind = TK_NOT_EQ;
    token.literal.len += 1;
    lexer_read_char(lx);
  }
  else
  {
    token.kind = TK_BANG;
  }
} break;

These all require a little more work, mainly because we need to (1) perform an additional advance of the lexer, (2) increment the length of the literal. Beyond that, if other operators are added (such as +=) the existing operator handling code will need to be updated to account for this.

Lexing Identifiers/Keywords

It is relatively trivial to deal with idents and keywords. For starters, under the default case of the lexer, we do the following:

default:
{
  if (is_digit(lx->ch))
  { ... }
  else if (lx->ch == '\"') 
  { ... }
  else if (is_char(lx->ch))
  {
    // NOTE(HS): ident/keyword/builtin parsing
    size_t pos = lx->pos;
    lexer_read_ident_or_keyword(lx);
    size_t len = lx->pos - pos;
    token.literal.len = len;
    token.kind = string_view_to_token_kind(token.literal);
    return token;
  }
} break;

Lexing an identifier/keyword is defined simply to advance the lexer until some non alphanumeric character is encountered:

void lexer_read_ident_or_keyword(Lexer *lx)
{
  while ((is_char(lx->ch) || is_digit(lx->ch)) && lx->ch != '\0')
  {
    lexer_read_char(lx);
  }
}

Then, in order to determine whether the lexed ident it a keyword or not, we run it through a match function:

Token_Kind string_view_to_token_kind(String_View sv)
{
#define lexer_sv_cmp_str(SV, S) \
  ((strlen((S)) == (SV).len) && (strncmp((S), (SV).str, (SV).len) == 0))

  Token_Kind kind;
  if      (lexer_sv_cmp_str(sv, "var"))     { kind = TK_VAR; }
  else if (lexer_sv_cmp_str(sv, "println")) { kind = TK_PRINTLN; }
  else                                      { kind = TK_IDENT; }

  return kind;
}

The main thing to discuss here is the macro, which is just a fancy way of checking:

the length of both the String_View and the C-string are identical
- if you have C-string var and a String_View pointing to variant this prevents variant incorrectly matching
- I had issuses only checking this with strncmp
the actual byte-wise equality of the String_View and string

By default, if no keywords are matched, we assume it is an identifier.

Lexing Number Literals (Integers)

For now, the language only supports integer literals, so we will only deal with their handling. Integers are lexed in the default case of the switch statement:

default:
{
  // TODO(HS): impl float support
  if (is_digit(lx->ch))
  {
    size_t pos = lx->pos;
    lexer_read_number(lx);
    size_t num_len = lx->pos - pos;
    token.kind = TK_INTEGER;
    token.literal.len = num_len;
    return token;
  }
  ...
} break;

The main function here is lexer_read_number, which is pretty simple:

void lexer_read_number(Lexer *lx)
{
  while (is_digit(lx->ch) && lx->ch != '\0')
  {
    lexer_read_char(lx);
  }
}

In essence, until a non-numeric character is encoutered (and if not end of input) just keep advancing the lexer. As noted, there is no support for floats, I have a few ideas on how to add that; there is also no support for alternate integer representations, such as hex notation (0x10), binary (0b1010), or octal (0o12).

Lexing String Literals

Strings are also fairly trivial, however there are a few points of nuance, for instance

should strings be definable by both single (') and double (") quotes?
should string literals be stored containing the original quotes?

I have opted to make strings only definable in this language as by being surrounded by double quotes, and by not storing the source quote marks when the token representing the string is lexed.

Like integer, and idents/keywords, this is handled under the default case:

default:
{
  ...
  else if (lx->ch == '\"')
  {
    // NOTE(HS): "consumes" the surroinding quotes as these aren't needed to store
    // the fact this is a string
    lexer_read_char(lx);
    size_t pos = lx->pos;
    lexer_read_string(lx);
    size_t str_len = lx->pos - pos;
    lexer_read_char(lx);
    
    token.kind = TK_STRING;
    token.location.pos = pos;
    token.literal = (String_View) {
      .str = token.literal.str + 1,
      .len = str_len
    };
  
    return token;
  }
  ...
} break;

Here we advance the lexer, “consuming” the opening quote mark, prior to lexing the output string. Then we call the major function for string processing lexer_read_string. Finally we advance the lexer again to consume the closing quote mark. From this, much like lexing integers and idents/keywords, we can calculate the size of the token literal by subtracting the start point of the lexer from the end. It should be noted that the way I chose to do it is one way this could be done, we could derive the length after consuming the closing quote, this would just require us to fudge the number by subtracting 1; personally I prefer this approach, it’s more symetric with the consume quote -> read -> consume quote method I have implemented in the other default cases.

The function lexer_read_string is quire similar to that of integers, with a few modifications:

void lexer_read_string(Lexer *lx)
{
  while (lx->ch != '\"' && lx->ch != '\0')
  {
    if (lx->ch == '\\' && lexer_peek_char(lx) == '\"')
    {
      lexer_read_char(lx);
    }
    lexer_read_char(lx);
  }
}

One such modification is handling the escaped chars, sort of. The extent of this is to handle \' and \" chars, the handling of \" in particular is to ensure that strings are not prematurely terminalted. This escape sequence handling is hardly comprehensive; the following set of chars is not handled and at some point should be \\, \n, \r, \t, \b, \f, \v, \0.

Handling All Other Input

Finally we have a catchall case, which is an incredibly blunt tool in this first pass, it could possibly be improved but I don’t want to deal with that until I have some actual usage examples of this language. When I say blunt, the code for handling all other input is

default:
{
  ...
  else
  {
    token.kind = TK_ILLEGAL;
  }
  ...
} break;

That’s it. Just set the kind to ILLEGAL, and continue. Now this isn’t something covered in the test cases, mainly because I don’t want to define illegal inputs in the test, since there will be a lot of changes comming in the future, this also is meant (really) to be a flag that “hey something unexpected was found here” for some diagnostics.

Integrating The Lexer

Now for something a little less dry, mainly as we can finally see a more dynamic response than just the tests pass… We’re creating a simple REPL! For now this is just going to take our input, and perform lexical analysis on it, echoing the tokens to the terminal.

Reading User Input - The Traditional C way

There are a handful of ways of reading user input in C, one (and the oldest) is gets. The manpage for gets⁹ reads Never use gets()¹⁰ as it reads into a buffer until a null-terminator (\0) is encountered. This can be problematic as there are no bounds checks performed on the supplied buffer, potentially resulting in buffer overruns (which can lead to a whole series of problems). As a result, this function will not be used.

Safely Reading User Input

There are a few ways available to us, depending on requirements, but the REPL constructed here will use fgets.¹¹

char* fgets( char* restrict str, int count, FILE* restrict stream );

This defines a function which takes a buffer to read input into, the size of the buffer (to perform bounds checks, reading at most count-1 bytes), and the source file stream to read from¹¹. Something else to note about this function is that it will stop parsing when a newline char (\n) is encountered, and the string will contain the newline; parsing also stops if EOF is encountered.

This is not amazing, but will suffice for now - this mainly is annoying as you can’t split statements over multiple lines. If I decide to implement this later, then I might check out getchar¹², coupled with my own bounds checking. I may also be able to get this behaviour with fgets, but I will need to do some experimentation to see what works.

Hooking It All Up

We know how to get user input, we have our function to tokenise input, all we need to do is hook this up so a user can run the program and get some feedback! The design decided on was eventually a single function call from main:

void repl_run(void)
{
  while (true)
  {
    fprintf(stdout, "tyger> ");

    char input_buffer[REPL_INPUT_BUFFER_SIZE];
    memset(input_buffer, '\0', sizeof(input_buffer));

    if (!fgets(input_buffer, sizeof(input_buffer), stdin))
    {
      fprintf(stderr, "[ERROR]: Error getting user input from stdin\n");
      break;
    }

    Lexer lexer;
    lexer_init(&lexer, input_buffer);

    Token token;
    while ((token = lexer_next_token(&lexer)).kind != TK_EOF)
    {
      char token_str[512];
      token_to_string(token, token_str, sizeof(token_str));
      fprintf(stdout, "    %s\n", token_str);
    }
  }
}

This function:

prints a prompt to the user
creates a buffer on the stack and calls memset to zero the array
attempts to read user input into the buffer
initialise the lexer on the stack
call lexer_next_token
compare the kind of token to check if end of input reached
stringify the token
echo the string to stdout

Then to make this available to the user, just add the following to main.c

#include "repl.h"

int main(void)
{
  repl_run();
}

With all that, when we call our executable, we now get the following:

It should be noted that in some of the example usage of fgets I have seen the following line added, to remove the newline:

  if (fgets(buffer, sizeof(buffer), stdin) != NULL) {
    // Remove newline character
    buffer[strcspn(buffer, "\n")] = 0;
    ...
}

In our case this is not important, since we just consume all whitespace anyway, and it has no signigicance in the language (this may be important later when the REPL is expanded, but can be ignored for now).

Closing Thoughts & Further Updates

There’s still a lot of ground to cover, but I think this is a solid foundation to build on. One of the areas lacking at the moment is appropriate error handling, for now the code is just littered with asserts, which in and of itself is not bad, but some asserts may be recoverable errors. I will be looking into alternative error handling methods later, during the next section, parsing.

Repo: https://github.com/hshakeshaft/tyger
Final Commit for entry: fd3c376
Series Page following development

Tyger Programming Language

Mon, 01 Jan 0001 00:00:00 +0000

Tyger is a dynamically typed, interpreted, scripting lanuage I am developing for fun, and education, and to hone my C programming skills. This is currenly a work in progress, and there is still a lot to do, however by the end of it I aim to have, among other things:

support for “regular” language features (such as control flow, and functions)
a functioning Garbage Collector
a module system

You can follow the development here, or check out the repository.