krgamestudios/Toy
1
0
Fork 0
mirror of https://github.com/krgamestudios/Toy.git synced 2026-04-15 14:54:07 +10:00
Code Issues Packages Projects Releases Wiki Activity

Updated Deep Dive Document

Browse Source
This commit is contained in:
Kayne Ruse
2023-07-25 21:44:34 +10:00
committed by GitHub
parent 744b74aed2
commit 7a46c9fc9a
9 changed files with 77 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
deep-dive/building-toy.md
View File

@@ -2,67 +2,30 @@
This tutorial assumes you're using git, GCC, and make.
To embed toy into your program, simply clone the [git repository](https://github.com/Ratstail91/Toy) into a submodule - here we'll assume you called it `Toy`.
To embed toy into your program, simply clone the [git repository](https://github.com/Ratstail91/Toy).
Toy's makefile uses the variable `TOY_OUTDIR` to define where the output of the build command will place the result. You MUST set this to a value, relative to the Toy directory.
Toy's makefile uses the exported variable `TOY_OUTDIR` to define where the output of the build command will place the result. If you're building Toy as a submodule (which is recommended), then you MUST set this value to a directory name, relative to the root directory.
```make
export LIBDIR = lib
export TOY_OUTDIR = ../$(LIBDIR)
export TOY_OUTDIR = out
```
Next, you'll want to run make the from within Toy's `source`, assuming the output directory has been created. There are two options for building Toy - `library` (default) or `static`; the former will create a shared library (and a .dll file on windows), while the latter will create a static library.
```make
toy: $(LIBDIR)
$(MAKE) -C Toy/source
$(LIBDIR):
mkdir $(LIBDIR)
```
Finally, link to the outputted library, and specify the source directory to access the header files.
```make
all: $(OBJ)
$(CC) $(CFLAGS) -o $(OUT) $(OBJ) -L../$(LIBDIR) $(LIBS)
```
Here's a quick example makefile template you can use:
```make
CC=gcc
export OUTDIR = out
export LIBDIR = lib
export TOY_OUTDIR = ../$(LIBDIR)
IDIR+=. ./Toy/source
CFLAGS+=$(addprefix -I,$(IDIR))
LIBS+=-ltoy
ODIR=obj
SRC=$(wildcard *.c)
OBJ=$(addprefix $(ODIR)/,$(SRC:.c=.o))
OUT=./$(OUTDIR)/program
all: toy $(OUTDIR) $(ODIR) $(OBJ)
$(CC) $(CFLAGS) -o $(OUT) $(OBJ) -L$(LIBDIR) $(LIBS)
cp $(LIBDIR)/*.dll $(OUTDIR) # for shared libraries
toy: $(LIBDIR)
toy: $(OUTDIR)
$(MAKE) -C Toy/source
$(OUTDIR):
mkdir $(OUTDIR)
$(LIBDIR):
mkdir $(LIBDIR)
$(ODIR):
mkdir $(ODIR)
$(ODIR)/%.o: %.c
$(CC) -c -o $@ $< $(CFLAGS)
```
Finally, link against the outputted library, with the source directory as the location of the header files.
```make
all: $(OBJ) toy
$(CC) $(CFLAGS) -o $(OUT) $(OBJ) -L$(TOY_OUTDIR) -ltoy
```
These snippets of makefile are only an example - the repository has a more fully featured set of makefiles which can also produce a usable repl program.

24
deep-dive/compiling-toy.md
View File

@@ -2,7 +2,7 @@
This tutorial is a sub-section of [Embedding Toy](deep-dive/embedding-toy) that has been spun off into it's own page for the sake of brevity/sanity. It's recommended that you read the main article first.
The exact phases outlined here are entirely implementation-dependent - that is, they aren't required, and are simply how the canonical version of Toy works.
The exact phases outlined here are entirely implementation-dependent - that is, they aren't required, and are simply how the canonical implementation of Toy works.
## How the Compilation works
@@ -12,7 +12,7 @@ There are four main phases to running a Toy source file. These are:
lexing -> parsing -> compiling -> interpreting
```
Each phase has a dedicated set of functions and structures, and there are intermediate structures between these stages that carry the information from one set to another.
Each phase has a dedicated set of functions and structures, as well as intermediate structures between these that carry information.
```
source -> lexer -> token
@@ -23,7 +23,7 @@ bytecode -> interpreter -> result
## Lexer
Exactly how the source code is loaded into memory is left up to the user, however once it's loaded, it can be bound to a `Lexer` structure.
Exactly how the source code is loaded into a C-string is left up to the user, however once it's loaded, it can be bound to a `Toy_Lexer` structure.
```c
Toy_Lexer lexer;
@@ -47,7 +47,7 @@ Toy_ASTNode* node = Toy_scanParser(&parser);
Toy_freeParser(&parser);
```
The parser takes tokens, one at a time, and converts them into structures called Abstract Syntax Trees, or ASTs for short. Each AST represents a single top-level statement within the Toy script. You'll know when the parser is finished when `Toy_scanParser()` begins returning `NULL` pointers.
The parser pumps the lexer for tokens, one at a time, and converts them into structures called Abstract Syntax Trees (or ASTs for short). Each AST represents a single top-level statement within the Toy script. You'll know when the parser is finished with the lexer's source when `Toy_scanParser()` begins returning `NULL` pointers.
The AST Nodes produced by `Toy_scanParser()` must be freed manually, and the parser itself should not be used again.
@@ -60,22 +60,20 @@ size_t size;
Toy_Compiler compiler;
Toy_initCompiler(&compiler);
Toy_writeCompiler(&compiler, node);
Toy_writeCompiler(&compiler, node); //node is an Toy_ASTNode
unsigned char* tb = Toy_collateCompiler(&compiler, &size);
Toy_freeCompiler(&compiler);
```
The writing step is the process in which AST nodes are compressed into bytecode instructions, while literal values are extracted and placed aside in a cache (usually in an intermediate state).
The writing step is the process in which AST nodes are compressed into bytecode instructions, while literal values are extracted and placed aside in a cache (usually in a compressed, intermediate state).
The collation phase, however is when the bytecode instructions, along with the now flattened intermediate literals and function bodies are combined. The bytecode header specified in [Developing Toy](developing-toy) is placed at the beginning of this blob of bytes during this step.
The collation phase, however is when the bytecode instructions, along with the now flattened intermediate literals and function bodies are combined. The bytecode header specified in [Developing Toy](deep-dive/developing-toy) is placed at the beginning of this blob of bytes during this step.
The Toy bytecode (abbreviated to `tb`), along with the `size` variable indicating the size of the bytecode, are the result of the compilation.
The Toy bytecode (abbreviated to `tb`), along with the `size` variable indicating the size of the bytecode, are the result of the compilation. This bytecode can be saved into a file for later consumption by the host at runtime - you must ensure that any bytecode files have the `.tb` extension.
This bytecode can be saved into a file for later consumption by the host at runtime - ensure that the file has the `.tb` extension.
The bytecode loaded in memory is consumed and freed by `Toy_runInterpreter()`.
Alternatively, the bytecode in memory can be passed directly to the interpreter.
## Interpreter
@@ -88,11 +86,11 @@ Toy_runInterpreter(&interpreter, tb, size);
Toy_freeInterpreter(&interpreter);
```
Exactly how it accomplishes this task is up to it - as long as the result matches expectations.
Exactly how it accomplishes this task is implementation dependant - as long as the results match expectations.
## REPL
An example program, called `toyrepl`, is provided alongside Toy's core. This program can handle many things, such as loading, compiling and executing Toy scripts; it's capable of compiling any valid Toy program for later use, even those that rely on non-standard libraries.
An example program, called `toyrepl`, is provided alongside Toy's core. This program can handle many things, such as loading, compiling and executing Toy scripts; it's capable of compiling any valid Toy program for later use, even those that rely on non-standard libraries. It also has a number of commonly needed libraries provided.
To get a list of options, run `toyrepl -h`.

12
deep-dive/developing-toy.md
View File

@@ -1,12 +1,12 @@
# Developing Toy
Toy's current version began as a specification, which changed and evolved as it was developed. The original specification was extremely bare-bones and not intended for a general audience, so this website is actually intended not just to teach how to use it, but also aspects of how to expand on the current canonical version of the language.
Here you'll find some of the implementation details, which must remain intact regardless of any other changes.
Here you'll find some of the implementation details.
## Bytecode Header Format
Every instance of Toy bytecode will be divvied up into several sections, by necessity - however the only one with an actual concrete specification is the header. This section is used to define what version of Toy is currently running, as well as to prevent any future version clashes.
The bytecode header format must not change.
Every instance of Toy bytecode will be divided up into several sections, by necessity - however the first one to be read is the header. This section is used to define what version of Toy is currently running, as well as to prevent any future version/fork clashes.
The header consists of four values:
@@ -15,7 +15,7 @@ The header consists of four values:
* TOY_VERSION_PATCH
* TOY_VERSION_BUILD
The first three are single unsigned bytes, embedded at the beginning of the bytecode in sequence. These represent the major, minor and patch versions of the language. The fourth value is a null-terminated string of unspecified data, which is *intended* but not required to specify the time that the langauge's compiler was itself compiled. The build string can also hold arbitrary data, such as the current maintainer's name, current fork of the language, or other versioning info.
The first three are single unsigned bytes, embedded at the beginning of the bytecode in sequence. These represent the major, minor and patch versions of the language. The fourth value is a null-terminated string of unspecified data, which is *intended* but not required to specify the time that the langauge's compiler was itself compiled. The build string can hold arbitrary data, such as the current maintainer's name, current fork of the language, or other versioning info.
There are some strict rules when interpreting these values (mimicking, but not conforming to [semver.org](https://semver.org/)):
@@ -26,4 +26,4 @@ There are some strict rules when interpreting these values (mimicking, but not c
All interpreter implementations retain the right to reject any bytecode whose header data does not conform to the above specification.
The latest version information can be found in [toy_common.h](https://github.com/Ratstail91/Toy/blob/main/source/toy_common.h#L7-L10)
The latest version information can be found in [toy_common.h](https://github.com/Ratstail91/Toy/blob/main/source/toy_common.h)

10
deep-dive/embedding-toy.md
View File

@@ -8,17 +8,17 @@ Be aware that when you create a new Literal object, you must call `Toy_freeLiter
## Embedded API Macros
The functions intended for usage by the API are prepended with the C macro `TOY_API`. The exact value of this macro can vary by platform, or even be empty. In addition, the macros defined in [literal.h](https://github.com/Ratstail91/Toy/blob/0.6.0/source/literal.h) are available for use when manipulating literals. These include:
The functions intended for usage by the API are prepended with the C macro `TOY_API`. The exact value of this macro can vary by platform, or even be empty. In addition, the macros defined in [literal.h](https://github.com/Ratstail91/Toy/blob/main/source/toy_literal.h) are available for use when manipulating literals. These include:
* `TOY_IS_*` - check if a literal is a specific type
* `TOY_AS_*` - cast the literal to a specific type
* `TOY_TO_*` - create a literal of a specific type
* `TOY_TO_*_LITERAL` - create a literal of a specific type
* `TOY_IS_TRUTHY` - check if a literal is truthy
* `TOY_MAX_STRING_LENGTH` - the maximum length of a string in Toy (can be altered if needed)
## Structures Used Throughout Toy
The main unit of data within Toy's internals is `Toy_Literal`, which can contain any value that can exist within the Toy langauge. The exact implementation of `Toy_Literal` may change or evolve as time goes on, so it's recommended that you only interact with literals directly by using the macros and functions outlined [above](#embedded-api-macros). See the [types](types) page for information on what datatypes exist in Toy.
The main unit of data within Toy's internals is `Toy_Literal`, which can contain any value that can exist within the Toy langauge - even identifiers. The exact implementation of `Toy_Literal` may change or evolve as time goes on, so it's recommended that you only interact with literals directly by using the macros and functions outlined [above](#embedded-api-macros). See the [types](types) page for information on exactly what datatypes exist in Toy.
There are two main "compound structures" used within Toy's internals - the `Toy_LiteralArray` and `Toy_LiteralDictionary`. The former is an array of `Toy_Literal` instances stored sequentially in memory for fast lookups, while the latter is a key-value hashmap designed for efficient lookups based on a `Toy_Literal` key. These are both accessible via the language as well.
@@ -26,7 +26,7 @@ These compound structures hold **copies** of literals given to them, rather than
## Compiling Toy Scripts
Please see [Compiling Toy](compiling-toy) for more information on the process of turning scripts into bytecode.
Please see [Compiling Toy](deep-dive/compiling-toy) for more information on the process of turning scripts into bytecode.
## Interpreting Toy
@@ -40,7 +40,7 @@ Toy_Interpreter interpreter;
Toy_initInterpreter(&interpreter);
Toy_runInterpreter(&interpreter, tb, size);
Toy_resetInterpreter(&interpreter); //You usually want to reset between runs
Toy_resetInterpreter(&interpreter); //You usually want to reset between runs with the same interpreter
Toy_freeInterpreter(&interpreter);
```

40
deep-dive/roadmap.md
View File

@@ -1,40 +0,0 @@
# Roadmap
There's a few things I'd like to do with the langauge, namely bugfixes, new features and implementing a game with it.
## Game And Game Engine
The Toy programming langauge was designed from the beginning as though it was supposed to be embedded into an imaginary game engine. Well, now that the lang is nearly feature complete, it's time to start on that engine.
To that end, I've begun working on this: [Airport Game](https://github.com/Ratstail91/airport).
This is a simple game concept, which I can implement within a reasonable amount of time, before extracting parts to create the engine proper. It feels almost like a mobile game, so I'm hoping this engine will be runnable on android (though at the time of writing, I've yet to investigate how).
## New Features I'd Like One Day
Some things I'd like to add in the future include:
* ~~A fully featured standard library (see below)~~
* ~~An external script runner utility library~~
* A threading library
* A random generation library (numbers, perlin noise, wave function collapse?)
* ~~A timer library~~
* Multiple return values from functions
* ~~Ternary operator~~
* interpolated strings
* ~~MSVC compilation~~
Some of these have always been planned, but were sidelined or are incomplete for one reason or another.
## Nope Features
Some things I simply don't want to include at the current time include:
* Classes & Structures
* Do-while loops
This is because reworking the internals to add an entirely new system like this would be incredibly difficult for very little gain.
Ironically, I've never used a do-while loop seriously until I started implementing this language.

29
deep-dive/roadmapping-toy.md Normal file
View File

@@ -0,0 +1,29 @@
# Roadmapping Toy
## Game And Game Engine
The Toy programming langauge was designed from the beginning as though it was supposed to be embedded into an imaginary game engine. Development on said engine and an associated game have proceeded smoothly so far.
## Microprocessor Support
A fork of the language intended to run on microprocessors is currently under development by 3rd parties. Support and advice will be provided to them where needed.
## Nice To Have Features
Potential future additions include:
* A threading library
* A timer library
* More random generation libraries (numbers, perlin noise, wave function collapse?)
* Multiple return values from functions
* Interpolated strings
Some of these have always been planned, but were sidelined or are incomplete for one reason or another.
## Nope Features
Some things which simply will not be added in the foreseeable future are:
* Classes & Structures
* Do-while loops

6
deep-dive/testing-toy.md
View File

@@ -1,7 +1,5 @@
# Testing Toy
Toy uses GitHub CI for comprehensive automated testing - however, all of the tests are under `test/`, and can be executed by running `make test`. Doing so on linux will attempt to use valgrind; to disable using valgrind, pass in `DISABLE_VALGRIND=true` as an environment variable.
Toy uses GitHub CI/CD for comprehensive automated testing - however, all of the tests are under `test/`, and can be executed by running `make test`. Doing so on linux will attempt to use valgrind; to disable using valgrind, pass in `DISABLE_VALGRIND=true` as an environment variable. GitHub CI also has access to the option `make test-sanitized` which attempts to use memory sanitation.
The tests consist of a number of different situations and edge cases which have been discovered, and should probably be thoroughly tested one way or another. There are also several "-bugfix.toy" scripts which explicitly test a bug that has been encountered in one way or another. The libs that are stored in `repl/` are also tested - their tests are under `/tests/scripts/lib`; some error cases are also checked by the mustfail tests in `/test/scripts/mustfail`.
GitHub CI also has access to the option `make test-sanitized` which attempts to use memory sanitation. I don't know enough about this to offer much comentary, only that several invisible issues are monitored this way.
The tests consist of a number of different situations and edge cases which have been discovered, and should probably be thoroughly tested one way or another. There are also several "-bugfix.toy" scripts which explicitly test a bug that has been encountered in one way or another, to prevent regressions. The libs that are stored in `repl/` are also tested - their tests are under `/tests/scripts/lib`; some error cases are also checked by the mustfail tests in `/test/scripts/mustfail`.

28
deep-dive/theorizing-toy.md
View File

@@ -1,32 +1,24 @@
# Theorizing Toy
Sooner or later, every coder will try to create their own programming language. In my case, it took me over a decade and a half to realize that was an option, but once I did I read through a fantastic book called [Crafting Interpreters](https://craftinginterpreters.com/). This sent me down the rabbit hole, so to speak.
Sooner or later, every coder will try to create their own programming language. In my case, it took me over a decade and a half to realize that was even an option, but once I did I read through a fantastic book called [Crafting Interpreters](https://craftinginterpreters.com/). This sent me down the rabbit hole, so to speak.
Now, several years later, after multiple massive revisions, I think my language is nearing version 1.0 - not bad for a side project.
The main driving idea behind the Toy programming langauge has remained the same from the very beginning - I wanted a scripting language that could be embedded into a larger host program, to allow for easy modification by the end user. Specifically, I wanted to enable easy modding of video games made in an imaginary game engine.
The main driving idea behind the language has remained the same from the very beginning. I wanted a scripting language that could be embedded into a larger host program, which could be easily modified by the end user. Specifically, I wanted to enable easy modding for games made in an imaginary game engine.
At the time of writing, I've started working on said engine, building it around Toy, and adjusting Toy to fit the engine as needed. I've also begun working on a game within that engine, as I believe the best way to build an engine is to build a game with it first. The engine has been dubbed "Box", and the game is called "Skylands".
At the time of writing, I've started working on said engine, building it around Toy and adjusting Toy to fit the engine as needed. I don't know how long the engine's development will take, but I personally think the best way to build an engine is to build a game first, and then extract the engine from it. Thus, the project is currently called "airport", though the engine proper will likely have a name like "box" or "toybox".
But this post isn't about the engine, it's about Toy - I want to explain, in some detail, my thought processes when developing it. Let's start at the beginning.
But this post isn't about the engine, it's about Toy - I want to explain, in some detail, my thought processes when developing it. Let's start at the beginning:
```
```toy
print "Hello world";
```
I've drawn the `print` keyword from Crafting Interpreter's Lox language, for much the same reason as explained in the book - it's a simple and easy way to debug issues - it's not intended for actual production use. You'll be able to print out any kind of value or variable from this statement - but it loses some context like function implementations and the values of `opaque` literals.
I've drawn the `print` keyword from Crafting Interpreter's Lox language, for much the same reason as explained in the book - it's a simple and easy way to debug issues. You'll be able to print out any kind of value or variable from this statement - but it loses some context like function implementations, and the values of `opaque` literals.
Let's touch on variables quickly - There's about a dozen variable types that can be used, depending on how you count them. This grew as the language progressed, and there are actually several literal types which you can't directly access (they're only used internally).
Let's touch on variables quickly - There's about a dozen variable types that can be used, depending on how you count them. They include `bool`, `int`, `float`, `string` and a couple of compound types - but strict typing in Toy is completely optional (`any` is used by default). There are also functions, which are reusable chunks of code, and a pretty standard set of operators with their traditional precedences.
There's also functions (which are also a type of literal), which are reusable chunks of bytecode that can be invoked with their names. OH! I haven't even talked about bytecode yet - one of the interesting aspects of Toy is that the source code must be compiled into an intermediate bytecode format (a trait also inherited from Lox) before it can be executed by the interpreter. Even I'm not entirely sure how the internals of how the bytecode is layed out in it's entirety, as the parsing and compilation steps take liberties when producing the final usable chunk of memory.
One way in which Toy stands out is the bytecode compilation step. Before execution, the source code must be compiled into an intermediate bytecode format (a trait also inherited from Lox) before it can be executed by the interpreter. The exact specifications of the bytecode formatting are not currently documented (yet). The intermediate bytecode stage, and the independance of the interpreter from the compiler, also allow unique features such as the possiblity of operating on a microcontroller.
I was originally not entirely certain that compiling to bytecode was the right choice as, for most programs to function and remain moddable, the source will need to be compiled on-the-fly. But after extensive benchmarking, it turns out that the compilation is the fastest part of execution.
One major native feature which is missing from Toy is an input system, such as from stdin. Instead, Toy is intended to receive its instructions from the host program, including any input needed. One such example would be a game controller library - something which takes in button presses, and calls certain Toy functions to move a character around the game world. Toy is almost infinitely extensible via the C API's hook injection system.
There's one major feature of most programming languages that Toy is missing - input. Currently, there's no standardized way to receive input from the user - however this will likely be aleviated by various custom libraries down the road.
One such example would be a game controller library - something which takes in button presses, and calls certain Toy functions to move a character around the game world. The thing is, not every game will need a controller - that's why each library is optional, and can be provided or omitted at the host's discretion. As a result, Toy is almost infinitely extensible, as most good scripting languages are.
I don't know how well this langauge will do in the wild, once it gets some battle testing from actual users - but I do know that it'll become more and more of a grizzled beast as time goes on - that's inevitable for any piece of code. However, I would like to keep the core language nice and simple, as much as possible - something you can explain with just the quickstart page.
Feedback, and constructive criticism are always welcome.
I would like to keep the core language nice and simple, as much as possible - something you can explain with just the quickstart page. However, feedback and criticism are always welcome.

4
index.md
View File

@@ -50,7 +50,7 @@ print tally(); //3
* [Random Library](getting-started/random-library)
* [Runner Library](getting-started/runner-library)
## Deep Dive
## Deep Dive Document
* [Theorizing Toy](deep-dive/theorizing-toy)
* [Building Toy](deep-dive/building-toy)
@@ -58,7 +58,7 @@ print tally(); //3
* [Compiling Toy](deep-dive/compiling-toy)
* [Developing Toy](deep-dive/developing-toy)
* [Testing Toy](deep-dive/testing-toy)
* [Roadmap](deep-dive/roadmap)
* [Roadmapping Toy](deep-dive/roadmapping-toy)
## Public C API