Translate
EnglishFrenchGermanItalianPortugueseRussianSpanish

The XL Engine is and will remain free, donations are purely optional but greatly appreciated.

Recent Comments

In order to facilitate improved modding support and faster iteration times when working on some of the tasks required for the Beta 1 release, I have added scripting support to the engine. Previous builds of DarkXL and DaggerXL already had scripting support but overall I was not happy with the results.

 

Considerations

DarkXL used “Angel Code” for scripting. Angel Code has some nice advantages over some scripting languages, such as Lua, in how nicely it integrates with C/C++ code and provides some nice syntax. However, due to the lack of JIT support, it is very slow when compared to other options such as LuaJIT, Javascript and C#.

I would like to be able to move some of the game code into scripts – such as AI and weapon code – in addition to using it for UI in both the engine and games. So this presents a few issues with the previous solution – I need the overhead of calling script functions and those functions calling back into the engine or game code to be very small, the scripts must execute quickly (ideally no virtual machine), the scripts should have very fast access to data shared with the engine, scripts must be hot-reloadable so no lengthy compilation or optimization steps, the language should have a common syntax that I don’t mind using (since I will be one of the biggest script writers) and it should be small.

Obviously the DarkXL script system, using Angel Code, does not fulfill these goals. LuaJIT is fast compared to other scripting languages but it can be difficult to find errors (syntax errors can stay hidden until code is executed) and garbage collection can be problematic. There are other reasons I would prefer not to use Lua in the this project that I won’t get into. That said, Lua has been successfully used in many projects and has many useful advantages. If you are looking for a scripting language for your own projects, you could do much worse than Lua. Finally C# and Javascript (using V8 or similar) are just too big for my taste, though C# in particular is a nice language.

 

XLC

XLC stands for XL Engine “C”, which uses JIT compiled “C99″ as the basis for the scripting system. In order to improve the scripting experience, the environment is sandboxed (only engine provided functions and services are available) and the API is written in a way that avoids or hides the use of pointers and avoids user memory management. The language is not hobbled, however, and advanced users can use the full power of C. The compiler is built into the engine and compiles code, as needed, directly into memory. This means that the engine can call script functions with very little overhead, memory and data can be shared between scripts and the engine and scripts can call into the engine with the same cost as calling any C or C++ function using a pointer.

The only tool required to write XLC scripts is a text editor. If the engine is running and you edit a script, the engine will automatically hot-reload and recompile the script – which takes a fraction of a second – and you can see the changes immediately. The compilation is fast enough that scripts can easily be included as part of the data – so levels and game areas can have their own scripts in mods. Scripts can also include other scripts in order to import their functionality, variables and structures – allowing scripts to be broken up into files and allowing people to provide encapsulated functionality that anyone can use in their scripts for any game (unless game specific functions are used).

Obviously extremely fast compilation times come at a cost, the generated code isn’t as fast as Visual Studio C++ or GCC optimized code. However performance is much better than debug code, interpreted languages and even a decent improvement over other JIT compiled scripting languages in most cases. In addition the overhead of passing the “script barrier” is much better then the alternatives – you would need to call many tens of thousands of script functions per frame before it starts to become a problem.

And honestly having the engine reload scripts that change automatically while running is pretty awesome for development. It’ll make finishing the UI work so much easier. Something doesn’t look right? Make a small tweak in the text editor, save and see the change instantly. :D  So much better then shutting down the program, making the tweak, recompiling, launching, getting back to the same place again just to find out your tweak wasn’t quite right.

To be clear, all of these features have already been implemented and are currently working. I have already started to move the XL Engine UI over to scripts to gain the iteration time benefits I mentioned above. And being “C” at heart – any game code I want to move to scripts required very little modification to work correctly.

Below is a small test script that I have used to test various features. Lines starting with // are comments, they describe various features being shown. /**/ type comments are also valid. Fixed size types are also included as well as standard C types. Sized types are defined as u/s/f (unsigned/signed/float) + sizeInBits and include: s8,u8,s16,u16,s32,u32,s64,u64,f32,f64. bool is also defined as a type, meaning true or false.

//include any script files that you wish to pull functionality from. All script functions, 
//constants/enums/defines, script global variables and script defined structures will be 
//accessible.
#include "test2.xlc"

//structures defined in the script, this is shorthand for C structure typedefs which 
//can also be used: typedef struct name { ... } name;
Struct(Test2)
{
    int y;
};

Struct(TestStruct)
{
    int x;
    int y;
};

Struct(Vec3)
{
    float x;
    float y;
    float z;
};

//script global variables using both built-in types, structures defined within this
//script and arrays.
Vec3 data0[1024];
Vec3 data1[1024];
Vec3 data2[1024];
int runCount;

//internal script functions - other scripts that include this one can use them but they 
//will not be used by the engine.
f32 blend(f32 t, f32 x, f32 y)
{
    return x + (y-x)*t;
}

void testFunc(string printMe)
{
    xlDebugMessage(printMe);
}

int fib(int n)
{
    if (n <= 2)
    {
        return 1;
    }
    else
    {
        return fib(n-1) + fib(n-2);
    }
}

//"public" functions can be called by the engine. Future tools will be able to list
//all of these, for example you could be editing a level, load a level script and 
//then select these public functions from a list to run them based on various events.

//This function was used to help test a certain kind of performance. If you can't 
//figure out the point, don't worry its only meant to test floating point math 
//performance and function call overhead.
public void perfTest(void)
{
    f32 blendfactor = 0.7594f;
    for (s32 k=0; k<1000; k++)
    {
        f32 value = 2.0f + (float)(k-50)*0.01f;
        //step 1. fill the data with values.
        for (s32 i=0; i<1024; i++)
        {
            data0[i].x = value; value *= 1.25987f;
            data0[i].y = value; value *= 2.25987f;
            data0[i].z = value; value /= 2.25987f;

            data1[i].x = value; value *= 2.25987f;
            data1[i].y = value; value *= 7.25987f;
            data1[i].z = value; value /= 20.25987f;
        }

        //step 2. blend between the values.
        for (s32 i=0; i<1024; i++)
        {
            data2[i].x = blend(blendfactor, data0[i].x, data1[i].x);
            data2[i].y = blend(blendfactor, data0[i].y, data1[i].y);
            data2[i].z = blend(blendfactor, data0[i].z, data1[i].z);
        }
        blendfactor *= 1.001f;
    }
}

//Testing a public script function with a different number of arguments.
public void simpleInc(int a, int b, int c, int d)
{
    runCount++;
}

//xl...() functions are provided by the engine and are available to all scripts.
//sqr() and someVar are both defined in "test2.xlc" and are available since that
//script is included.
public void simple_main(int arg0, int arg1, int arg2)
{
    int r = fib(32);
    xlDebugMessage("fib(32) = %d.", r);
    runCount++;

    //MAX_MAPPING_COUNT is a engine provided define.
    xlDebugMessage("MAX_MAPPING_COUNT = %d", MAX_MAPPING_COUNT);
    xlDebugMessage("Clock = %d", xlGetClock());

    //Using a script defined structure.
    Test2 test;
    test.y = 3;

    //Another script defined structure.
    TestStruct test2;
    test2.x = sqr(test.y) + someVar - 2;
    xlDebugMessage("test2.x = %d.", test2.x);

    //testing string passing.
    testFunc("this is a string.");

    xlDebugMessage("Test inputs: %d, %d, %d", arg0, arg1, arg2);
}

12 Responses to “Introducing XLC – XL Engine Scripting System”

  • ayyyy:

    This is fucking amazing, tbh.

    Never thought I’d see the day old games would be capable of being worked on with such details, specially for missing code titles like Blood.

    Keep up the good work.

  • John:

    Oh man, I am amazed on how quickly you can work Lucius. Keep up the good work! I cannot wait to dig into the XLC when its released!

  • Potato:

    What do you mean when you say no virtual machine? Is the script being compiled to native machine code then? Won’t that make the engine less portable when you have to basically include a native compiler in it? The LuaJIT FAQ says that the reason LuaJIT does not run on every platform Lua runs on is precisely because LuaJIT is a compiler. This effectively limits XL Engine to only support systems you want to support. That’s not a big deal now, but what about a couple of years down the road?

    • luciusDXL:

      Many games use LuaJIT despite the limitations due to the vastly increased performance it offers. That is the same case here (not Lua but the same idea) – this should work on Windows and Linux/Unix based platforms and platforms that use ARM including OS X, WinCE, kFreeBSD and Hurd.

      And if that is still not enough, it wouldn’t be too hard to add a C interpreter to handle any addition platforms – at a substantial loss of performance of course.

      • 111:

        You a seriously, want to support die-things like WinCE?… :) Thats all = time(+of your life)…
        P.S.
        If HW platform unsupported by your-compiler, you can make availability to play – via p-code VM.

        • luciusDXL:

          I’m not planning on support WinCE for the XL Engine but the compiler should support it (i.e. thats not the limiting factor).

  • Ebbpp:

    I’m ware pretty sure this project is dead, good I’ve made a mistake. Maybe I’ll play my beloved Daggerfall with nice graphic at some time after all….

  • 01010101:

    Love it.

  • 111:

    And… where my comment! And(or) your answers for it.

  • xrror:

    Lucius, your attention to the long term scalability of the XL Engine – on top of everything else (this is.. your hobby project? I honestly am afraid of what you do “as a job”) is incredibly impressive.

    I’m writing this because, okay … long story short (I ramble) the latest “crowd-sourced” Unreal Tournament out of curiosity. After “signing up” for an Epic account (*sigh* yeay more personal information) I got the privilege of downloading a “launcher/installer” that surprise! Is basically only downloads and launches “portal/storefront” for other Unreal engine products.

    Except that Unreal Engine is now free to use for personal use. Then it dawned on me…

    The path you are taking lucius with XL Engine. You really are (inadvertently?) creating a potential standard bearer for anyone wanting a cleanly written “retro-engine.”

    I don’t really have a point, I just thought you might find that amusing. Keep up the good work, it is appreciated =D

Leave a Reply

The XL Engine is and will remain free, donations are purely optional but greatly appreciated.