Introducing XLC – XL Engine Scripting System

XL Engine News
luciusDXL
XL Engine Developer / Site Admin
XL Engine Developer / Site Admin
Posts: 778
Joined: Thu Apr 14, 2011 4:05 pm
Contact:

Introducing XLC – XL Engine Scripting System

Postby luciusDXL » Sat Feb 20, 2016 11:19 pm

In order to facilitate improved modding support and faster iteration times when working on some of the tasks required for the Beta 1 release, I have added scripting support to the engine. Previous builds of DarkXL and DaggerXL already had scripting support but overall I was not happy with the results.


Considerations
DarkXL used “Angel Code” for scripting. Angel Code has some nice advantages over some scripting languages, such as Lua, in how nicely it integrates with C/C++ code and provides some nice syntax. However, due to the lack of JIT support, it is very slow when compared to other options such as LuaJIT, Javascript and C#.

I would like to be able to move some of the game code into scripts – such as AI and weapon code – in addition to using it for UI in both the engine and games. So this presents a few issues with the previous solution – I need the overhead of calling script functions and those functions calling back into the engine or game code to be very small, the scripts must execute quickly (ideally no virtual machine), the scripts should have very fast access to data shared with the engine, scripts must be hot-reloadable so no lengthy compilation or optimization steps, the language should have a common syntax that I don’t mind using (since I will be one of the biggest script writers) and it should be small.

Obviously the DarkXL script system, using Angel Code, does not fulfill these goals. LuaJIT is fast compared to other scripting languages but it can be difficult to find errors (syntax errors can stay hidden until code is executed) and garbage collection can be problematic. There are other reasons I would prefer not to use Lua in the this project that I won’t get into. That said, Lua has been successfully used in many projects and has many useful advantages. If you are looking for a scripting language for your own projects, you could do much worse than Lua. Finally C# and Javascript (using V8 or similar) are just too big for my taste, though C# in particular is a nice language.


XLC
XLC stands for XL Engine “C”, which uses JIT compiled “C99″ as the basis for the scripting system. In order to improve the scripting experience, the environment is sandboxed (only engine provided functions and services are available) and the API is written in a way that avoids or hides the use of pointers and avoids user memory management. The language is not hobbled, however, and advanced users can use the full power of C. The compiler is built into the engine and compiles code, as needed, directly into memory. This means that the engine can call script functions with very little overhead, memory and data can be shared between scripts and the engine and scripts can call into the engine with the same cost as calling any C or C++ function using a pointer.

The only tool required to write XLC scripts is a text editor. If the engine is running and you edit a script, the engine will automatically hot-reload and recompile the script – which takes a fraction of a second – and you can see the changes immediately. The compilation is fast enough that scripts can easily be included as part of the data – so levels and game areas can have their own scripts in mods. Scripts can also include other scripts in order to import their functionality, variables and structures – allowing scripts to be broken up into files and allowing people to provide encapsulated functionality that anyone can use in their scripts for any game (unless game specific functions are used).

Obviously extremely fast compilation times come at a cost, the generated code isn’t as fast as Visual Studio C++ or GCC optimized code. However performance is much better than debug code, interpreted languages and even a decent improvement over other JIT compiled scripting languages in most cases. In addition the overhead of passing the “script barrier” is much better then the alternatives – you would need to call many tens of thousands of script functions per frame before it starts to become a problem.

And honestly having the engine reload scripts that change automatically while running is pretty awesome for development. It’ll make finishing the UI work so much easier. Something doesn’t look right? Make a small tweak in the text editor, save and see the change instantly. :D So much better then shutting down the program, making the tweak, recompiling, launching, getting back to the same place again just to find out your tweak wasn’t quite right.

To be clear, all of these features have already been implemented and are currently working. I have already started to move the XL Engine UI over to scripts to gain the iteration time benefits I mentioned above. And being “C” at heart – any game code I want to move to scripts required very little modification to work correctly.

Below is a small test script that I have used to test various features. Lines starting with // are comments, they describe various features being shown. /**/ type comments are also valid. Fixed size types are also included as well as standard C types. Sized types are defined as u/s/f (unsigned/signed/float) + sizeInBits and include: s8,u8,s16,u16,s32,u32,s64,u64,f32,f64. bool is also defined as a type, meaning true or false.

Code: Select all

//include any script files that you wish to pull functionality from. All script functions,
//constants/enums/defines, script global variables and script defined structures will be
//accessible.
#include "test2.xlc"

//structures defined in the script, this is shorthand for C structure typedefs which
//can also be used: typedef struct name { ... } name;
Struct(Test2)
{
    int y;
};

Struct(TestStruct)
{
    int x;
    int y;
};

Struct(Vec3)
{
    float x;
    float y;
    float z;
};

//script global variables using both built-in types, structures defined within this
//script and arrays.
Vec3 data0[1024];
Vec3 data1[1024];
Vec3 data2[1024];
int runCount;

//internal script functions - other scripts that include this one can use them but they
//will not be used by the engine.
f32 blend(f32 t, f32 x, f32 y)
{
    return x + (y-x)*t;
}

void testFunc(string printMe)
{
    xlDebugMessage(printMe);
}

int fib(int n)
{
    if (n <= 2)
    {
        return 1;
    }
    else
    {
        return fib(n-1) + fib(n-2);
    }
}

//"public" functions can be called by the engine. Future tools will be able to list
//all of these, for example you could be editing a level, load a level script and
//then select these public functions from a list to run them based on various events.

//This function was used to help test a certain kind of performance. If you can't
//figure out the point, don't worry its only meant to test floating point math
//performance and function call overhead.
public void perfTest(void)
{
    f32 blendfactor = 0.7594f;
    for (s32 k=0; k<1000; k++)
    {
        f32 value = 2.0f + (float)(k-50)*0.01f;
        //step 1. fill the data with values.
        for (s32 i=0; i<1024; i++)
        {
            data0[i].x = value; value *= 1.25987f;
            data0[i].y = value; value *= 2.25987f;
            data0[i].z = value; value /= 2.25987f;

            data1[i].x = value; value *= 2.25987f;
            data1[i].y = value; value *= 7.25987f;
            data1[i].z = value; value /= 20.25987f;
        }

        //step 2. blend between the values.
        for (s32 i=0; i<1024; i++)
        {
            data2[i].x = blend(blendfactor, data0[i].x, data1[i].x);
            data2[i].y = blend(blendfactor, data0[i].y, data1[i].y);
            data2[i].z = blend(blendfactor, data0[i].z, data1[i].z);
        }
        blendfactor *= 1.001f;
    }
}

//Testing a public script function with a different number of arguments.
public void simpleInc(int a, int b, int c, int d)
{
    runCount++;
}

//xl...() functions are provided by the engine and are available to all scripts.
//sqr() and someVar are both defined in "test2.xlc" and are available since that
//script is included.
public void simple_main(int arg0, int arg1, int arg2)
{
    int r = fib(32);
    xlDebugMessage("fib(32) = %d.", r);
    runCount++;

    //MAX_MAPPING_COUNT is a engine provided define.
    xlDebugMessage("MAX_MAPPING_COUNT = %d", MAX_MAPPING_COUNT);
    xlDebugMessage("Clock = %d", xlGetClock());

    //Using a script defined structure.
    Test2 test;
    test.y = 3;

    //Another script defined structure.
    TestStruct test2;
    test2.x = sqr(test.y) + someVar - 2;
    xlDebugMessage("test2.x = %d.", test2.x);

    //testing string passing.
    testFunc("this is a string.");

    xlDebugMessage("Test inputs: %d, %d, %d", arg0, arg1, arg2);
}
User avatar
jet800
Senior-Member
Posts: 748
Joined: Fri Apr 15, 2011 10:46 am
Location: Moscow, Russia

Re: Introducing XLC – XL Engine Scripting System

Postby jet800 » Sun Feb 21, 2016 5:06 am

Sounds good, but could you show some larger examples regarding of how user managed memory is avoided, how script values are being (or not) reset during reload, what happens when script does not respond. How script interoperability is done, how versioning is implemented(if any).
luciusDXL
XL Engine Developer / Site Admin
XL Engine Developer / Site Admin
Posts: 778
Joined: Thu Apr 14, 2011 4:05 pm
Contact:

Re: Introducing XLC – XL Engine Scripting System

Postby luciusDXL » Sun Feb 21, 2016 11:37 am

jet800 wrote:Sounds good, but could you show some larger examples regarding of how user managed memory is avoided, how script values are being (or not) reset during reload, what happens when script does not respond. How script interoperability is done, how versioning is implemented(if any).
Those sound like great topics for a future post and go deeper then an "introduction." Moving the UI code into scripts will be the first "real" usage of the system, I plan on documenting the system once that is complete and I will be sure to cover these topics in addition to other information people need to use this.
User avatar
BrotherBrick
Member
Posts: 60
Joined: Fri Oct 28, 2011 12:03 am
Contact:

Re: Introducing XLC – XL Engine Scripting System

Postby BrotherBrick » Mon Feb 22, 2016 2:16 am

Would you be willing to split XLC out into a library?

OpenMW has been wrestling with this same exact problem: python, lua, angelscript... etc.
We haven't settled on one yet, but it would be interesting to see this jointly developed.

Sound interesting?
Chris
Member
Posts: 106
Joined: Fri Apr 15, 2011 5:06 pm

Re: Introducing XLC – XL Engine Scripting System

Postby Chris » Mon Feb 22, 2016 8:35 pm

luciusDXL wrote:DarkXL used “Angel Code” for scripting. Angel Code has some nice advantages over some scripting languages, such as Lua, in how nicely it integrates with C/C++ code and provides some nice syntax. However, due to the lack of JIT support, it is very slow when compared to other options such as LuaJIT, Javascript and C#.

From what I understand, AngelScript can use JIT compiling. It can also reuse compiled code modules, can use precompiled bytecode (e.g. cached scripts), applies some optimizations, can directly link up data structures (i.e. reading/writing to a script variable can directly access a program variable), and scripts can directly call back to registered engine functions.

I was actually interested in moving OpenMW's scripting system to use AngelScript as a base, but was only held back by how OpenMW's scripting engine is currently set up (I'd probably have a bit of an easier time if I was starting fresh).
luciusDXL
XL Engine Developer / Site Admin
XL Engine Developer / Site Admin
Posts: 778
Joined: Thu Apr 14, 2011 4:05 pm
Contact:

Re: Introducing XLC – XL Engine Scripting System

Postby luciusDXL » Mon Feb 22, 2016 9:10 pm

Chris wrote:
luciusDXL wrote:DarkXL used “Angel Code” for scripting. Angel Code has some nice advantages over some scripting languages, such as Lua, in how nicely it integrates with C/C++ code and provides some nice syntax. However, due to the lack of JIT support, it is very slow when compared to other options such as LuaJIT, Javascript and C#.
From what I understand, AngelScript can use JIT compiling. It can also reuse compiled code modules, can use precompiled bytecode (e.g. cached scripts), applies some optimizations, can directly link up data structures (i.e. reading/writing to a script variable can directly access a program variable), and scripts can directly call back to registered engine functions.

I was actually interested in moving OpenMW's scripting system to use AngelScript as a base, but was only held back by how OpenMW's scripting engine is currently set up (I'd probably have a bit of an easier time if I was starting fresh).
Unfortunately the JIT solution is not developed alongside the scripting system, the only solution I know about doesn't work with the latest version. When I did use Angel Code there were issues I had that were fixed with later versions of the library, so I'm not comfortable sticking with an earlier version for JIT support. Cached scripts are great but still do not provide the same level of performance as scripting systems with JIT support and XLC is no exception.

Granted I'm sure this will be resolved with time, I've read that the original author wants to develop a "built-in" JIT solution and the BlindMindStudios guys have shown interest in updating their JIT compiler. However I decided to go a different route for the XL Engine rather than waiting to see how this pans out.

That said, AngelScript does have some nice advantages and syntax - so I see no reason not to use it for Open MW.


BrotherBrick wrote:Would you be willing to split XLC out into a library?

OpenMW has been wrestling with this same exact problem: python, lua, angelscript... etc.
We haven't settled on one yet, but it would be interesting to see this jointly developed.

Sound interesting?
I read the responses on the OpenMW forum and the consensus, when I looked at it at least, was that XLC is not the right solution for that project. In addition the code is not mature and tested enough that I would feel comfortable suggesting that it be used in other projects. Finally I honestly don't want to have to worry about the needs of other projects while working on the XL Engine at this time.
User avatar
jet800
Senior-Member
Posts: 748
Joined: Fri Apr 15, 2011 10:46 am
Location: Moscow, Russia

Re: Introducing XLC – XL Engine Scripting System

Postby jet800 » Tue Feb 23, 2016 10:44 am

I've just checked wiki. So each scripting tree of includes would result in copying of each included script, right?
What about multi-threading? And filesystem access, you've mentioned it's sandboxed runtime but up to which level?
luciusDXL
XL Engine Developer / Site Admin
XL Engine Developer / Site Admin
Posts: 778
Joined: Thu Apr 14, 2011 4:05 pm
Contact:

Re: Introducing XLC – XL Engine Scripting System

Postby luciusDXL » Tue Feb 23, 2016 8:39 pm

jet800 wrote:I've just checked wiki. So each scripting tree of includes would result in copying of each included script, right?
What about multi-threading? And filesystem access, you've mentioned it's sandboxed runtime but up to which level?
About sandboxing, the only functions available are those provided by the engine or game code. Currently I have exposed a limited set of file manipulation functions but these call engine functions rather than system functions, allowing for safety checks. There is no way to include non-XLC code from the scripts or get system level access. As for what the final API looks like, that is still a work in progress.

To be brief script execution can be threaded at the script module level and scripts can have coroutine execution behavior - allowing for suspends and delays (similar to ACS or Lua coroutines). Of course there is a scheduling system and the system will be able to monitor execution times in order to forcibly stop scripts that are taking too long. Anyway I will post more details in the documentation soon.

Edit: I updated the XLC documentation to talk a bit about script execution.
User avatar
jet800
Senior-Member
Posts: 748
Joined: Fri Apr 15, 2011 10:46 am
Location: Moscow, Russia

Re: Introducing XLC – XL Engine Scripting System

Postby jet800 » Wed Feb 24, 2016 1:51 pm

Aha, thank you! I've got the idea. So this will be actually quite restricted(as engine scripting language should be), but full featured in terms of language constructs available. Co-routines look good, however what are limits on effective amount of scripts per module till it will degrate, given each script is quite lightweight(less than 100 floating point operations)? Would be great to see some performance results later on.
Looking forward to it! :)
luciusDXL
XL Engine Developer / Site Admin
XL Engine Developer / Site Admin
Posts: 778
Joined: Thu Apr 14, 2011 4:05 pm
Contact:

Re: Introducing XLC – XL Engine Scripting System

Postby luciusDXL » Thu Feb 25, 2016 12:37 am

jet800 wrote:Aha, thank you! I've got the idea. So this will be actually quite restricted(as engine scripting language should be), but full featured in terms of language constructs available. Co-routines look good, however what are limits on effective amount of scripts per module till it will degrate, given each script is quite lightweight(less than 100 floating point operations)? Would be great to see some performance results later on.
Looking forward to it! :)
Thanks. I obviously don't know what the limits are yet, its still too early to tell.

I updated the documentation with more examples and information on how to synchronize between scripts using signals, see the XLC Documentation on the wiki for more details. If you are interested in how the scripting system develops be sure to check out the wiki regularly, I will be updating it over time.

Return to “News”

Who is online

Users browsing this forum: No registered users and 0 guests