Translate
EnglishFrenchGermanItalianPortugueseRussianSpanish

The XL Engine is and will remain free, donations are purely optional but greatly appreciated.

Recent Comments

Archive for February 2016

In order to facilitate improved modding support and faster iteration times when working on some of the tasks required for the Beta 1 release, I have added scripting support to the engine. Previous builds of DarkXL and DaggerXL already had scripting support but overall I was not happy with the results.

 

Considerations

DarkXL used “Angel Code” for scripting. Angel Code has some nice advantages over some scripting languages, such as Lua, in how nicely it integrates with C/C++ code and provides some nice syntax. However, due to the lack of JIT support, it is very slow when compared to other options such as LuaJIT, Javascript and C#.

I would like to be able to move some of the game code into scripts – such as AI and weapon code – in addition to using it for UI in both the engine and games. So this presents a few issues with the previous solution – I need the overhead of calling script functions and those functions calling back into the engine or game code to be very small, the scripts must execute quickly (ideally no virtual machine), the scripts should have very fast access to data shared with the engine, scripts must be hot-reloadable so no lengthy compilation or optimization steps, the language should have a common syntax that I don’t mind using (since I will be one of the biggest script writers) and it should be small.

Obviously the DarkXL script system, using Angel Code, does not fulfill these goals. LuaJIT is fast compared to other scripting languages but it can be difficult to find errors (syntax errors can stay hidden until code is executed) and garbage collection can be problematic. There are other reasons I would prefer not to use Lua in the this project that I won’t get into. That said, Lua has been successfully used in many projects and has many useful advantages. If you are looking for a scripting language for your own projects, you could do much worse than Lua. Finally C# and Javascript (using V8 or similar) are just too big for my taste, though C# in particular is a nice language.

 

XLC

XLC stands for XL Engine “C”, which uses JIT compiled “C99″ as the basis for the scripting system. In order to improve the scripting experience, the environment is sandboxed (only engine provided functions and services are available) and the API is written in a way that avoids or hides the use of pointers and avoids user memory management. The language is not hobbled, however, and advanced users can use the full power of C. The compiler is built into the engine and compiles code, as needed, directly into memory. This means that the engine can call script functions with very little overhead, memory and data can be shared between scripts and the engine and scripts can call into the engine with the same cost as calling any C or C++ function using a pointer.

The only tool required to write XLC scripts is a text editor. If the engine is running and you edit a script, the engine will automatically hot-reload and recompile the script – which takes a fraction of a second – and you can see the changes immediately. The compilation is fast enough that scripts can easily be included as part of the data – so levels and game areas can have their own scripts in mods. Scripts can also include other scripts in order to import their functionality, variables and structures – allowing scripts to be broken up into files and allowing people to provide encapsulated functionality that anyone can use in their scripts for any game (unless game specific functions are used).

Obviously extremely fast compilation times come at a cost, the generated code isn’t as fast as Visual Studio C++ or GCC optimized code. However performance is much better than debug code, interpreted languages and even a decent improvement over other JIT compiled scripting languages in most cases. In addition the overhead of passing the “script barrier” is much better then the alternatives – you would need to call many tens of thousands of script functions per frame before it starts to become a problem.

And honestly having the engine reload scripts that change automatically while running is pretty awesome for development. It’ll make finishing the UI work so much easier. Something doesn’t look right? Make a small tweak in the text editor, save and see the change instantly. :D  So much better then shutting down the program, making the tweak, recompiling, launching, getting back to the same place again just to find out your tweak wasn’t quite right.

To be clear, all of these features have already been implemented and are currently working. I have already started to move the XL Engine UI over to scripts to gain the iteration time benefits I mentioned above. And being “C” at heart – any game code I want to move to scripts required very little modification to work correctly.

Below is a small test script that I have used to test various features. Lines starting with // are comments, they describe various features being shown. /**/ type comments are also valid. Fixed size types are also included as well as standard C types. Sized types are defined as u/s/f (unsigned/signed/float) + sizeInBits and include: s8,u8,s16,u16,s32,u32,s64,u64,f32,f64. bool is also defined as a type, meaning true or false.

//include any script files that you wish to pull functionality from. All script functions, 
//constants/enums/defines, script global variables and script defined structures will be 
//accessible.
#include "test2.xlc"

//structures defined in the script, this is shorthand for C structure typedefs which 
//can also be used: typedef struct name { ... } name;
Struct(Test2)
{
    int y;
};

Struct(TestStruct)
{
    int x;
    int y;
};

Struct(Vec3)
{
    float x;
    float y;
    float z;
};

//script global variables using both built-in types, structures defined within this
//script and arrays.
Vec3 data0[1024];
Vec3 data1[1024];
Vec3 data2[1024];
int runCount;

//internal script functions - other scripts that include this one can use them but they 
//will not be used by the engine.
f32 blend(f32 t, f32 x, f32 y)
{
    return x + (y-x)*t;
}

void testFunc(string printMe)
{
    xlDebugMessage(printMe);
}

int fib(int n)
{
    if (n <= 2)
    {
        return 1;
    }
    else
    {
        return fib(n-1) + fib(n-2);
    }
}

//"public" functions can be called by the engine. Future tools will be able to list
//all of these, for example you could be editing a level, load a level script and 
//then select these public functions from a list to run them based on various events.

//This function was used to help test a certain kind of performance. If you can't 
//figure out the point, don't worry its only meant to test floating point math 
//performance and function call overhead.
public void perfTest(void)
{
    f32 blendfactor = 0.7594f;
    for (s32 k=0; k<1000; k++)
    {
        f32 value = 2.0f + (float)(k-50)*0.01f;
        //step 1. fill the data with values.
        for (s32 i=0; i<1024; i++)
        {
            data0[i].x = value; value *= 1.25987f;
            data0[i].y = value; value *= 2.25987f;
            data0[i].z = value; value /= 2.25987f;

            data1[i].x = value; value *= 2.25987f;
            data1[i].y = value; value *= 7.25987f;
            data1[i].z = value; value /= 20.25987f;
        }

        //step 2. blend between the values.
        for (s32 i=0; i<1024; i++)
        {
            data2[i].x = blend(blendfactor, data0[i].x, data1[i].x);
            data2[i].y = blend(blendfactor, data0[i].y, data1[i].y);
            data2[i].z = blend(blendfactor, data0[i].z, data1[i].z);
        }
        blendfactor *= 1.001f;
    }
}

//Testing a public script function with a different number of arguments.
public void simpleInc(int a, int b, int c, int d)
{
    runCount++;
}

//xl...() functions are provided by the engine and are available to all scripts.
//sqr() and someVar are both defined in "test2.xlc" and are available since that
//script is included.
public void simple_main(int arg0, int arg1, int arg2)
{
    int r = fib(32);
    xlDebugMessage("fib(32) = %d.", r);
    runCount++;

    //MAX_MAPPING_COUNT is a engine provided define.
    xlDebugMessage("MAX_MAPPING_COUNT = %d", MAX_MAPPING_COUNT);
    xlDebugMessage("Clock = %d", xlGetClock());

    //Using a script defined structure.
    Test2 test;
    test.y = 3;

    //Another script defined structure.
    TestStruct test2;
    test2.x = sqr(test.y) + someVar - 2;
    xlDebugMessage("test2.x = %d.", test2.x);

    //testing string passing.
    testFunc("this is a string.");

    xlDebugMessage("Test inputs: %d, %d, %d", arg0, arg1, arg2);
}

As an addendum to the previous post, I uploaded a youtube video of Shadow Warrior played with the XL Engine. It is best viewed in 1080p60 on youtube.

Progress Updates

It has been about 2 weeks since my last update but I have not been idle during this time. I decided to scale back on the rate of updates for a variety of reasons but work is ongoing and I check the blog/forums regularly so feel free to comment or ask questions. In the future I plan on posting updates about once a week or so, though the time between updates may occasionally be longer or shorter.

Also remember that you can see the full sized images by clicking on the pictures, they are scaled down to fit the blog format.

 

Decompiling

I have made good strides in the decompilation process, due to a variety of enhancements and fixes to my tools. Previously functions with multiple return statements or complicated flow control with dead code were problematic, resulting in sometimes non-nonsensical code, missing code or incomplete functions. The tool now properly follows the flow control and prunes unused code. Of course this results in functions were code is ordered based on when a branch was hit, so the resulting code must be reodered at the end of the process. Jumps with dynamic offsets are still problematic though, there is still more work to be done.

Next I fixed the “root function” determination code so it can successfully find “main” automatically. Previously I had to do it by hand which can result in errors. The unused function removal is now more robust.

Initialized static memory is now automatically mapped to the memory addresses used by the disassembled code, meaning that pre-initialized static data is now directly available. This is great for many reasons, including using the text referenced by the code to help identify the names and functions in many cases. This allows me to move a lot of the code to the correct files, so the structure of the source tree – in some cases anyway – can mimic what the original source might have looked like.

 

Unified Software Renderer

Recently I resurrected the software renderer I was writing for DaggerXL and start re-integrating it into the XL Engine. For the Beta 2 release, it will serve as a basis for the “unified 3D renderer” that can be used not just for Daggerfall but potentially other XnEngine games and for model-based elements of 2.5D games (such as Dark Forces). The “unified 3D renderer” will support both hardware and software rendering and – with the original code available – be able to functionally match the original visuals but with better performance for high resolutions and the ability to more easily add new features and fix bugs. Of course the original Daggerfall rendering will be available, at least until the new renderer can match it exactly (minus obvious bugs). Beta 1 probably won’t ship with this renderer but it will be available for Beta 2.

Below you can see some screenshots of Daggerfall using the cylindrically mapped sky and tweaked settings. Note that all of the 2D rendering has been disabled (weapons, UI, etc. – obviously not counting sprites in the 3D world). Also note that the rendering is not 100% correct but that will be fixed – including the wrong ground tiles being used, not a problem for the original renderer of course.

Build, Blood and the Newcomer

When working with the Blood code, one of the things I wanted to do was match up the decompiled source with the Build source in order to reduce the amount of work I had to do. I realized, however, that for best results I should test the Build source integration with something working and complete. So to that end, I added the first – and currently only – game that uses the original source code: Shadow Warrior. The purpose of this integration was as follows:
* Get something working with Build in the XL Engine.

* Start refactoring the code and getting it ready for Blood.

* Test the XL Engine functionality, including game life cycle, XL Engine services, sound, input and other systems.

* Have a complete experience to test with the engine and UI.

* Test the performance and memory usage with a complete game running.

 

When, in previous updates, I mentioned matching up Blood code with the Build code – I already had this working in the engine. In fact Shadow Warrior is 100% playable, including sound, music, controls, memory management and so on in the engine and has been for a month or two.

So why play Shadow Warrior using the XL Engine instead of an existing port? Honestly there aren’t really any compelling reasons, the existing ports do a great job. Of course I plan on changing this with future releases and in the future there will be more reasons to play Shadow Warrior on the XL Engine. Regardless it has helped me tremendously with the engine and Blood – so its a worthwhile addition even if no one actually plays it. :D  That said, when it comes time to build the unified sector engine it will be very useful – like Blood it pushes the Build and adds Room over Room, drive able vehicles and other features.

This is a surprise I’m sure but, if things go well anyway, Shadow Warrior will not be the biggest surprise in store for the release.

As you can see, the UI itself has gone through some iterations since I last showed it. Of course any effects are optional and only work if you have a GPU capable of OpenGL 2.0 or newer (see the fullscreen view to get a better idea of what I mean). In addition sound effects have been added to the UI to improve the experience.

And some screenshots of Shadow Warrior in the XL Engine. As said above the port is complete and playable now. But I’m still going to hold off on releasing anything until the other games are ready.

 

 

Final Words

The XL Engine is progressing nicely and all of the features needed by the games are already implemented (as shown with Shadow Warrior). In the next update I hope to show screenshots of some of the other games in action. Finally here is the XL Engine Readme file that will packaged with the release (XLEngine.txt). In it is the copyright notice, description and credits. Take a look and let me know if I am forgetting anything, I want the release to go smoothly when it finally gets here. :)

The XL Engine is and will remain free, donations are purely optional but greatly appreciated.