This has been posted on the forums a few weeks ago but I neglected to post it on the blog until now.
There have been concerns regarding XLC and supporting different platforms. To address those concerns and describe some changes that I am implement (or will implement in the future), I decided to make a new post. If you have not already, please read the Introducing XLC – XL Engine Scripting System post. Note that most of the features described here will become available after Beta 1.
By default, XLC code will be JIT compiled and executed as native code without using a VM. However byte code compilation and execution using a virtual machine will be available for platforms that the JIT compiler does not support. While executing scripts using a VM will be slower, it will still be as fast as I can make it. This will allow XLC to run on any platform that the XL Engine supports – though I will obviously like to add JIT support to those platforms to maximize script speed. Initially the JIT compiler will support common CPU/instruction sets, including x86, x64 and ARM. (Note, this does NOT mean that the XL Engine will support ARM based platforms for Beta 1 or even give any timelines for that – it merely means the JIT Compiler will support ARM)
Optional Bytecode Optimization
Whenever iterating on scripts, reloading and JIT compiling XLC scripts is the most efficient way to work. In many cases, just writing the code in a text editor and using it directly should be fast enough. However, in cases where the JIT compiled code isn’t fast enough (or you need to use the VM), it will be possible to pre-compile to a binary format – platform independent bytecode – and then spend much more time during the optimization process. This should allow the script execution speed to get even closer to full native performance. A tool will be available, tentatively called “xlcc” which will generate binary files for each script file “.xlo”. Since the bytecode is platform independent, it will still be JIT compiled on load – and no additional tools are required (except for xlcc of course).
In addition to the existing language extensions – such as every script being executed as a coroutine and cooperative threading incoporated as a core feature – I plan on adding a few more language extensions. Primary, I plan on adding HLSL/GLSL like vector and matrix math capabilities – such as swizzles and basic mathematical operations (code such as: float3 a, b, c; a.xy = b.zz * c.xz;). This should support a similar set of types, including float2/3/4, int2/3/4, uint2/3/4, bool2/3/4, float2x2, float2x3, float2x4, and so on. Why? Multiple reasons – the first is to make standard 2D and 3D math operations simpler, which should be useful for the targeted games.
The second reason for the language extensions is to support “kernels” – optimized JIT compiled code designed to operate on multiple values simultaneously (either 4, 8 or 16) that can be used by the software renderer (i.e. software renderer “shaders”). Kernels are still JIT compiled, but instead of generating scalar code – the compiler will generate data parallel vectorized code that will make the best use of the CPU specific vector instruction set such as SSE2, AVX, AVX2, AVX-512 and NEON. I also plan on using XLC as an API independent shader language, to allow even GPU shaders to be written once and available for OpenGL, OpenGL ES and later Vulcan and other APIs. This will replace the current GLSL shaders and make supporting different platforms very easy for the shader writer. Kernels will be accessible to normal scripts in the cases where the scripter needs to process larger data sets, where the per-element operations are homogenous and simple. Note that kernels will also process data in batches of a multiple of 16 elements.
In conclusion, I have big plans for XLC – to both accelerate my own work flow (runtime updating when editing UI code for example), to allow modders with a platform independent scripting language with near native performance and to even allow CPU optimized vectorized kernels, which can take advantage of CPU specific features while allowing code to be written in XLC, to provide better then native performance in those cases.
Of course I need to stay focused on Beta 1 so some of these features will not come until later, though probably not as far off as it sounds.