Translate
EnglishFrenchGermanItalianPortugueseRussianSpanish

The XL Engine is and will remain free, donations are purely optional but greatly appreciated.

Recent Comments

Uncategorized

I originally promised an update this week. Unfortunately there isn’t a lot to tell, I’m still hammering away at the goal of completing the gameplay functionality for the Beta. I should make some good progress next week with the closed testing starting shortly afterwards – I already have enough testers lined up for this (see the previous blog post). My plan is to have a gameplay video up next week, showing the “Pure Mode” in action – showing things like proper NPC behavior, quests and so forth. :)

This is a short post this time but I’m planning on having a blog post per week from here on out (or more when warranted), though some may be short and informal like this one.

**UPDATE**

I have enough people to test now, I will post another update next week. I will also make an announcement once the testing has actually started.

 

So I’ve been back to work on the actual gameplay side of things. I’m currently working on getting character generation to 100% completeness (class questions, all the UI working properly, custom class generation). After that I have to complete the Quest system.

Anyway, due to reverse engineering efforts, the “Pure Mode” will be really close to the original (source port close) – with the addition of fixing game breaking bugs. To that end I plan on having a small closed testing period, with just the Pure Mode enabled, in the next couple of weeks. The goal will be to a)make sure it works on a sampling of computers, b) make sure that there aren’t any critical bugs that make it into the broader release (game doesn’t start, black screen, crashes, etc.) and c) make sure the game is fully playable (even beatable if you’re fast enough to get through before the testing period ends).

While this testing is being done, I’ll finish up other elements needed for the Beta release (settings and options, remaining high-end terrain work, etc.). The full Beta release will be available to everyone, of course.

So I’ll probably enlist about 8 people from here for this initial phase, which will start in a couple of weeks. Just post in the forum thread with this same name. Make sure you are willing to test, accept that this may be buggy and understand that only the Pure Mode will work (i.e. no fancy graphics, no advanced terrain, etc.). If you are interested, just post a response in the forum thread with system specs (you can PM me the specs if you aren’t comfortable posting them publicly). Also this is only DaggerXL, though there will be similar tests for the other big releases as well.

 

Obviously progress has hit some snags in the last few weeks but things are continuing, if a little slow at the moment. However the Beta is still getting steadily closer. :)  The gameplay in the Beta is the highest priority but I’m going to talk about something else today.

 

You’ve all read about the high-end terrain that will allow for greatly increased view distances, especially after the heightmap controversy. However, other then some heightmaps, I’ve never shown the feature in action. I’ve decided to rectify that today and show the current state of that system, which I plan on finishing for the Beta release – once the gameplay is complete. (For the record, the only heightmap used in this video currently is from Daggerfall itself). For terrain rendering, you’ll have several options with DaggerXL:

* (Low) Daggerfall terrain – exactly as it is in the original Dos game. This terrain uses finer heightmap / tiles and ignores the large scale heightmap (for the most part). The good part is that it matches the original game. The bad part is that the terrain quality is low – the fine scale height data, due to how it was made and used, is not very good. That is why only the large scale height map (1000×500) is used in the following modes.

* (Medium) Improved terrain with increased view distance, which will work both in hardware and software modes. This is what I’ve shown in the past, including the last video. This should work for most people who want to use DaggerXL and currently is mostly complete, it supports foliage, locations, terrain types and so forth as shown in the last video.

* (High/Very High) High quality terrain – in most cases renders to the horizon and beyond, a view distance of hundreds of kilometers. This is still not finished and won’t be until the gameplay is complete. However I did work on it a little recently so I’ve decided now is a good time to show it for the first time – in the form of a video. However, like everything, there are some downsides –

– the system requirements for using this quality level are higher, though Medium quality terrain still gives a large increase in quality and view distance over vanilla Daggerfall.

– the original textures are used, the original texel density is honored (mostly) but the mesh density is higher. This is a good thing for visuals but due  to the higher detail (mesh and normals), it doesn’t quite match the original look as well as Medium quality terrain. I think the locations and sprites still work pretty well with it but you’ll have to decide for yourself.

 

In the video below you’ll see several glitches due to the fact that the system is still not complete and that I did something recently to make terrain cracks show up again – which will be fixed for the build of course. However this means you’ll have to put up with some issues when watching the video. In addition the terrain only has one forest section, in the northern Daggerfall province, and the rest of the terrain uses the same procedurals and textures (with no foliage). These issues will be fixed when the system is finished, allowing for much more variety and matching what the terrain is supposed to be like closer. Also no locations are loaded – mostly because the LOD system is incomplete – but they still work just fine for Low and Medium quality terrain, as seen in the last video, and will work for the build. Finally, this terrain uses only the large scale heightmap (the 1000×500 one) bicubically upscaled with procedural layers to provide additional detail. I still plan on upscaling that heightmap and applying some proper erosion in the future to make the terrain more realistic, but that is not done for this video. So the end result should be higher quality.

The video starts with the player inside a thick forest, testing how thick a forest can be – so its really dense. You’ll see trees on distant hills/mountains as well. Later smaller plants/rocks will be added back in as well as using the other tree species as appropriate. But the main point was to test proper forests for DaggerXL. Once I leave the forest, I travel towards the coast, with Sentinel visible in the background. I then fly across the Bay to Sentinel, fly around the coast and eventually land at the tallest location in the map. Finally I fly up so you can see more of the map – with the Daggerfall province still visible once the camera is high enough in the sky.

 

(I would suggest watching this in HD)

I have recently taken a quick break for the Daggerfall specific work towards the Beta to change and improve the non-game specific UI system. I have integrated a Webkit based browser/UI system into the engine using the Awesomium library. Before I talk about specifics, I’ll show you a preview – running in DaggerXL:

(click to see the full size image)

NewUI

 

 

 

 

 

 

 

 

 

 

I will be using this to build cross-platform elements for XL Engine specific UI using HTML5/CSS/Javascript controls. This has several benefits:

* Anyone with knowledge of web technologies can edit or build UI elements using this system – no proprietary new systems to learn.

* I can embed useful elements directly into the engine such as in-game access to the IRC channel, documentation from the Wiki, the ability to search the forums to resolve issues or post about a problem while in game, tutorial videos built right in and more.

* Robust UI, text, language, formatting, layout and animation – all built in.

 

I am also planning on rolling the launcher into the main executable, allowing you to start the engine and switch games without having to exit out first. The flexible, robust UI system that this allows will be vital later, when its time to start working on the editing tools again as well. Another thing I am planning on doing is allowing game specific code/script to be edited and updated right inside the engine (WYSIWYG) – using CodeMirror.

Here you can see an early version of the code editor (still incomplete):

(click on the image for a larger view)

CodeEditor_1

A while back I put up this post: DaggerXL Terrain in the Beta Part 1. I searched the internet for heightmap images (or as close as I could find anyway) of all of Tamriel in order to build the heightmap for the rest of Tamriel. I found such an image:

tamrielContourMap

 

 

 

 

 

 

 

 

 

 

 

 

I then took this image, converted it into a heightmap, rotated it to fit the Daggerfall data while scaling and filtering it and then added in the heightmap data from Daggerfall itself. However, as it turns out, I did not do a very good job digging into who put this map together, thinking it was from earlier worked released to the public. It turns out it was put together by someone named Morcroft Darkes – which I didn’t find out about until recently. Of course this is entirely my fault for not doing fact checking before using the map. To that end I will not be using this as a base for Tamriel, after all. I will attempt to find another decent source or merely leave it up to the community to help out later on. So until/if that happens, I will temporarily remove the extreme view distance feature, since basing it off another person’s work, who does not necessarily wish for it to be used, is clearly unethical behavior.

Lesson learned: always figure out where a source is coming from. Usually I do this, but not in this case obviously. :(

 

On a brighter note, DaggerXL is still continuing – I will put up a more upbeat post shortly.

I’m posting another update to show a new screen from my reverse engineering efforts – “DaggerfallDOS.” This picture is important, however – what this shows will be a feature in the DaggerXL Beta – DaggerXL will fully support Daggerfall saves as-is. I mentioned this as an “aside” in one of my previous forum posts but I wanted to explicitly announce the feature now that it actually works.

You’ll have the option of pointing DaggerXL to you Daggerfall saves (as I’ve done in this screenshot) or specifying a new area for the DaggerXL saves. If you do the latter, you can still copy the saves over manually if you want to use them.

(click on the screenshot to see the full size version)

DaggerfallDOS_LoadFix

 

 

 

 

 

 

 

 

 

 

 

 

What you see here are my Daggerfall saves, from my current play throughs. (One “real” and one for testing some things). Of course this uses disassembled Daggerfall code in DaggerfallDOS to render this but it will look and act the same in the DaggerXL Beta (at least in “pure” mode at 320×200). There will still be the option of showing the date and time for the saves as well, like the current build of DaggerXL.

 

Once the Beta is released you will be able to continue the game from your current Daggerfall save games and play them in DaggerXL.

I’ve said in previous posts that I planned on completing the gameplay for the Beta – making DaggerXL a full DosBox replacement for Daggerfall. I also said that with the most authentic settings – extended features disabled – that DaggerXL would behave almost identically to Daggerfall, with the exception of fixing game breaking bugs like crashes and allowing the game to run at a fluid framerate without the gameplay elements messing up (no fighting with cycles or dealing with low framerate so NPCs can walk). What I haven’t said is how I am accomplishing these goals.

In the past I’ve done some reverse engineering of certain contained elements (like reading some table data from the executable) and spent a fair amount of time puzzling through file formats, testing stats and their reactions and so forth. But I’ve been working on something else without telling a lot of people that will help me make DaggerXL as accurate an emulation of the original as I want it to be, literally. To put it simply I’ve been decoding the original executable and reconstructing it in my own C code – completely (not just bits and pieces). Then I take that C code and modify or write the systems properly in DaggerXL, fixing game breaking bugs and refactoring to fit the rest of the game/engine. What this allows me to do is make DaggerXL just as faithful and authentic as a pure source port would be even without having access to the original source code.

During this process, I’ve been putting together a temporary executable (“DaggerfallDOS”) which I can run and verify that I’ve interpreted the code correctly. This uses just the original Daggerfall code – decompiled and rewritten in C – with the exception of OS calls and Windows boilerplate. I don’t plan on releasing this but all the code that goes into this is being used to make DaggerXL as authentic as a source port.

If you want an example, look at this page: Decompile-Example. It shows just one function, much much more of this process has been completed. And this code (and the functions that it needs that aren’t shown) were used to render this:

DaggerfallDOS

 

 

 

 

 

 

 

 

 

 

 

 

 

This is being rendered directly from the decompiled Daggerfall code, in addition the mouse and keyboard input works, you can select the appropriate option from the keyboard shortcuts or mouse. I’m currently working through from beginning to end, making sure that each part works properly by using the new executable. This means that with the Beta, DaggerXL will be 100% vanilla Daggerfall gameplay complete – and will be nearly 100% authentic with the appropriate settings. Of course the extended features I’ve already shown will still be available – higher resolutions, color depth, texture filtering, higher quality lighting, bloom, large scale terrain rendering, improved terrain heights and visuals and much more. :D This process is going extremely well and should be complete in a few weeks, after which I can release the Beta – after a short period of closed testing to get rid of any remaining unknown issues.

So what does this mean for the other games? Its quite simple – every supported game will be as good as a source port when they reach Beta, DarkXL and BloodXL included.

As you may have noticed, I’ve switched focus a bit lately. I will continue with the terrain and post results in the relatively near future – much of the work is complete – and it will still be in the Beta as previously discussed. However I’ve spent most of my time lately figuring out gameplay elements, in order to get the final – proper – Daggerfall gameplay in pace for the Beta. This has involved additional testing of Daggerfall as well as debugging the executable running in DosBox and reverse engineering. There will be settings to determine how authentic the Daggerfall experience is in DaggerXL, or how enhanced – with the most authentic settings being very close to DOS Daggerfall – except for fixes to game breaking bugs (and things like modern compatibility, performance and so on that you’ll get from the XL Engine). On the other end of the spectrum you can have texture filtering, bloom, smoother motion and control, extremely long view distances and more. The key is that you can pick and choose how authentic or enhanced you wish your experience to be in different areas. :)

Earlier today I took a small break from all the gameplay stuff to put the various texture filtering options in the engine again. Usually you get the choice between disabling filtering entirely – which gives you the sharp but blocky and jagged look to the textures and bilinear/trilinear which smooths out the jagged edges but blurs out the details in the low resolution textures. For the XL-Engine, I’m adding a third option: “Sharp Bilinear.”  Sharp Bilinear preserves the blocky nature of the original textures but adds smoothing to the edges, giving the cleaner filtered look without blurring out the texture details.

Click on the thumbnails for full size screenshots, the effect is somewhat subtle:

From left to right: Bilinear Upscale + Trilinear, Sharp Bilinear Upscale + Trilinear

FilterModes_Trilinear

FilterModes_SharpBilinear

 

 

 

 

 

 

 

 

 

 

 

 

Sharp Bilinear Upscale + Anisotropic

FilterModes_SharpBilinear_Aniso

FilterModes_SharpBilinear2

 

 

 

 

 

 

 

 

 

 

 

 

As you can see, assuming you looked at the full size images, “Sharp Bilinear” preserves the “pixels” when upscaling the textures but still applies some filtering to avoid aliasing and shimmering as you move around. In my opinion at least, this gives you the best of both worlds and looks much better then pure bilinear for older games with smaller textures. In addition it does not require any additional texture lookups, just some shader code that alters the texture coordinates to adjust the effect of the hardware bilinear filter.

Motivation

In the past I talked about the terrain system as seen in the current release of DaggerXL. For the Beta, the terrain on older hardware will be done in a similar way, though with the enhancements as shown in the last video – improved coastlines, animated water, proper ground texture usage, etc.. In this post I will be talking about the system I hinted at in the Continuing and OpenGL Support post – “This renderer will feature view distances of greater then 100km as well as the highest quality rendering.”

So you might be wondering, why bother extending the rendering distance so far when emulating an old game like this? To answer that we’ll have to think about more modern Elder Scrolls games – Morrowind, Oblivion, Skyrim. The view distances in those games aren’t much bigger then the current version of DaggerXL, smaller in some cases. Yet the terrain has more variety and interesting features. Why is that? To put it simply, these games – like many games – are like themeparks, packing all the interesting locations together and removing or drastically reducing the space in-between. In just a few hundred meters you can see entire biomes, an arduous climb up the tallest mountain takes minutes rather then hours or days. This allows these games to pack a lot of interest and variety into a relatively small view distance. Unfortunately this can also result in obvious absurdity if the designers aren’t careful – like tripping over ancient hidden ruins constantly in Oblivion.

Daggerfall uses a realistic scale for terrain which has many benefits such as making the world feel much larger and more authentic. Unfortunately it trades the “themepark” feel for an environment that feels homogeneous and boring. One of the reasons is that the interesting features are spread over a much larger scale – so the proper thing to do is extend the view to realistic distances as well, this way one can appreciate the large scale vistas currently hidden in the fog. It is obvious why this couldn’t be done when Daggerfall came out on DOS and of course the amount of the view distance can be extended on older hardware is limited – though still a significant improvement. But with modern hardware there is no good reason that we can’t extend the view distance to realistic levels now.

 

Beyond the Map

But wait! What happens when you look over the edge of the Daggerfall map? With the previous DaggerXL view distances, stretching the height map and adding in procedural details was enough to be convincing – which I actually implemented, see the Terrain Beyond the Map Borders post. However when using realistic view distances this obviously doesn’t work and looks downright silly. I considered other options, such as shrouding the edges in thick fog – which is something that is oftentimes done to artistically hide the edges of a limited map

(example Zelda map from http://www.zeldacapital.com – note this has no relation to this project, just a nice picture to illustrate the idea, though I am a big Zelda fan too)

 

However this would still result in really odd visuals and give the impression that the Daggerfall area is shrouded and cutoff from the rest of Tamriel – which was not Bethesda’s original intent. So I decided on a new approach, with the help of suggestions from some others.

 

Tamriel

So what is beyond the edges of the Daggerfall map? The rest of Tamriel of course. :D  So what I’ve decided to do is to put together a heightmap of Tamriel – at the same scale as Daggerfall – and fit it to the Daggerfall data. Below is a screenshot of that map, though still unfinished –

[img removed]

The actual height map is over 5000 pixels in width, where each pixel is the same size as the pixels in Daggerfall’s 1000×500 heightmap. Daggerfall is inserted into the correct place, with the rest of the world rotated and scaled to fit. I consider the Daggerfall data to be ‘correct’ since this is for DaggerXL after all, so the Daggerfall data will still be pulled from the original data files, I inserted it into this map so I can fit the map to it rather then the other way around. There are several issues with this map still, I have to fix up coastlines and the Black Marsh area and fit the map to a 500×500 pixel grid, so that Daggerfall takes up exactly 2 grid cells. If you can see off the edge, it’ll wrap to the other side – same for swimming, flying or sailing across an edge. This way only 9 heightmap cells need to be in memory at once in order to render the world. Of course, for now, only the area from Daggerfall will have locations. However part of the intent of this is that you guys will have the rest of Tamriel already available when the modding tools come online. Don’t like parts of the heightmap – you can edit the file and distribute it. If you guys, as a community, come up with the definitive heightmap and it works with the existing Daggerfall data – then I’ll replace this map with that one in the future DaggerXL releases.

That said wait until you get the final heightmap before doing anything with it. It will have to conform to the grid, which this one does not yet. Also there is a seperate map, not shown here, that contains data such as the proper terrain textures to use.

 

In Part 2, which should come out next week, I’ll show the terrain system in action and talk about how it actually works under the hood.

In the last post, I talked a little about how OpenGL support is going to work and about extending the support to include Fixed Function and even software rendering (though that will come online a little later). In this post I will start a little about the architecture of the XL Engine and how supporting different games has been implemented.

The main XL_Engine executable contains the core engine, it contains all the platform specific code, OpenGL and other core systems (sound system, scripting engine, networking layer, etc.).

The engine also contains an “Archive” system, which transparently allows file handling code to lookup and read files from many Archive file formats – such as GOB, LAB, BSA, ART and so on. Then for files that perform specific functions – say image/texture files or level/location/chunk data – the system can transparently load the appropriate data and have it converted internally to the correct format based on the current settings and type. For example, when using the Software renderer 8 bit images are kept in that format but are converted to 32 bit as needed, based on the current palette and color map, for the OpenGL renderer. Currently the file format data is all internal but it is also possible to write Dynamic Library based plugins, similar to the way different games are supported (DLL/so).

Game code is implemented as plugins,  each game consists of a game library (DLL/so), script files and any game specific assets (in the case of games like DaggerXL – these assets are things like icons, title screen, etc.). In addition there is a small text file that describes the game so that it can be added to the list of games in the launcher. The engine has a dynamic library handler that supports Windows, Linux and OS X – there is very little code difference between systems and it is all handled by the Plugin system built into the engine. So for game and plugin code, the engine is used to handle all the OS and engine work – including things like input, file handling, script engine and so forth – allowing the game code to be platform agnostic. Obviously the same is true for all the scripts written as well.

Game plugins register various callbacks with the engine – so that they can receive input, start up game flow and UI scripts, handle pre and post render events, update and handle other events. The game will setup console functions (and potentially variables) and script functions, create entities and register game specific “logics.” Logics are self-contained functional elements, written in code or script, that make objects do things. As an example, a “Door” logic will cause a door to open or close when it receives and activation message. An entity is controlled by one or more of these logics, which are either assigned by the game code itself, through scripts or from the level data. Entities are handled by the core engine but Logics allow them to be controlled from game code, scripts and even shared logics built into the engine.

Anyway, this description just scratches the surface – there is plenty more to talk about in future posts, such as threading model, resource loading, virtual processes allowing for multiple plugins to run simultaneously (such as the editor and one or more games), networking model, script based game flow, script based UI and more. Stay tuned for future posts. :)

The XL Engine is and will remain free, donations are purely optional but greatly appreciated.