I have recently taken a quick break for the Daggerfall specific work towards the Beta to change and improve the non-game specific UI system. I have integrated a Webkit based browser/UI system into the engine using the Awesomium library. Before I talk about specifics, I’ll show you a preview – running in DaggerXL:

(click to see the full size image)

I will be using this to build cross-platform elements for XL Engine specific UI using HTML5/CSS/Javascript controls. This has several benefits:

* Anyone with knowledge of web technologies can edit or build UI elements using this system – no proprietary new systems to learn.

* I can embed useful elements directly into the engine such as in-game access to the IRC channel, documentation from the Wiki, the ability to search the forums to resolve issues or post about a problem while in game, tutorial videos built right in and more.

* Robust UI, text, language, formatting, layout and animation – all built in.

I am also planning on rolling the launcher into the main executable, allowing you to start the engine and switch games without having to exit out first. The flexible, robust UI system that this allows will be vital later, when its time to start working on the editing tools again as well. Another thing I am planning on doing is allowing game specific code/script to be edited and updated right inside the engine (WYSIWYG) – using CodeMirror.

Here you can see an early version of the code editor (still incomplete):

(click on the image for a larger view)

A while back I put up this post: DaggerXL Terrain in the Beta Part 1. I searched the internet for heightmap images (or as close as I could find anyway) of all of Tamriel in order to build the heightmap for the rest of Tamriel. I found such an image:

I then took this image, converted it into a heightmap, rotated it to fit the Daggerfall data while scaling and filtering it and then added in the heightmap data from Daggerfall itself. However, as it turns out, I did not do a very good job digging into who put this map together, thinking it was from earlier worked released to the public. It turns out it was put together by someone named Morcroft Darkes - which I didn’t find out about until recently. Of course this is entirely my fault for not doing fact checking before using the map. To that end I will not be using this as a base for Tamriel, after all. I will attempt to find another decent source or merely leave it up to the community to help out later on. So until/if that happens, I will temporarily remove the extreme view distance feature, since basing it off another person’s work, who does not necessarily wish for it to be used, is clearly unethical behavior.

Lesson learned: always figure out where a source is coming from. Usually I do this, but not in this case obviously.

On a brighter note, DaggerXL is still continuing – I will put up a more upbeat post shortly.

I’m posting another update to show a new screen from my reverse engineering efforts – “DaggerfallDOS.” This picture is important, however – what this shows will be a feature in the DaggerXL Beta - DaggerXL will fully support Daggerfall saves as-is. I mentioned this as an “aside” in one of my previous forum posts but I wanted to explicitly announce the feature now that it actually works.

You’ll have the option of pointing DaggerXL to you Daggerfall saves (as I’ve done in this screenshot) or specifying a new area for the DaggerXL saves. If you do the latter, you can still copy the saves over manually if you want to use them.

(click on the screenshot to see the full size version)

What you see here are my Daggerfall saves, from my current play throughs. (One “real” and one for testing some things). Of course this uses disassembled Daggerfall code in DaggerfallDOS to render this but it will look and act the same in the DaggerXL Beta (at least in “pure” mode at 320×200). There will still be the option of showing the date and time for the saves as well, like the current build of DaggerXL.

Once the Beta is released you will be able to continue the game from your current Daggerfall save games and play them in DaggerXL.

I’ve said in previous posts that I planned on completing the gameplay for the Beta – making DaggerXL a full DosBox replacement for Daggerfall. I also said that with the most authentic settings – extended features disabled – that DaggerXL would behave almost identically to Daggerfall, with the exception of fixing game breaking bugs like crashes and allowing the game to run at a fluid framerate without the gameplay elements messing up (no fighting with cycles or dealing with low framerate so NPCs can walk). What I haven’t said is how I am accomplishing these goals.

In the past I’ve done some reverse engineering of certain contained elements (like reading some table data from the executable) and spent a fair amount of time puzzling through file formats, testing stats and their reactions and so forth. But I’ve been working on something else without telling a lot of people that will help me make DaggerXL as accurate an emulation of the original as I want it to be, literally. To put it simply I’ve been decoding the original executable and reconstructing it in my own C code – completely (not just bits and pieces). Then I take that C code and modify or write the systems properly in DaggerXL, fixing game breaking bugs and refactoring to fit the rest of the game/engine. What this allows me to do is make DaggerXL just as faithful and authentic as a pure source port would be even without having access to the original source code.

During this process, I’ve been putting together a temporary executable (“DaggerfallDOS”) which I can run and verify that I’ve interpreted the code correctly. This uses just the original Daggerfall code – decompiled and rewritten in C – with the exception of OS calls and Windows boilerplate. I don’t plan on releasing this but all the code that goes into this is being used to make DaggerXL as authentic as a source port.

If you want an example, look at this page: Decompile-Example. It shows just one function, much much more of this process has been completed. And this code (and the functions that it needs that aren’t shown) were used to render this:

This is being rendered directly from the decompiled Daggerfall code, in addition the mouse and keyboard input works, you can select the appropriate option from the keyboard shortcuts or mouse. I’m currently working through from beginning to end, making sure that each part works properly by using the new executable. This means that with the Beta, DaggerXL will be 100% vanilla Daggerfall gameplay complete – and will be nearly 100% authentic with the appropriate settings. Of course the extended features I’ve already shown will still be available – higher resolutions, color depth, texture filtering, higher quality lighting, bloom, large scale terrain rendering, improved terrain heights and visuals and much more. This process is going extremely well and should be complete in a few weeks, after which I can release the Beta – after a short period of closed testing to get rid of any remaining unknown issues.

So what does this mean for the other games? Its quite simple – every supported game will be as good as a source port when they reach Beta, DarkXL and BloodXL included.

As you may have noticed, I’ve switched focus a bit lately. I will continue with the terrain and post results in the relatively near future – much of the work is complete – and it will still be in the Beta as previously discussed. However I’ve spent most of my time lately figuring out gameplay elements, in order to get the final – proper – Daggerfall gameplay in pace for the Beta. This has involved additional testing of Daggerfall as well as debugging the executable running in DosBox and reverse engineering. There will be settings to determine how authentic the Daggerfall experience is in DaggerXL, or how enhanced – with the most authentic settings being very close to DOS Daggerfall – except for fixes to game breaking bugs (and things like modern compatibility, performance and so on that you’ll get from the XL Engine). On the other end of the spectrum you can have texture filtering, bloom, smoother motion and control, extremely long view distances and more. The key is that you can pick and choose how authentic or enhanced you wish your experience to be in different areas.

Earlier today I took a small break from all the gameplay stuff to put the various texture filtering options in the engine again. Usually you get the choice between disabling filtering entirely – which gives you the sharp but blocky and jagged look to the textures and bilinear/trilinear which smooths out the jagged edges but blurs out the details in the low resolution textures. For the XL-Engine, I’m adding a third option: “Sharp Bilinear.”  Sharp Bilinear preserves the blocky nature of the original textures but adds smoothing to the edges, giving the cleaner filtered look without blurring out the texture details.

Click on the thumbnails for full size screenshots, the effect is somewhat subtle:

From left to right: Bilinear Upscale + Trilinear, Sharp Bilinear Upscale + Trilinear

Sharp Bilinear Upscale + Anisotropic

As you can see, assuming you looked at the full size images, “Sharp Bilinear” preserves the “pixels” when upscaling the textures but still applies some filtering to avoid aliasing and shimmering as you move around. In my opinion at least, this gives you the best of both worlds and looks much better then pure bilinear for older games with smaller textures. In addition it does not require any additional texture lookups, just some shader code that alters the texture coordinates to adjust the effect of the hardware bilinear filter.

Motivation

In the past I talked about the terrain system as seen in the current release of DaggerXL. For the Beta, the terrain on older hardware will be done in a similar way, though with the enhancements as shown in the last video – improved coastlines, animated water, proper ground texture usage, etc.. In this post I will be talking about the system I hinted at in the Continuing and OpenGL Support post – “This renderer will feature view distances of greater then 100km as well as the highest quality rendering.”

So you might be wondering, why bother extending the rendering distance so far when emulating an old game like this? To answer that we’ll have to think about more modern Elder Scrolls games – Morrowind, Oblivion, Skyrim. The view distances in those games aren’t much bigger then the current version of DaggerXL, smaller in some cases. Yet the terrain has more variety and interesting features. Why is that? To put it simply, these games – like many games – are like themeparks, packing all the interesting locations together and removing or drastically reducing the space in-between. In just a few hundred meters you can see entire biomes, an arduous climb up the tallest mountain takes minutes rather then hours or days. This allows these games to pack a lot of interest and variety into a relatively small view distance. Unfortunately this can also result in obvious absurdity if the designers aren’t careful – like tripping over ancient hidden ruins constantly in Oblivion.

Daggerfall uses a realistic scale for terrain which has many benefits such as making the world feel much larger and more authentic. Unfortunately it trades the “themepark” feel for an environment that feels homogeneous and boring. One of the reasons is that the interesting features are spread over a much larger scale – so the proper thing to do is extend the view to realistic distances as well, this way one can appreciate the large scale vistas currently hidden in the fog. It is obvious why this couldn’t be done when Daggerfall came out on DOS and of course the amount of the view distance can be extended on older hardware is limited – though still a significant improvement. But with modern hardware there is no good reason that we can’t extend the view distance to realistic levels now.

Beyond the Map

But wait! What happens when you look over the edge of the Daggerfall map? With the previous DaggerXL view distances, stretching the height map and adding in procedural details was enough to be convincing – which I actually implemented, see the Terrain Beyond the Map Borders post. However when using realistic view distances this obviously doesn’t work and looks downright silly. I considered other options, such as shrouding the edges in thick fog – which is something that is oftentimes done to artistically hide the edges of a limited map

(example Zelda map from http://www.zeldacapital.com - note this has no relation to this project, just a nice picture to illustrate the idea, though I am a big Zelda fan too)

However this would still result in really odd visuals and give the impression that the Daggerfall area is shrouded and cutoff from the rest of Tamriel – which was not Bethesda’s original intent. So I decided on a new approach, with the help of suggestions from some others.

Tamriel

So what is beyond the edges of the Daggerfall map? The rest of Tamriel of course.  So what I’ve decided to do is to put together a heightmap of Tamriel – at the same scale as Daggerfall – and fit it to the Daggerfall data. Below is a screenshot of that map, though still unfinished -

The actual height map is over 5000 pixels in width, where each pixel is the same size as the pixels in Daggerfall’s 1000×500 heightmap. Daggerfall is inserted into the correct place, with the rest of the world rotated and scaled to fit. I consider the Daggerfall data to be ‘correct’ since this is for DaggerXL after all, so the Daggerfall data will still be pulled from the original data files, I inserted it into this map so I can fit the map to it rather then the other way around. There are several issues with this map still, I have to fix up coastlines and the Black Marsh area and fit the map to a 500×500 pixel grid, so that Daggerfall takes up exactly 2 grid cells. If you can see off the edge, it’ll wrap to the other side – same for swimming, flying or sailing across an edge. This way only 9 heightmap cells need to be in memory at once in order to render the world. Of course, for now, only the area from Daggerfall will have locations. However part of the intent of this is that you guys will have the rest of Tamriel already available when the modding tools come online. Don’t like parts of the heightmap – you can edit the file and distribute it. If you guys, as a community, come up with the definitive heightmap and it works with the existing Daggerfall data – then I’ll replace this map with that one in the future DaggerXL releases.

That said wait until you get the final heightmap before doing anything with it. It will have to conform to the grid, which this one does not yet. Also there is a seperate map, not shown here, that contains data such as the proper terrain textures to use.

In Part 2, which should come out next week, I’ll show the terrain system in action and talk about how it actually works under the hood.

In the last post, I talked a little about how OpenGL support is going to work and about extending the support to include Fixed Function and even software rendering (though that will come online a little later). In this post I will start a little about the architecture of the XL Engine and how supporting different games has been implemented.

The main XL_Engine executable contains the core engine, it contains all the platform specific code, OpenGL and other core systems (sound system, scripting engine, networking layer, etc.).

The engine also contains an “Archive” system, which transparently allows file handling code to lookup and read files from many Archive file formats – such as GOB, LAB, BSA, ART and so on. Then for files that perform specific functions – say image/texture files or level/location/chunk data – the system can transparently load the appropriate data and have it converted internally to the correct format based on the current settings and type. For example, when using the Software renderer 8 bit images are kept in that format but are converted to 32 bit as needed, based on the current palette and color map, for the OpenGL renderer. Currently the file format data is all internal but it is also possible to write Dynamic Library based plugins, similar to the way different games are supported (DLL/so).

Game code is implemented as plugins,  each game consists of a game library (DLL/so), script files and any game specific assets (in the case of games like DaggerXL – these assets are things like icons, title screen, etc.). In addition there is a small text file that describes the game so that it can be added to the list of games in the launcher. The engine has a dynamic library handler that supports Windows, Linux and OS X – there is very little code difference between systems and it is all handled by the Plugin system built into the engine. So for game and plugin code, the engine is used to handle all the OS and engine work – including things like input, file handling, script engine and so forth – allowing the game code to be platform agnostic. Obviously the same is true for all the scripts written as well.

Game plugins register various callbacks with the engine – so that they can receive input, start up game flow and UI scripts, handle pre and post render events, update and handle other events. The game will setup console functions (and potentially variables) and script functions, create entities and register game specific “logics.” Logics are self-contained functional elements, written in code or script, that make objects do things. As an example, a “Door” logic will cause a door to open or close when it receives and activation message. An entity is controlled by one or more of these logics, which are either assigned by the game code itself, through scripts or from the level data. Entities are handled by the core engine but Logics allow them to be controlled from game code, scripts and even shared logics built into the engine.

Anyway, this description just scratches the surface – there is plenty more to talk about in future posts, such as threading model, resource loading, virtual processes allowing for multiple plugins to run simultaneously (such as the editor and one or more games), networking model, script based game flow, script based UI and more. Stay tuned for future posts.

First I apologize for making a promise that I couldn’t keep. However I still believe Beta is within the original time frame I estimated but other things got in the way, as usual. Anyway, enough about that.

The goal is still the same, the next release will be the Beta release for DaggerXL. Everything is up and running again and I’ve started to get back to the project, which is coming along fairly well. The next release will still be the Beta – which means NPCs, quests, dialog, shopping, spells, guilds and so forth. Later I’ll talk about progress on the game and such but first I’ll talk a bit about support plans.

The Linux version will not be coming out with the Beta. While the engine supports Linux now, my new computer does not… Well its a long story but suffice it to say that I have issues to work out before development on the Linux version can continue. This is something I’m planning on doing but I’m not going to delay the next release any more then I already have because of it.

I’m shelving the software renderer for the next release and am going to focus on the OpenGL side of things for this release to save a little time. The software renderer will become available with the following release instead so it won’t be shelved for long. However, for those with older systems, I will still be supported fixed function hardware with a reduced feature set. Below I’ll list the major OpenGL versions that will be supported:

——————————————

* OpengGL 1.2 - simple renderer with single textures and basic OpenGL fog. In terms of lighting this will be the least faithful renderer compared to the “vanilla” game – but there is only so much that can be done without shaders or multi-texture support. Only the first terrain LOD will be available, limiting the view distance to the same range as supported by the software renderer (see the last video to get an idea how far that is). You’ll be able to decrease the view distance to get performance on really old systems. This mode should support GPUs all the way back to the GeForce 256, released in 1999. Intel Integrated graphics should be fully supported from 1999 on (as well as other integrated chipsets that support OpenGL).

* OpenGL 1.5 - this is basically the same as the above but adding support for multi-texture and Vertex Buffer Objects. This will allow for faster rendering and slightly better lighting but will otherwise be the same as above. GPUs made in 2003 or newer should be able to use this mode.

* OpenGL 2.1 - this adds shader support, which will allow the lighting and fogging to look close to the original. Effects like Bloom will be available as well. Cards from 2005 and newer should support this mode. [for NVidia owners: GeForce 6000 series or newer]

* OpenGL 3.1 - this will be the “fully featured” renderer. This renderer will feature view distances of greater then 100km as well as the highest quality rendering. GPUs made in 2007/2008 and later (depending on brand) should support this mode. So if you have a discrete GPU and bought your computer in the last couple of years, you should be able to enjoy the game with maximum quality. [for NVidia owners: GeForce 8000 series or newer]

——————————————

If you have any concerns or questions about support for your GPU or computer, please let me know. Unfortunately I cannot guarantee performance on really old systems but I can at least make sure the XL Engine runs and have settings you can turn down to help eek out the performance you need. In previous builds many people had performance issues in areas with lots of lights or foliage – that should no longer be the case with the Beta.

I’ve been away for a while, I went on a long vacation (first one in years by the way) and have been busy with the holidays. I haven’t forgotten about this project and will give a small update about what’s going on. This is going to be a short post but important.

The wait is not been for nothing though, I plan on releasing the DaggerXL Beta within the first month of the new year and then will proceed to get the DarkXL Beta out soon afterwards. Just to be clear, by Beta I mean the games playable from beginning to end and fully completable. Of course there may still be bugs and they won’t have all the features I want to call them “done” (as close to done as projects like these every get anyway) – thus the Beta – but they should be good enough to replace playing them in DosBox.

Unfortunately I probably won’t be around the forums much until after Christmas but then I’ll return to posting regularly again.

Hello, sorry about the long absence. I’ve only been able to poke my head above water, as the saying goes, recently. I’m beginning work again on the DaggerXL release in earnest.

My first task is to finish the dungeon elements and make sure everything works properly from the point of starting a new game, creating a character, loading, saving, etc.. This will include having proper textures on dungeons in different areas, making sure the character screen and inventory are completely functional and proper water levels.

Once that is done, I will finish the exterior/town work which will include walking NPCs, NPC conversations, making sure interiors are properly functioning, in-town automap and finishing up the ship work. Also finishing the fast travel map UI so it has all the features of the original (i.e. zooming in, find location, etc.). Then I’ll put out the release. If there is enough time, I might also include buying and selling from shops – though the quests and guilds will have to wait for the following release.

Once the dungeon work is complete I’ll put out another video showing different dungeons with everything working. I’ll also put out a video once the exteriors/towns/ship is done as well.