Guide: Siege University - 304: Deep Modding: .dsdll extensions

Introduction

This article is about how to extend the Dungeon Siege engine with new functions that you create. Even though there are thousands of functions available to be called from skrit, sometimes you just have to write your own function to get what you want done. For example, say you are creating a mod that needs to talk to a database to register player stats, or you want to do something simple like query the system time in order to make your mod "real time". To do either of these things you will need to create a set of functions in an engine extension that can be called from within your mod's skrit code.

In a nutshell, to extend Dungeon Siege with new engine functions for skrit to use:

1. Create a dynamically linked library (DLL) and set its output extension to be .dsdll rather than .dll.
2. Export the functions using the compiler's "declspec( dllexport )" tag that you want the game to see.
3. Put the resulting .dsdll file in the same directory as DungeonSiege.exe.It's pretty simple! Most of this doc is about the rules governing what you can and cannot export, and some of the quirks in the system.
   

What You Need For This Tutorial

• Dungeon Siege v1.1 or above The .dsdll features were added after Dungeon Siege originally shipped, so make sure to get the latest version through your ZoneMatch AutoUpdate feature.
  
• A Microsoft C++ compiler This is not absolutely required, as Delphi and other languages are perfectly capable of creating .dsdll's, however it is easiest with a Microsoft compiler so that the name mangling format is guaranteed to be compatible.
  Visual C++ 6 and Visual C++.NET are guaranteed to work for creating .dsdll's, and it's been reported that Visual C++ 5 works fine as well though I have not verified this myself.
  
• The WebMod sample This is a little mod I put together that contains a sample .dsdll, mod, and map. It demonstrates how to download a web page, process it a bit, and put the results up onscreen. Not terribly exciting, but the most important part of the sample is the dsdll.h file, which is a set of macros, docs, and some helper templates. Be sure to check this file out!
  The mod can be downloaded by clicking here.
  
• The DbgHelp.dll redistributable This is a DLL that is required by the game when using .dsdll's to decode the mangled names that the C++ compiler exports. It's only required in Win98 and WinME, as Win2K and WinXP both have the file already. The latest version of DbgHelp.dll can be found in Microsoft's "Debugging Tools for Windows" (freely available on their web page) or in the System32 directory of Win2k/XP, plus I've also included it in the WebMod sample for convenience. To make sure that your .dsdll-based mod runs on all versions of Windows, DbgHelp.dll should be installed to the same directory as DungeonSiege.exe.
  
• FuBi documentation on the web This is not required reading, but if you want to know how the system works underneath, I have written some detailed documentation on FuBi, our function binder that drives much of the low level systems of Dungeon Siege. You can find information including much of the source code to the system here:

  http://www.drizzle.com/~scottb/gdc/ (see GDC 2001 material)

  As this was written nearly two years ago, some of it may be out of date (FuBi has progressed considerably since then) but it's probably about 90% accurate. For the original paper on FuBi, you can see my "gem" from the first Game Programming Gems book:

  http://www.drizzle.com/~scottb/publish/gpgems1_fubi.htm

  This paper is even older, but it provides a good walkthrough of the problem that FuBi was intended to solve: the lack of reflection capability in C++.

Building and Distributing a .dsdll

Here is how to build a basic .dsdll with a single function:

1. Start a new DLL project in Visual C++.

2. Modify the linker settings to change the name of the output file to have an extension of .dsdll.

3. Create a .cpp file and add it to the project - it must be compiled as C++, not C, for this to work. Here is a basic sample:

// dlltest.cpp
#include "windows.h"
#include "dsdll.h" 
BOOL APIENTRY DllMain( HANDLE, DWORD, LPVOID )
{
  return TRUE;
}
class DllTest
{
  FEX static const char* GetTestString( void )
  {
    return ( "Hi there!!!\n" );
  }
};  

The FEX keyword is actually a macro that comes from the dsdll.h file. More detailed information on the macros available in this file is included in a later section of this article.

4. Compile and link the .dsdll and then copy it to the same directory as DungeonSiege.exe. You may find it convenient to just modify the project settings to output to that directory directly so you don't need to copy the file manually (or you can set a post-link build step to do so). Another option is to run Dungeon Siege with a shortcut that sets the dsdll_path command-line option (or in the INI file) to point to the location of your .dsdll. For example:

dsdll_path=c:\mystuff\dsdll\debug

You can verify that your functions have been exported by running the "Depends.exe" utility that is distributed with the Microsoft Platform SDK and looking at the export table.

5. The next time you launch the game your new functions will be available. Here's an example of how to use the function created in #3 above:

Report.ScreenF( "Your dsdll says: %s", DllTest.TestString );

This ought to print the text "Your dsdll says: Hi there!!!" to the top of the screen. You can insert this code in any skrit in the game and it will work. You can do other things like assign it to a string:

string s$ = DllTest.TestString;

In short, it's just like any other function in the game, the engine doesn't know the difference!

The .dsdll is simply a plain DLL renamed to .dsdll and loaded with exported functions. As such, it can do whatever it likes in the system, calling out to the Internet, playing sounds, reading the keyboard, using files on the hard drive, etc. - there are no limitations placed on what DLL's can do in Windows.

Distributing your .dsdll-based mod is about the same as distributing any other mod, with two exceptions. First, you must install a special file DbgHelp.dll into the same directory as DungeonSiege.exe ; second, you must create a shortcut or batch file that launches your mod. See the FAQ entry below called "I can't play multiplayer games with my friends any more! What's wrong?" for more information on this.

Usage Notes

During system initialization, the game looks for all .dsdll files found in the dsdll_path directory (which defaults to the game's EXE dir), then sorts them alphabetically and iterates through them for import. Importing is done by iterating through the export table on each .dsdll and de-mangling each entry found there (this is what the DbgHelp.dll is needed for), then mapping the function and its entry point into the system.
There are a lot of rules and oddities about how the system works, so let's just go through those as a set of randomly ordered notes. Note that most of the following is pulled straight from the documentation at the top of dsdll.h.

These are specifically supported in exports:

• Functions with external linkage - this includes nonstatic global functions and static or nonstatic class member functions. They may be overloaded by name, but be careful, as skrit's overload resolution algorithm isn't the best.
  
• Any number of parameters per function, including variable argument (...) lists. Exported functions can also return parameters.
  
• All built-in types (char, int, float, etc. but not doubles!) are supported for parameters or return values.
  
• Strings (const char*) are specifically supported through some special case code.
  
• Pointers and references to user-defined types (classes, structs, etc.) are ok as parameters and return values.
  The system automatically "understands" any type you throw at it. If you have two functions that mess with pointers or references to a class called Jooky then it will just work. To the engine it's always a 4-byte pointer, and all it does is make sure that the types match before permitting the calls to compile. This feature is used in many parts of the game, where pointers are queried from one system and then passed to others, without skrit messing with the pointer at all.
  
• Functions that match this signature:

T GetSomeVariable( void ) const
void SetSomeVariable( const T& | T )

Will get mapped in skrit as a variable. While it's still possible to call those functions directly using the full function names, you can treat them as the variable "SomeVariable". If the Get version is left out, then it's write-only, and if the Set version is left out, it's read-only.

There are also some important restrictions:

• __fastcall is not supported as a calling convention.
  
• Don't prefix any names with FUBI_ or allow any names to contain '$' in the middle of the name - these are reserved as special tags that FuBi uses for documentation and internal functions attached to exports. If you look closely at the macros in dsdll.h you'll see how this works.
  
• Skrit is case-insensitive and FuBi has to accommodate this. Do not export two functions with the same name but different case (for example, Test() and TEST()).
  
• For consistency reasons, do not use underscores in any exported names - some_function_call() is bad, SomeFunctionCall() is good.
  
• If a SetXXX and GetXXX pair exists but the parameters do not match as described above, it will print an error.
  
• "Complicated" exports won't even parse properly and you'll get an error from FuBi about how it isn't able to understand some function you're trying to export. This will typically happen with templates or functions that take function pointers as parameters. Not supported.
  
• Unions are not supported. Never will be.
  
• Passing complex types by value is not supported. If you export a function that takes a GUID by value, you'll get an error. Pass it by reference and it'll be okay. Specific support can be added to allow passing these by value if you tag the type as POD (and it really has to be plain old data! no ctor/dtors called!).
  
• Virtual functions are supported but are very unsafe. A virtual function exported to FuBi will get called directly. The vtbl is not consulted ('cause I don't know how to query its contents without a PDB) to figure out which version of the function to call. To prevent Bad Things happening from this I issue a warning for exported virtual functions. Work around this by exporting a non-virtual function that just redirects the call to a virtual function. Then the compiler will do the proper vtbl work. Use FUBI_RENAME (documented below) to give it the same name as the virtual function from Skrit's point of view.
  
• Sometimes the DbgHelp.dll symbol undecoration code just bugs out. This happens more often on Win9x for some reason. Tweak the prototype until it stops.
  
• Exports of constructors, destructors and overloaded operators are not supported.
  
• Exporting functions from nested classes and namespaces is not supported (though documentation and singleton exporting of nested types is supported), mainly because Skrit doesn't need this complication.
  
• Data exports (exporting a variable directly rather than using FUBI_VARIABLE and function-based Get/Set exports) will always get scary warnings. Don't export data.

Reference for dsdll.h

The dsdll.h file (included in the WebMod sample) contains a set of macros and templates that will help make it easier to export functions and types via .dsdll into the game engine. This section documents each of them.

FEX

Put this in front of any function in order to export it. It just resolves to a dllexport tag.

FUBI_DOC( NAME, PARAMS, DOCS )
FUBI_MEMBER_DOC( NAME, PARAMS, DOCS )

These macros will insert documentation about a specific function into the type system, which can be retrieved using the various commands in the Help namespace. Be sure to use the MEMBER version when documenting a member function of a class, and the non-MEMBER version when documenting globals. Here is an example from the game's C++ SkritSupport.h file:

FEX vector_3& MakeVector( float x, float y, float z );
FUBI_DOC( MakeVector, "x,y,z", "Returns a vector composed of x, y, and z.
This is a temporary and will change the next time this function is
called so do not store the results." );

Put the macro next to the function that it is documenting - it is a standalone statement that exports a function which is meant to document another function. NAME is the unquoted name of the function, PARAMS is a quoted string containing a comma-delimited list of parameter names, and DOCS is a quoted string containing documentation on the function.

FUBI_CLASS_DOC( NAME )

This macro is similar to the FUBI_DOC except that it is intended to doc a class or type. Just put it inside of a class definition, where DOCS is a quoted string containing documentation on the type.

FUBI_RENAME( NAME )

This can be used to rename a function from how C++ sees it to how skrit sees it. This is most useful for wrapping virtual functions. Because skrit does not have access to the vtable, the compiler needs to provide the necessary lookup code for virtual functions. To use this macro, wrap it around the function name that is being declared. Here is an example:

virtual void SetVisible( bool bVisible );

FEX void FUBI_RENAME( SetVisible )( bool bVisible )
   { SetVisible( bVisible ); }

This sample is pulled from the code for the UIWindow class. The first function SetVisible() is virtual and for a call from skrit to call the proper dynamically bound method, the compiler must generate a function to select it. The second function that is wrapped with FUBI_RENAME is that function, and will be the one that is called from skrit.

FUBI_VARIABLE ( T, PREFIX, NAME, DOCS )
FUBI_VARIABLE_BYREF( T, PREFIX, NAME, DOCS )
FUBI_VARIABLE_READONLY( T, PREFIX, NAME, DOCS )
FUBI_VARIABLE_BYREF_READONLY( T, PREFIX, NAME, DOCS )
FUBI_VARIABLE_WRITEONLY( T, PREFIX, NAME, DOCS )
FUBI_VARIABLE_BYREF_WRITEONLY( T, PREFIX, NAME, DOCS )

These macros are all variations on a simple concept of simultaneously declaring and exporting functions to set/get members of a class. The READONLY postfix means that the variable cannot be modified from skrit (i.e. no "Set" function), the WRITEONLY postfix means that the variable cannot be read from skrit (i.e. no "Get" function), and the BYREF tag means that set/get is to be done by reference, not by value (needed for non-builtins). Here is an example from the game's code:

struct JobResult
{
   FUBI_VARIABLE_BYREF( double, m_, TimeFinished, "Time when job was finished.");
   FUBI_VARIABLE( eJobResult, m_, Result, "Result at job finish.");
   FUBI_VARIABLE( DWORD, m_, Count, "Times this job was attempted");
   FUBI_VARIABLE( eJobAbstractType, m_, Jat, "Job Abstract Type");
   FUBI_VARIABLE( eJobTraits, m_, Traits, "Job Traits");
   FUBI_VARIABLE( eActionOrigin, m_, Origin, "Job assignment origin");
   FUBI_VARIABLE( Goid, m_, GoalObject, "Job param - goal object");
   FUBI_VARIABLE( Goid, m_, GoalModifier, "Job param - goal modifier");
   FUBI_VARIABLE_BYREF( SiegePos, m_, GoalPosition, "Job param - goal position");
};

The parameters to feed the macros are: T is the type of the variable, PREFIX is the C++ prefix to put on the variable (our convention is m_VarName so m_ is our prefix), NAME is the name of the variable, and DOCS is a quoted string describing the variable.

FUBI_CLASS_INHERIT( T, BASE )
FUBI_CLASS_INHERIT2( T, BASE1, BASE2 )
FUBI_CLASS_INHERIT3( T, BASE1, BASE2, BASE3 )
FUBI_CLASS_INHERIT4( T, BASE1, BASE2, BASE3, BASE4 )

These macros tell the type system about class inheritance. This is useful in skrit so that a pointer to a derived class can be implicitly upcasted to a base class by the skrit compiler. This is a convenience for skrit so that a pointer to a derived class can have base class methods called on it. The T, BASE1…BASE4 parameters are all just the unquoted names of your types, where up to four base classes are supported per type. Put the macro inside of the class definition to export the relationship. Here is an example from the UIEditBox type to say how it is derived from UIWindow:

class UIEditBox : public UIWindow
{
   FUBI_CLASS_INHERIT( UIEditBox, UIWindow );
   ...

Note that skrit does not provide downcasting capabilities, because it has no way to verify if a pointer to a base actually points to a derived type instead. For this you should export your own functions to do the casting, and verify that the cast will work properly before returning the casted pointer. Here is an example from the UI again, for casting a pointer to a UIWindow to the derived UIEditBox:

FEX UIEditBox* QueryDerivedEditBox( UIWindow* base )
{
   if ( (base != NULL) && (base->GetType() == UI_TYPE_EDITBOX) )
   {
      return ( (UIEditBox*)base );
   }
   return ( NULL );
}

FUBI_SINGLETON_CLASS( T, DOCS )
FUBI_SINGLETON_NESTED_CLASS( T, NAME, DOCS )

These macros tell the engine about your singleton types (singletons are objects that have one and only one instance in the system). They are designed to be used with the Singleton templates provided in dsdll.h but basically require that you declare a static function GetSingleton() in your class that returns a reference to the class's singleton.

Put these macros inside of the class definition. The parameters to pass into the macro are "T", the name of the type, and "DOCS", a quoted string documenting the class. These macros actually contain the FUBI_CLASS_DOC macro and pass the DOCS parameter through to it.

The NESTED version of this macro is meant for nested types, where NAME is the name of the nested type combined with the name of its outer type. For example a nested class Tel inside outer class Baz would be named BazTel for the NAME parameter, but left as Tel for the T parameter.

FUBI_POD_CLASS( T )

This macro tells the engine that the given type is POD, or "plain ol' data", meaning that it does not contain any advanced types (such as std::strings) or pointers, and does not require a constructor or destructor to set up or shut down instances of the type. This makes it easy to save/load instances of the class, send them over the network, or pass them by value to local functions.

Note that .dsdll's will probably not find this macro useful currently because savegame and network functions are not exportable, but the macro will remain for future mod capability expansion.

Singleton <T>AutoSingleton <T>OnDemandSingleton <T>

These are all variations on a theme of automating registration of class singletons. Just inherit your type from one of them in order to have it automatically registered. Singleton is for where you allocate/deallocate the singleton yourself, AutoSingleton is used for the compiler to automatically allocate it (statically), and OnDemandSingleton is used to tell the compiler to generate an instance on demand (i.e. the first time it is referenced). For more details, see the documentation inside the dsdll.h file. Background info on the concept of singleton registration via this recursive template trick is documented in another Gem at this address:

http://www.drizzle.com/~scottb/publish/gpgems1_singleton.htm

Frequently Asked Questions

Here are some questions I've been asked since .dsdll's were introduced to the community, plus a few other miscellaneous things that didn't really fit anywhere else in this article.

Coding is too difficult for me, but I need .dsdll functions for my mod! What can I do?

While writing a .dsdll may be difficult for some, it's not a problem for programmers! Why not grab a spot on SourceForge and start a "public .dsdll" project that contains all kinds of functions that anyone might need? Set up a process where people can request features (like "I need a standard deviation function!!") and programmers quickly write and export them, then rebuild a new version of the .dsdll for people to download. One uber-.dsdll could easily serve the vast majority of extra-engine needs for the community.

Can I call engine functions from my .dsdll?

The answer to this is currently "no" for security reasons, although community members have already discovered and published at least one safe workaround that does not break the security model. Be sure to check the forums on the Dungeon Siege community sites for more information.

Can I create my own networked (RPC) functions?

No. FuBi requires direct access to a number of internal Dungeon Siege systems that we simply cannot export. If you want to pass messages over the network, then you will need to set up and communicate through your own sockets via .dsdll functions.

Can I override engine functions with my own?

Yes, mostly! FuBi is built in such a way that it doesn't care where its functions come from, and when it finds duplicate functions it will just ignore the second version. So to override game functions, you just need to make sure that your functions are scanned before the game imports its own functions. Dungeon Siege supports this directly if you rename the DLL's extension to .dsdl0 (that's a zero at the end). The system scans functions in this order: .dsdl0 files alphabetically, DungeonSiege.exe, then .dsdll files alphabetically.
Important: the signatures for your override functions must be identical to the functions they're trying to override, otherwise they will either get ignored, or you'll get a crash.
Unlike normal .dsdll exports, a .dsdl0 that overrides game functions can have crazy unexpected behavior. Your overrides will be called for skrit and incoming RPC's, but not for in-game C++ code (those will still call the old versions). If messing with .dsdll's is considered deep modding, then using .dsdl0 overrides is extreme level 150 burrowing-to-the-Earth's-core modding. So use them with extreme caution, or you may shoot your eye out!

Can I extend the "dev console" with .dsdll's?

Yes! The console lets you type skrit code in directly by prefixing it with a forward slash. So to call any .dsdll functions of yours from the console, just put a slash at the front before typing in the console. You won't have the fancy auto completion, of course, but you'll be able to type in whatever you like, which can be very powerful. To speed your development efforts even further, you may find it useful to create a special debug.dsdll file that contains all kinds of reporting and analysis functions. Then keep a text file open that is filled with commonly used skrit code to access those functions. You can copy commands as needed to the clipboard, then alt-tab back into the game and paste the command directly into the console. It helps to run the game in windowed mode (fullscreen=false on the command line or in the INI file).

What about viruses??

This question comes up a lot, and comes from the basic fact that a DLL has full access to the entire system, and can infect it with a virus, or even reformat the hard drive. This fear is not unique to Dungeon Siege, in fact most games (notably the first person shooters) require you to download and run DLL's as part of any mod that is created. A Dungeon Siege mod that does not use .dsdll's is only capable of "playing in the sandbox" of the engine, and will be safe.
If you would like to learn more about what exactly is involved here, I suggest searching through the Half-Life forums to see how their community has handled this. It is the biggest moddable game on the market, with millions of players.

I can't play multiplayer games with my friends any more! What's wrong?

Simply having a .dsdll available directly modifies the type system of the game, which is nearly guaranteed to cause a latent crash or unexpected behavior in multiplayer if it's not identical to other games joining in. So for security and compatibility reasons, the game takes a checksum of all its .dsdll files on startup, and includes this in the multiplayer info packet, which is used to determine when it's possible for two games to join each other. If one person has a .dsdll installed, then either everyone else must have the same file, or that one person must delete it (or move it elsewhere).
The recommended way to use a .dsdll is demonstrated in the WebMod sample, which is intended to be launched using the webmod.bat file. This will choose the proper .dsdll file, and install the mod and map into the game's resource space during startup. A shortcut can be created by an installer to accomplish similar results (our Yesterhaven map did this as another example).

I love Delphi and hate C++, what can I do?

Have no fear! It's still possible to make .dsdll's, it's just going to be more work. There are a couple options. First is to reverse-engineer the name-mangling format of C++ and modify the function exports from your Delphi DLL to match your function prototypes, or perhaps write a postprocessor to rename your functions directly in the export table to match how the game is expecting. This is a ton of work, of course.
Another option is to learn enough of C++ to write a proxy .dsdll that routes calls from the game straight through to your Delphi-based DLL. This will let you write 99% of your code in your favorite language without having to get your hands too dirty with C++.

How do I export a class?

The most important thing to do is make sure that the .dsdll takes care of all the memory management for your class. To do this, export a function to create an instance, and another to destroy it. The skrit can call the create function to acquire an instance of the class, and then call member functions on it or pass it around to other systems. When it's done with it, it can call the destroy function to delete it. It's also worthwhile including some automatic garbage collection code to clean up as a safety measure.

Conclusion

Well that's it for this article! Hopefully this has provided you with enough information to create your own extensions to Dungeon Siege. Be sure to check out the WebMod sample and the dsdll.h in detail before diving into your own project.