DFHack - Dwarf Fortress Modding Engine News

[p]This is a prerelease build for final testing![/p][h2]Highlights[/h2][h3]Rework for Steam client compatibility[/h3][p]The overwhelming purpose for this release is to rework DFHack so that it will work seamlessly when installed via Steam. Steam recently changed its client in a way that made our prior method of installing DFHack where DF could "see" it stop working. With a single tiny exception, the entirety of this update is to facilitate being able to install DFHack via Steam with a minimum of user hassle.[/p][h2]Highlights[/h2][p]Rework for Steam client compatibility; Windows console encoding[/p][h3]Rework for Steam client compatibility[/h3][p]The overwhelming purpose for this release is to rework DFHack so that it will work seamlessly when installed via Steam. Steam recently changed its client in a way that made our prior method of installing DFHack where DF could "see" it stop working. With a single tiny exception, the entirety of this update is to facilitate being able to install DFHack via Steam with a minimum of user hassle.[/p][h3]Windows console encoding[/h3][p]Dwarf Fortress recently was updated to force UTF-8 for all interactions with the operating system in Windows; however, this change did not reach to the text console used by DFHack on Windows, which is still defaulting to the user's default system code page. This has caused some anomalous behavior for users whose default system code page is not already UTF-8. This update includes a tiny change that forces this console to also use UTF-8 regardless of the user's default system code page.[/p][h2]Announcements[/h2][h3]Special instructions for this update[/h3][p]One of the side effects of the changes made by the Steam client is that people who were using DFHack before the Steam client change now have a "stranded" installation of DFHack still living in their Dwarf Fortress installation folder. This installation must be removed for DFHack to work correctly going forward. Our installer has code to detect this. On Windows, the user will be asked for permission to remove the legacy installation, while on Linux a popup with instructions on how to remove the legacy installation will be displayed.[/p][p]In addition, for technical reasons, DFHack can no longer completely install itself into Dwarf Fortress solely by installing the app on Steam. As a result, it is now necessary to launch from the DFHack application at least once after installing DFHack so DFHack can install itself into Dwarf Fortress. After this, it should be possible to launch from either Dwarf Fortress or DFHack the same way as before.[/p][p]If you have problems with DFHack not working after installing this update which are not resolved by launching the DFHack application, we recommend the following process:[/p]

• [p]Uninstall DFHack via Steam[/p]
• [p]Remove the following files and folders from the Dwarf Fortress installation folder:[/p]
  • [p][c]hack[/c] (entire folder/directory)[/p]
  • [p][c]stonesense[/c] (entire folder/directory)[/p]
  • [p][c]binpatch.exe[/c] (Windows) / [c]binpatch[/c] (Linux)[/p]
  • [p][c]dfhack-run.exe[/c] (Windows) / [c]dfhack-run[/c] (Linux)[/p]
  • [p]all files named [c]allegro_*.dll[/c] (Windows) or [c]liballegro_*.so[/c] (Linux)[/p]
  • [p][c]dfhack-client.dll[/c] (Windows) / [c]libdfhack-client.so[/c] (Linux)[/p]
  • [p][c]dfhooks_dfhack.dll[/c] (Windows) / [c]libdfhooks_dfhack.so[/c] (Linux)[/p]
  • [p][c]lua53.dll[/c] (Windows) / [c]liblua53.dll[/c] (Linux)[/p]
  • [p][c]protobuf-lite.dll[/c] (Windows) / [c]libprotobuf-lite.so[/c] (Linux)[/p]
• [p]Reinstall DFHack via Steam[/p]
• [p]Launch DFHack once[/p]

[p]DFHack Lua scripters need to be aware that the [c]hack[/c] folder is no longer guaranteed to be in [c]hack[/c]. A path to it can be obtained with the [c]dfhack.getHackPath()[/c] call. While all standard scripts have been updated to reflect this, user written scripts have not. The [c]dfhack-config[/c] folder has not moved with this update.[/p][h3]DFHack Steam Cloud support[/h3][p]DFHack has, for some time now, used Steam Auto-Cloud to synchronize the [c]dfhack-config[/c] folder that contains each user's personal configuration of DFHack. For technical reasons, we can no longer do this; as a result, we will be turning off DFHack Steam Cloud. You may wish to consider manually backing up the contents of your [c]dfhack-config[/c] folder.[/p][h3]Legacy support branches on Steam[/h3][p]We will not be porting support for the new installation to the legacy support branches currently available on Steam. As these branches won't work for Steam users anyway (as they install to the wrong place), there's no value in keeping them around, and so we will be deleting them. Note that you can still download legacy versions of DFHack from GitHub for far further back than we keep on Steam, and these legacy versions will still work as they always have by manually installing them into the installation folder of the matching version of Dwarf Fortress.[/p][p]We will continue to provide legacy support branches on Steam going forward, subject to the limits on how many branches Steam allows, of course.[/p][h3]PSAs[/h3][p]As always, remember that, just like the vanilla DF game, DFHack tools can also have bugs. It is a good idea to save often and keep backups of the forts that you care about.[/p][p]Some DFHack tools that worked in previous (pre-Steam) versions of DF have not been updated yet and are marked with the "unavailable" tag in their docs. If you try to run them, they will show a warning and exit immediately. You can run the command again to override the warning (though of course the tools may not work). We make no guarantees of reliability for the tools that are marked as "unavailable".[/p][p]The in-game interface for running DFHack commands ([c]gui/launcher[/c]) will not show "unavailable" tools by default. You can still run them if you know their names, or you can turn on dev mode by hitting Ctrl-D while in [c]gui/launcher[/c] and they will be added to the autocomplete list. Some tools listed as "unavailable" in the docs do not compile yet and are not accessible at all, even when in dev mode.[/p][p]If you see a tool complaining about the lack of a cursor, know that it's referring to the keyboard cursor (which used to be the only real option in Dwarf Fortress). You can enable the keyboard cursor by entering mining mode or selecting the dump/forbid tool and hitting Alt-K (the DFHack keybinding for [c]toggle-kbd-cursor[/c]). We're working on making DFHack tools more mouse-aware and accessible so this step isn't necessary in the future.[/p][h2]Changelog[/h2][p]New tools, fixes, and improvements[/p][h3]Fixes[/h3]

• [p]Core: Windows console will always use UTF-8 regardless of system code page settings[/p]
• [p]Steam launcher: Switch to injection strategy, allowing Dwarf Fortress and DFHack to be installed in disparate locations[/p]

[h3]Misc Improvements[/h3]

• [p]Make DFHack relocatable so that it doesn't depend on being fully co-installed with Dwarf Fortress[/p]

[h2][/h2]

[h2]Situation report[/h2][p]Work is now nearly complete on updating DFHack to once again install (mostly) seamlessly with the new changes to Steam's client. There are several significant changes that I want to warn the community of in advance:[/p][p]First, most of you will have to do a one-time cleanup of your previous DFHack installation, nestled as it is in Dwarf Fortress's install folder, either because it was installed there by the Steam installer previously or because you manually copied it there to keep things working. While we could try to write a tool to clean these up automatically, I am not comfortable with silently deleting files that someone might want to keep, so we're going to leave this to you to do on your own. Preliminary instructions on how to do this are below.[/p][p]Second, we are going to disable Steam Cloud synchronization of DFHack's configuration folder ([c]dfhack-config[/c]). This is because, for now, we are going to leave this folder in DF's folder (where it was before) which we cannot configure to synchronize with Steam Auto-Cloud (this is a technical limitation of Steam Auto-Cloud). We're still trying to figure out how to best deal with this folder going forward; all of the potential solutions have drawbacks and we haven't decided what the best way to move forward is. Thus, you might want to make a manual backup of your [c]dfhack-config[/c] folder before installing this coming update.[/p][p]Third, we are going to drop all of the existing legacy branches of DFHack currently on Steam. They won't work seamlessly anyway due to the installation path changes, so the convenience value they once offered is now nonexistent and so there's no value in continuing to provide them. Note that we willl still have legacy versions of DFHack available from Github, which will still work (just as they always have) simply by dropping them directly into DF's installation folder. [/p][p]None of the changes we're making with this update will directly impact users who choose to install DFHack by downloading it from Github; the builds published there will continue to work, as they always have, by unpacking them directly into DF's installation folder. Also, if you subscribe to DFHack on Steam, but not DF, and are using the Classic version of DF that you can download from Bay12 installed into DFHack's installation folder, again, you should experience no change in the behavior of the application (except for the Steam Cloud issue mentioned above).[/p][h2]Test build is available![/h2][p]A test build of DFHack that works with the current release of DF and works with the new Steam client is available on our "testing" branch. If you choose to use this build, be aware of the following EXTREMELY IMPORTANT INFORMATION:[/p]

• [p]This build does not self-install. You will have to launch from DFHack once after installation in order for it to detect and install into DF. After this one launch, you can launch from either DF or DFHack. We're hopeful that we can get it to be self-installing for Windows users, but this still has to be implemented and tested in a way that won't break existing builds or cause unexpected behavior.[/p]
• [p]This build does not clean out any previous DFHack installation in your DF install folder. While this has not been extensively tested, it's likely that if you have a previous installation in your DF folder, it will take precedence over the new version. You will be able to tell with the test build by the DFHack version displayed at launch: if there isn't a [c](git: nnnnnnn)[/c] tag after the version number, you're still running on the public release version and need to clean that old installation out to proceed.[/p]
• [p]This build has only been tested on Windows and has not been extensively tested, especially for exceptional conditions. If you use it, please provide feedback, especially if you run into problems.[/p]
• [p]Going back to the release version will require manually copying that install into your DF folder; you won't be able to just switch back to the public branch without further steps to get DFHack back which you will have to do manually.[/p]

[h3]How to clean out a prior DFHack install installed by Steam[/h3][p]Go to Dwarf Fortress's installation folder (Manage > Browse Local FIles). (Let me again stress that this is under the Dwarf Fortress app, not under DFHack's app.) Delete the following files and folders/directories, if they are present:[/p]

• [p][c]hack[/c] (entire folder/directory)[/p]
• [p][c]stonesense[/c] (entire folder/directory)[/p]
• [p][c]binpatch.exe[/c] (Windows) / [c]binpatch[/c] (Linux)[/p]
• [p][c]dfhack-run.exe[/c] (Windows) / [c]dfhack-run[/c] (Linux)[/p]
• [p]all files named [c]allegro_*.dll[/c] (Windows) or [c]liballegro_*.so[/c] (Linux)[/p]
• [p][c]dfhack-client.dll[/c] (Windows) / [c]libdfhack-client.so[/c] (Linux)[/p]
• [p][c]dfhooks_dfhack.dll[/c] (Windows) / [c]libdfhooks_dfhack.so[/c] (Linux)[/p]
• [p][c]lua53.dll[/c] (Windows) / [c]liblua53.dll[/c] (Linux)[/p]
• [p][c]protobuf-lite.dll[/c] (Windows) / [c]libprotobuf-lite.so[/c] (Linux)[/p]

[p]Do not delete [c]dfhooks.dll[/c] (Windows), [c]libdfhooks.so[/c] (Linux), or [c]dfhooks_dfhack.ini[/c] (both WIndows and Linux); these files are injected by DFHack's one-time installer and are how DFHack connects to DF. (If you do delete them, launching from DFHack will reinstate them.)[/p][p]The most important one to delete is [c]dfhooks_dfhack.dll[/c] (Windows) / [c]libdfhooks_dfhack.so[/c] (Linux); if you don't delete this one, you'll almost certainly get the old version of DFHack instead of the new one.[/p]

[p]I'm sure some of you have seen this message over the past few weeks and not understood why you're getting it.[/p][p]Recently, Steam changed the way the Steam client installs applications. Specifically, Steam changed their installer so that two apps can no longer install into the same location. On the whole this was probably a good choice by Steam, since it definitely prevents one Steam app from maliciously overwriting another Steam app. However, DFHack has always relied on the ability to do this as its installation method, this is because of the way a DF extension module (which is what DFHack is) connects to DF. (And, yes, we have and have always had Bay12's explicit permission to do this.) As a result of Steam's decision here, DFHack no longer seamlessly links up with DF, which leads to the above message and makes DFHack not work, which obviously makes us all sad.[/p][p]Some people have also asked why DFHack is not a DF mod, and so I'm going to take a moment to address this. [/p][p]DFHack is not a DF mod because DF's modding system only allows the mod to provide graphics assets and text files to be parsed as DF "raws". DFHack is neither a graphics asset nor a text file to be parsed as raws, and so it cannot function as a DF mod. DF mods simply cannot do what DFHack does.[/p][p]Instead, DFHack operates as a DF extension module, which works by dropping a specifically named file in a specific location in the DF installation that DF detects during its startup and connects to if that file meets certain specified criteria. There is, at present, no way for a workshop mod to drop a file in this location, and so no workshop mod can be, by itself, a DF extension module. [/p][p]In the past Steam allowed DFHack to install in the same folder as DF, and so we could just passively use Steam's installer to drop the required file in the right location. Steam has changed their installer so this is no longer possible, so we will now instead have to use some sort of active method to put the the required file in the right location. This is actually the easy part, which is why I haven't done it yet.[/p][p]We could simply have rewritten our launcher to to find DF and copy all of DFHack into DF, creating the same situation that we had when Steam just let us install DFHack in the same location as DF, but this approach makes updating much harder (we'd have to detect when an update occurred, remove all of the old version, and install all of the new version, which is a pain) so we're not doing that; instead, we will only be copying two small files that will hopefully change very infrequently, and, being small, will be easy to copy even if they do change. On Windows, this will be completely transparent to the user; however, on Linux, players will have to manually launch DFHack once after installing DFHack for the first time. The discrepancy here is because Steam does not provide for custom install scripting in the Linux version of its client, and so on Windows we can automatically run the post-update script to find and copy into DF's folder, but on Linux the user will have to "do something" to run the post-update script, so what we're going to be doing is having the DFHack launcher check at startup if the post-update script has been run, and if it hasn't run it itself. This is the part I haven't written yet, because it's frankly easy and I wanted to save the easy part for last.[/p][p]The harder part is that DFHack has long operated on the assumption that it will live in the same folder as DF, an assumption we can no longer make. Thus, we have had to go through all of DFHack's code and rewrite every place where the code assumed that DFHack was installed in DF's folder, so as to make DFHack work no matter where DFHack is installed. At this point, I think the programming aspect of this is complete, but we still need to test to verify that it works correctly under what we hope are a sufficiently wide range of scenarios to cover the majority of player use cases.[/p][p]While this testing continues, I'm going to now go to work on the "easy" part, updating the launcher, and once both of these are done, we'll publish a PTB on Steam in our "testing" branch, hopefully in the next week or three.[/p][p]We do apologize for all the trouble this has created for players. If Steam gave us any advance notice of this change, we didn't see it, and so it pretty much caught us unawares. And as DFHack is and always has been a 100% volunteer project, we obviously have limited resources to develop, test, and implement changes, especially in a hurry. [/p][p]For the DFHack Team,[/p][p]rome of oxtrot[/p]

[h2]Highlights[/h2][h3]Bug fixes[/h3][p]This release fixes several bugs, most notably the [c]rename[/c] tool, which was hanging, and several tools that managed work orders that were creating them wrong due to an underlying infrastructure change that interacted in an unexpected way.[/p][p]In addition, a change in DF itself caused the DFHack console to work differently which caused problems displaying names with accent marks; this has been fixed, and the DFHack console will now work more reliably on Windows systems.[/p][h2]Announcements[/h2][h3]Steam client issues [/h3][h3]IMPORTANT PLEASE READ THIS[/h3][p]Steam has changed the way the Steam client installed DFHack in a way that prevents it from working automatically. Until we update our deployment mechanics, this means that users using DFHack via Steam may have to manually copy DFHack from the location Steam installs DFHack into, into DF's folder. Steam users who have to do this may also have to launch using the "Dwarf Fortress" entry in their Steam library, not from the "DFHack" entry. We have been aware of this as a coming thing for a while now, but we didn't have time to get it addressed before Steam's changes went live. We apologize for the inconvenience. Fixing this is a top priority right now, and and we expect to have this fixed within the next few weeks.[/p][h3]PSAs[/h3][p]As always, remember that, just like the vanilla DF game, DFHack tools can also have bugs. It is a good idea to save often and keep backups of the forts that you care about.[/p][p]Some DFHack tools that worked in previous (pre-Steam) versions of DF have not been updated yet and are marked with the "unavailable" tag in their docs. If you try to run them, they will show a warning and exit immediately. You can run the command again to override the warning (though of course the tools may not work). We make no guarantees of reliability for the tools that are marked as "unavailable".[/p][p]The in-game interface for running DFHack commands ([c]gui/launcher[/c]) will not show "unavailable" tools by default. You can still run them if you know their names, or you can turn on dev mode by hitting Ctrl-D while in [c]gui/launcher[/c] and they will be added to the autocomplete list. Some tools listed as "unavailable" in the docs do not compile yet and are not accessible at all, even when in dev mode.[/p][p]If you see a tool complaining about the lack of a cursor, know that it's referring to the keyboard cursor (which used to be the only real option in Dwarf Fortress). You can enable the keyboard cursor by entering mining mode or selecting the dump/forbid tool and hitting Alt-K (the DFHack keybinding for [c]toggle-kbd-cursor[/c]). We're working on making DFHack tools more mouse-aware and accessible so this step isn't necessary in the future.[/p][h2]Changelog[/h2][h3]Fixes[/h3]

• [p][c]autoclothing[/c], [c]autoslab[/c], [c]tailor[/c]: orders will no longer be created with a repetition frequency of [c]NONE[/c][/p]
• [p][c]gui/rename[/c]: skip [c]NONE[/c] when iterating through language name options[/p]
• [p][c]quickfort[/c]: work orders will no longer be created with a repetition frequency of [c]NONE[/c][/p]

[h3]Misc Improvements[/h3]

• [p]General: DFHack will unconditionally use UTF-8 for the console on Windows, now that DF forces the process effective system code page to 65001 during startup[/p]

[h3]Structures[/h3]

• [p]reordered XML attributes for consistency: [c]ret-type[/c], [c]base-type[/c], [c]type-name[/c], and [c]pointer-type[/c] now appear first (and in that exact order)[/p]

[h2][/h2]

[p]Please report any issues (or feature requests) on the DFHack GitHub issue tracker. When reporting issues, please upload a zip file of your savegame and a zip file of your [c]mods[/c] directory to the cloud and add links to the GitHub issue. Make sure your files are downloadable by "everyone with the link". We need your savegame to reproduce the problem and test the fix, and we need your active mods so we can load your savegame. Issues with savegames and mods attached get fixed first![/p][h2]Highlights[/h2][h3]Dwarf Fortress 53.11 compatibility[/h3][p]This release is primarily for compatibility with Dwarf Fortress 53.11.[/p][h3]One bug fix[/h3][p]We fixed one bug in 53.10-r2 in this release; the "hates combat" filter on the squad selection filter will now work correctly.[/p][h2]Announcements[/h2][h3]Known defect[/h3][p]53.10-r2 has a reported defect in the "rename" tool which we have not had time to fix; this release also has this defect. We are working to find and fix this bug and will release an update as soon as we can once we've got a fix.[/p][h3]Issues with Steam Client Beta[/h3][p]The beta of the Steam client alters the way Steam applications are deployed in a way that impacts DFHack, and we have had reports of DFHack not working as expected for people who are using the Steam beta client. We are aware of this issue, have a plan to mitigate it in development, and hope to have a solution in place before the change Steam is making makes it to the release client.[/p][h3]PSAs[/h3][p]As always, remember that, just like the vanilla DF game, DFHack tools can also have bugs. It is a good idea to save often and keep backups of the forts that you care about.[/p][p]Some DFHack tools that worked in previous (pre-Steam) versions of DF have not been updated yet and are marked with the "unavailable" tag in their docs. If you try to run them, they will show a warning and exit immediately. You can run the command again to override the warning (though of course the tools may not work). We make no guarantees of reliability for the tools that are marked as "unavailable".[/p][p]The in-game interface for running DFHack commands ([c]gui/launcher[/c]) will not show "unavailable" tools by default. You can still run them if you know their names, or you can turn on dev mode by hitting Ctrl-D while in [c]gui/launcher[/c] and they will be added to the autocomplete list. Some tools listed as "unavailable" in the docs do not compile yet and are not accessible at all, even when in dev mode.[/p][p]If you see a tool complaining about the lack of a cursor, know that it's referring to the keyboard cursor (which used to be the only real option in Dwarf Fortress). You can enable the keyboard cursor by entering mining mode or selecting the dump/forbid tool and hitting Alt-K (the DFHack keybinding for [c]toggle-kbd-cursor[/c]). We're working on making DFHack tools more mouse-aware and accessible so this step isn't necessary in the future.[/p][h2]Changelog[/h2][h3]Fixes[/h3]

• [p][c]sort[/c]: correct misspelling of [c]PERSEVERENCE[/c]; fixes "hates combat" filter in squad selection screen[/p]

[h3]Structures[/h3]

• [p]added codegen support for [c]static-wstring[/c] ([c]wchar_t
• [/c]), required to support DF 53.11[/p]