nifxml spec formalization #75
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Scene/SceneDesc were relatively old and unused and also quite complex to implement so they never got ported to XML from the 010 templates. Note: Half the files that actually use Scene/SceneDesc are in Empire Earth III, which its version 20.2.0.8 behaves differently for these blocks than other 20.2.0.8 versions. The only way to tell the difference is by the file extension, which is .nifcache. This is beyond the scope of nif.xml, and so only the game files that match the version conditions of Gamebryo proper are supported.
Several objects used separate center/radius when it's actually an NiBound object. This was apparent with the addition of additional data to NiBound for custom 20.3.0.9 versions for Divinity 2.
Only MdlManBinary.nif cannot be read for Divinity 2, and it appears to be some kind of rig filled with custom classes serialized to NIF. There is no desire to decode this single NIF with 10 custom classes. Also the Divinity 2 XML files that have a Gamebryo header are ignored as they appear to just contain XML strings in more custom classes.
In C++ and other langs, this compound member gets generated to "union" which clashes with the reserved keyword.
Enhanced support for WorldShift. In order to not complicate the objects, I fudged the official version conditions slightly to accommodate 10.4.0.1 given that no official versions exist between 10.2 and 20.0. Aside from the new custom block, only modified the conditions that were already there.
Made `<version>` mean distinct combination of header values which produce identical block layouts. This meant splitting up several main Versions into many based on User Version or Bethesda Version. id = Used to uniquely identify this collection of NIF versions. Can also be used for enum generation, map/dictionary keys. user = List of User Versions which are functionally equivalent bsver = List of Bethesda Versions which are functionally equivalent custom = If user/bsver are default but the Version is actually specific to a developer and not an official Gamebryo version supported = Whether or not the XML fully supports reading/writing of this version. ext = List of custom file extensions associated with this version.
The attempt here was to exclude Bethesda from several rows on NiBoneLODController, since in the XML it inherits from NiBoneLODController for its own custom block that does not have those rows. The reasoning is actually that it does *not* inherit from NiBoneLODController in the engine, but the inheritance will stay fudged here for now.
Formalization of what makes type and member name strings valid requires the removal of non-alphanumerics with the exception of the colon.
They were kept separate for clarity in actual code using the enumerations; clarity that doesn't transfer over to XML which is using the raw integers. ``` (A & X) != 0 && (A & Y) != 0 && (A & Z) == 0 ``` Is the same as ``` (A & (X|Y|Z)) == (X|Y) ``` Basically != 0 you want that particular bit on the RHS of the expression, and == 0 you do not. Anything tested you put on the LHS. Also switched the order of HalfVector3/Vector3 as the larger type should come first.
A concise way of listing which versions are most important to each game.
Each game should have only one set of `{{}}`.
Note: In the case of Oblivion, there are two "primary" versions.. 20.0.0.4 (10) is used for KF and 20.0.0.5 is used for NIF. Thus I changed the "game name" to `{{Oblivion KF}}` for 20.0.0.4 (10).
First batch of fixed issues detected by the new linter. Also some slight cleanup, plus an interitance fix a Bethesda particle class.
First-pass implementation of `since`/`until` and `versions`. Avoided nearly all Bethesda blocks for now, as they require the token system or would become absurdly repetitious.
Moved all the DEM specific stuff out of NiMesh and into compounds. Did some decoding while I was at it. There are many small customizations to other blocks for DEM, but I am not including them in this commit. Regardless, the existing XML only worked for DEM1 and now at least some DEM2 will load fully.
Added V10_2_0_0__1 and scoped NiFurSpringController to it since it is a custom block only for that game.
Fixed inherited duplicate names, as well as the layout of BSTriShape for Particle emitter geometry data in SSE. Left the name for Dynamic Data as "Vertices" though it is technically just called "Dynamic Data" in engine. It contains the vertices and bitangent X for any BSDynamicTriShape, so the naming is apt.
Removed redundant descriptions for unknown values, removed incorrect descriptions for previously unknown values. Any custom extension unknowns were prefixed with their relevant game. Anything that was "Unk" was changed to "Unknown" for consistency. A few inherited duplicates were also fixed in the process.
Hey look at that, linting helped immediately. Too bad I forgot to lint before committing the last commit.
The `ver` attribute was never actually formalized and just meant as a shorcut for using both ver1/ver2 on the same version. `userver` was almost never used and `userver2` was becoming increasingly uncommon as most comparisons were switched to inequalities. To make things easier, these should just be part of the expression.
Tokens are grouped by name and associated with a list of attributes. Only listed attributes should be processed for tokens of that name. Order of token groups matters. Token strings can themselves include tokens, so any token group used in other tokens needs to come after.
Using tokens from niftools#70 it is now feasible to version the Bethesda blocks.
Cleaned up class naming e.g. Descriptor->CInfo to match the engine and other classes already named that way. Added hk/bhk to front of type names where appropriate. Changed Info/Property names to be consistent e.g. "Rigid Body Info", "Entity Info" instead of CInfo. Changed all "Unused" fields to a consistent naming scheme, using byte array types and binary="true". Class hierarchy fixes. Class documentation complete overhaul. Some existing naming collides with new naming of different fields, e.g. bhkMoppBvTreeShape "Scale" used to be "Shape Scale" and now "Scale" is "MOPP Code" -> "Scale".
Made a correct bitfield using `uint64` as the underlying type, split into nibbles for the sizes and offsets of the vertex attributes. Changed VertexFlags to VertexAttributes and it no longer has the empty nibble in front. ARG passing of the BSVertexDesc is treated as the underlying numeric value of the bitfield, so that the attributes can be passed into BSVertexData using `#RSH# 44` instead of the member accessor syntax. This makes the first use of the basic `uint64` type, so (de)serialization now requires support of 64-bit values.
Also use `true` instead of `1` to match other XML booleans.
The correct name for StringIndex is NiFixedString. Also added NiTFixedStringMap which is used once in the NiPhysX module.
Second default values pass, plus some general changes to accommodate them.
For niftools#72 Renamed modules from Gamebryo: NiParticle, NiPSParticle. As of Gamebryo 2.5, old NiParticle was renamed NiParticleOld and the new NiMesh-based particle system was named NiParticle. I didn't like this naming scheme given NiParticleOld is more important to us. Ni Modules not in Gamebryo: NiCustom, NiLegacy These are to handle third party blocks and blocks from NetImmerse (pre-Gamebryo). Ignored from Gamebryo: NiCollision, NiPortal, NiPhysXParticle These were instead absorbed into other modules. Bethesda modules: BSMain, BSAnimation, BSParticle, BSHavok, BSLegacy Animation, Particle, Havok, and Legacy are not modules used by Bethesda. Legacy is for Morrowind-specific blocks (NetImmerse). In addition, BSShader is a module for all the shader related blocks and that has all been folded into BSMain.
For niftools#72 So that first party modules can be singled out if needed.
Improved class descriptions and fixed Flags values and defaults.
For niftools#76 Implement initial subclass defaults. Had to add additional attribute to deal with versioning of defaults, like block versioning. Added more defaults to normal fields. Added default to Vertex Color array, also enforcing that members with `arr1` can have a `default`.
Specify SizedString vs NiFixedString where appropriate. The compound type `string` is an extra level of indirection in many cases, where the block can never be used on versions less than 20.1.0.3 where the CString to NiFixedString transition happened. NiFixedString is a basic type, so using it where possible is recommended over the `string` compound type. `string` could NOT be changed when dealing with multiple rows with the same name but version exclusive, such as "Accum Root Name". Changed ShortString to ExportString as it was already documented as being exclusive to the Bethesda stream header. Added SizedString with ushort length type, for the SSF File path.
BSGeometrySegmentData - Combined rows which did not be separated by version. Adopted better naming from FO4. AspectFlags type for NiParticlesData for BS. Changed the unused W component of what should be a Vector4 for Vertex to 'Unused W' for all rows. Should seriously consider making these all Vector4s for endian correctness. Misc comment cleanup.
For niftools#70 and niftools#76 Added tokens for range attribute strings. Did initial pass of ranges, many from Bethesda's official export scripts for FO4, which limit the values in the UI. Also have begun using data analysis to find implied min/max ranges (as well as sane defaults).
The way that BSPackedCombinedSharedGeomDataExtra refers to the shared geometry was decoded. It stores the hash of the filename and the offset to the data.
NetImmerse NiParticle (called NiOldParticle after Gamebryo 1.1) was severely undecoded. This affects several NetImmerse games like Morrowind and Freedom Force. The per-particle info in NiOldParticle actually remained identical to the class in NiParticle, which is named "NiParticleInfo". It did however have a different name in NetImmerse which was "NiPerParticleData". Since they were the same data, the two compounds were merged and NiPSysData was updated.
Decoded some unknown fields and blocks for WorldShift and Guild 2. NiSourceTexture is now correctly decoded. The unknown byte actually controls whether the image data Ref gets serialized or not, but it was always 1. The addition of another conditional value increases the complexity for Pixel Data to 3 rows, all exclusive by version or condition.
Even though in these cases it doesn't matter because of version exclusivity, keep them ordered by type instead of grouped by version.
Renamed BSLightingShaderPropertyShaderType and its associated name field ("Skyrim Shader Type" -> "Shader Type").
BSDistantObjectInstancedNode is mostly undecoded but fully readable.
Treatment of shader flags is preliminary and will change over time.
So that "value" can be used for bit patterns (collections of individual bits).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for all of #55, #69, #70, #71, #72, #73, and #74 as I work on them. I have done initial commits for #69, #71, and incorporates fixes for issues found by the linter in #74 but have not committed the linter itself yet.
It also includes non-spec fixes that came up along the way such as decoding of several missed blocks.