Asset Load/Serialization Versioning issue

Asset Load/Serialization Versioning issue

[APOLOGIES IN ADVANCE FOR THE LONG POST - JUST WANT TO BE THOROUGH!]

{Also - i've tried umpteen Edits to try and keep this formatted with
line breaks etc but it refuses...}

Info:

-> Im running the latest version of the Content Tools and the binary
only Physics+Animation modules.

-> I can successfully create runtime primitives and display them on
the VDB.

-> Using 3ds Max 2012 with the Havok Export Tool Plug In.

-> Using Visual Studio 2010 Express with a view to integrate with
OpenGL

So i've been trying for the better part of a week now to successfully
import some physics and animation data into my engine.

I've tried a vast variety of Havok, my own and Skyrim/Fallout hkt/hkx
files to no avail. I've read the Content Tools and Serialization/Versioning
sections over and over and they just don't provide enough detailed explanation
for my issue. Obviously I've googled extensively as well as searched the Intel
Forums.

Everytime I try to load in an asset I get the following Assert error
message ( or a similar derivative of the versioning problem ) :

Unable to version data of class hkaAnnotationTrackAnnotation, version
0x00000000

Patching to latest version failed. Have you registered the necessary
patches? e.g. hkFeature_serializeRegisterKeycodePatches() See the hkError
output for more detailsDBGHELP: SymSrv load failure: symsrv.dll

Now I'm pretty sure I'm not smart enough to figure out the whole runtime
patching thing - but I was forced to go through a few tutorials in order to
hack and figure out how to apply the content patching. I tried several dabbles
and eventually gave up because I couldnt really see how the demo framework tied
in with the whole version patching.

So the next step was to use the AssetCC2 tools to try and convert the
Havok_Girl_Rig file located in resources (hkRig_4011.hkx). I tried that to no
avail really. It converts a .hkx over to a .hkt tagfile for me, but doesnt
really do anything to resolve the versioning issue. AssetCC1 doesnt allow me to
do any conversions - just says something about "hkx is mutually
exclusive" to another format etc. An example of what I've been doing is
something like :

AssetCC2 -g hkRig_1404.hkx hkRig.hkt

I did have one success and that was when I used AssetCC2 to convert the
SimpleLoad examples' simple.hkx to simple.hkt. That basically allowed me to
serialize the physicsData but obviously didnt have any Skeleton/Animation data
in it for me to test the Animation import. That is basically the ONLY file that
I've been able to get working without a version error - I suspect because its a
really simple file in terms of contents.

I've literally tried loads of methods from NifTools on converted .KF to
.Nif and then exporting that as a .hkx/ .hkt/ .xml etc. I'm out of options at
this point and bashing my skull into my keyboard - but I'd really like to just
persevere with the Havok tools as I really believe they are a great, powerful
product.

The more concerning issue for me is that even if I create a brand new simple asset in Max and then export it from the havok plugin - I still get the versioning issue... Which I dont understand because surely I'm using the latest Content Tools with the latest Havok (2011.3 i believe).

1.) Am I using AssetCC2 correctly to
convert the example Havok files from the older rule .hkx to new .hkt? And is
this conversion from older packfile to newer tagfile meant to auto patch to the
latest version?

2.)I would
really appreciate some insight into what I'm doing incorrectly in terms of trying
to import the assets and how come even my new assets are giving me versioning
issues. I'd prefer not to go the complicated patching route as I intend to
create and use my own modern assets.

3.)Would it
have something to do with how I export from the Havok plugin in Max?
I've tried to create a single cube and attach 2 bones + skin + IK +
simple animation(or combinations of those) and then I've tried various export
filter from just Skeleton to only Animation or even only the Inches to Meters
transform.

I've attached my 2 source code files in the hopes that it may provide
some insight into my issue. (I've noted inside where the Assert errors occur).

As you can see I keep throwing in as many includes as I can get my hands
on for troubleshooting.

Also - in order to pre-empt any of the include/define issues I've played
around with alot of the following:

#define HK_EXCLUDE_FEATURE_MemoryTracker

#define HK_EXCLUDE_FEATURE_SerializeDeprecatedPre700

#define HK_EXCLUDE_FEATURE_RegisterVersionPatches

as well as :

#include

#include

#include

Thanks a bunch for any assistance/insight you may be able to offer!!!


The End

13 posts / novo 0
Último post
Para obter mais informações sobre otimizações de compiladores, consulte Aviso sobre otimizações.
imagem de havok_joe

Hi danothom:

Seems like you have a versioning problem, so let's break it
down a bit.

First problem, you want to load some old version model files.

In fact, most of the Havok Demos are using quite old version
data (3.0).Havok provides a way

to do this conversion.

1. Let havok patch the file for you while loading it.

Havok will do it automatically after you included
hkProductFeatures.cxx in your code. BUT,

because you want the system to do it, you SHOULD NOT have
those two defines

HK_EXCLUDE_FEATURE_SerializeDeprecatedPre700

HK_EXCLUDE_FEATURE_RegisterVersionPatches

in your code.

Because, obviously, those two defines will tell havok to not
patch the file for you (it strips all versioning-related code, which you'll
need).

2. Use Packfile Convert tool AssetCc2

This tool can update asset versions and can only save the
latest sdk version.If you use this tool to convert the asset and then load it,
it is not necessary for Havok topatch the data anymore. As the reason, you can enable those
two defines to exclude the featureand make your code size a bit small.

Here is the documentation you can refer to.

Common Havok Components / Serialization / File Formats

For now I'd suggest using option #1 from above since you
won't have to worry about using the AssetCc2 tool and can rule out the
possibility of your tool version being different from your runtime version.

For your second problem, you want to serialize HAVOK_PHYSICS
and HAVOK_ANIMATION objects in a binary file format.

To do that, you should make sure all the necessary classes
are registered and all the features youneed are included. This should get taken care of
automatically as a result of including keycode.cxx before hkProductFeatures.cxx
(you'll note that keycode.cxx should define HK_FEATURE_PRODUCT_ANIMATION and
HK_FEATURE_PRODUCT_PHYSICS to enable those products, and hkProductFeatures.cxx
should by default include hkKeyCodeClasses.h to define
HK_FEATURE_REFLECTION_PHYSICS/HK_FEATURE_REFLECTION_ANIMATION and setup which
classes can be serialized/loaded).

Hope this helps. And if you have any question, please attach
some of your code and let me know.

Thanks

Joe

Developer Support Engineer

Havok

www.havok.com

Joe Developer Support Engineer Havok www.havok.com

Hi Joe,

Thanks for the response - it was
quite informative. I ended up using most of your advice as a golden source
starting point and then playing around with the ordering and assigment of
includes and some of the feature excluded. It seems that some of the includes
really don't play nicely when split between the header and cpp files...

I think one of the main
resolutions was changing to hkKeyCodeClasses.hinstead of using :

#define HK_CLASSES_FILE

Anyway, with your guidance I
finally got it to serialize the Havok Girl rig after converting it to a tagfile
using AssetCC2.

Follow up questions for when you
have a moment :

1.) I did actually upload my
source files to this site but they didnt seem to attach anywhere on my
post.
-> Do I then need to link in the actual hyperlink to each individual file on
the post or something?

2.) There seems to be quite a bit
of deprecated functionality and code floating about on the forums, demo
examples and net etc. I'm getting a tad overwhelmed by the sheer amount of ways
to accomplish something in Havok - and obviously I'd like to learn the most
current, optimal and effective method for each task.

For example do I use
hkSerializeUtil and load data into a resource and then grab what I need in an
appropriate container - or do I use hkLoader which I see is quite prevelant?

3.) I'd like to grab the vertices
and indices from the model I have loaded. Is the best route to use an
hkAnimationContainer-->hkMeshBindings-->hkMesh-->hkMeshSection-->hkVertex/IndexBuffer?

If so - what would be the demo example showing this?
Ideally I'd like to grab the Vertex and Index data, load it into the buffers
and then render the mesh in OpenGL. I've got the OpenGL stuff all ready to roll
- just need the mesh data.

imagem de havok_joe

Hi danothom:
Very glad to see you've figured out how to convert the resources. Here are some advices for your new questions.
1.) Please refer to
http://software.intel.com/en-us/articles/how-to-attach-file-insert-code-into-forum-post/

It will tell you how to do it.

2.) Serialization problem.Here is a good documentation for you to understand the relationship between hkSerializeUtil and hkLoader
Common Havok Components / Serialization / Basic Loading and Saving
Generally speaking, hkLoader is a thin wrapper on top of hkSerializeUtil. It is quite simple and easy to use but italways uses the "flattened loading style". Which mean all the resources you loaded, no matter you are going to useit or not, will keep in the memory until the game finished (hkLoader destroyed.)
On the other side, if you directly use hkSerializeUtil, you can use the "object style". So you can load theresource temporally, grab and keep the necessary objects which you need in the memory, and then release the wholeresource.
hkLoader is always used in Demos because we do not need to care too much about the memory usage.

3.) I guess you want to directly use graphic mesh data from Havok file and skin it by yourself.
Actually most of users export .x or some other format to save the graphics mesh data, and skin it by themselves.Havok does not provide a commercial skinning solution. For demo purposes, the simple, internal graphics frameworkwhich uses DirectX is good enough.
To use Havok animation, you can only export skeleton and animation data through .hkx file, and then use them todrive your model.
Demo/Demos/Animation/Api/Blending/Normal/NormalBlendingDemo
shows you how to load and blend the animations.
In NormalBlendingDemo.cpp, you can see we load rig data, some aniamtion data, and finally load "hkLowResSkin.hkx"which is havok skin mesh data. And then, we call

env->m_sceneConverter->convert( scene );
to convert the mesh data into graphics output buffers. If you want, you can dig into this function to figure outhow the vertex buffer created.
Also, the function
AnimationUtils::skinMesh(...);
is the main function how Havok skin the mesh.

Hope the reply can give you a brief idea about how to use Havok animations. Let me know if you have any questions.
By the way, you probably can have a look this ticket as well
http://software.intel.com/en-us/forums/showthread.php?t=104148

Joe Developer Support Engineer Havok www.havok.com

Hi Joe,Again - thanks for the prompt response. The skinning info is useful and I'll definately utilize that later on I'm sure. At the moment however I'm really struggling to just get my hands on the raw vertices contained in the animation mesh data. Essentially I just want to dump them into an array, load the data into a vertex buffer (possibly with the associated indices) and then test out the results by sending the buffers through to my OpenGL pipeline. If thats successful then I can move onto the skinning with weights side of things etc.A few things I really need to clear up before moving on:1.) In the documentation it recommends that I use the more modern binary Tagfiles (.hkt). Thats great and all, but from what I can see, most of the useful/necessary methods and functionality all pertain to the older Packfile format (.hkx). In fact - even you referred to them above in your post. Which format should I be using and exporting?2.) I had already gone through the NormalBlending example you mentioned. The problem is that the m_sceneConverter->convert method is useless to us as the binaries only contain the header implementation files (.h) and no the meaty code files (.cpp) which is what I really need to get to. Since most of the useful AnimationUtils methods utilize this method of converting the scene its of no use to anyone who wants to build a standalone compilation etc.3.) A lot of examples from the demos, and the older forum posts use the hkSceneData classes quite a bit - but I'm getting the picture that this is an older, less current implementation. Is it safe to say, especially for my purposes, that I should be going down the AnimationContainer->MeshBindings->Mesh->MeshSections->vertex/indexBuffers?I can't be the first person trying to get my hands on this information as its vital to any rendering pipeline. So why is it so hard to expose this information and how come there are no useful examples/demos on it anywhere (I have literally scoured the net looking for modern, usable examples). I can see a few similar posts but they all peter-out with the same replies and references by the Havok team etc.What would be really useful would be a small standalone Havok example build that showcasesa.) loading/serializing a .hkt assetb.) converting that loaded asset into an animationcontainer or something similar and exposing the useful primitive datac.) priming and getting that data ready for pushing out to a DirectX/OpenGL rendering pipeline etcKind Regards
Dan

imagem de havok_joe

Hi Dan:
1.) I think you've already read the DocCommon Havok Componets \ Serialization \ File Formats
Generally speaking, Tagfile is platform independent, smaller and slower to use. And packfile is platform dependent,bigger and much faster. We suggest user use tagfiles for development and use packfiles for release the game.Tagfiles are more malleable and much easier for programmer to analyze the problem.
Yes, we are using .hkx in Demo, since it is quite release ^_^
2.) I wonder which Version of Havok SDK you are using and how do you download it ? By default, the free Havok SDKyou can download fromhttp://www.havok.com/try-havok
should include our lever_1 source file for common, phylsics, animation and demo related code. You can findhkgSceneDataConverter.cpp in the folder:Demo\Graphics\Bridge\SceneData
and the mesh converter related file is in hkgSceneDataMeshConverter.cpp
3.) hkSceneData is a project. It contains the data descriptions that are extracted from the toolchain. It has thesupport for Havok internal rendering module. (such as renderer for Demos, Visual Debugger...) It is a differentconcept. I think you want to understand the structure of "The Havok Scene Data". I suggest you can refer toHavok Content Tools \ Integrating with the Havok Content Tools \ The Havok Scene Data Format
And meanwhile, try to use "View XML" filter in content tools to have a look the data(try not use "Prune Types"filter at first.)
In fact, both hkxScene and hkaAnimationContainer CAN export hkxMesh data and they are actually point to the sameone.And if your exported the mesh data into the animation container(which mean you did not use "Prune Types" todelete the Mesh Binding data), you definitely can find those data underAnimationContainer->MeshBindings->Mesh->MeshSections

Again, I still want to mentioned that havok mesh data(hkxMesh, hkxMaterial...) is mostly used for havok internalgraphic lib. If you want to use havok graphic (mesh,material) data to pass into your own render pipeline, you need to convert them by yourself.Surely, you have all necessary raw data after loading, and if you can step into the ConvertMesh function, it should be a very good reference.

Joe Developer Support Engineer Havok www.havok.com

Thank you havok_Joe, that's more useful then any of the answers previously given on the forums...

So I understand it's all a confidentiality thing, you can't tell us how to get the vertex format of the buffer... and you can't tell us how to get any of the vertex/indices/material infos from the hkaAnimationContainer. You just pointed us at a cpp file that is wrapped in the lib and we can't reach the code.

Yes, we want to convert the data by ourselves, but where do we find it and why should it be a quest to find it? I thought Havok Animation liscence approved of us implementing it within our game engines... so my guess would be it should have the slightest information(in either demo or docs) about where the data required for a graphics pipeline is to be found so we can collect it and arrange it convert it to conform to the graphics device.

Otherwise, forgive me I love you guys a lot and these are great quality products. The only thing being dissapointing is how we find numerous people asking for the same thing yet nothing is done about it when it's not breaking the confidentiality of information policy you have.

imagem de havok_joe

Hi aresthegod:
I do apologize. I downloaded the SDK from the website and there IS NO source code which I menetioned in the last reply. I should make sure by myself at first. Sorry again.
Havok do not have documentation about the detail format of the graphic datas which exported for skinmesh from content tools. BUT, there is the way you can easily figure it out by yourself.
For example, we provided 3dmax file for havok girls in the free SDK located in:
Demo\Demos\Resources\Animation\HavokGirlPlease openhkLowResSkin.max in 3DMax, it is the file used to output thehkLowResSkin.hkx which loaded in NormalBlendingDemo.cpp. It has all skined mesh and material data.
Then use havok content tools to export the file. By default, you can see there is "Write to Platform" filter after "Prune Types" filter. Remove all"Write to Platform" filter, instead, add "View XML" filter into the list. And the list will look like:

Convert To PngTransform SceneCreate SkeletonsCreate SkinsPreview ToolPrune TypesView XML
Then Run Configuration, the preview tool will be opened with a skined havok girl rendered , as well as a View XML window.And then, go to

hkRootLevelContainer -> Animation Container -> hkaAnimationContainer
It is quite simple because there is no animation data inside. The section
0x5e63c520 0x48415e60 0x48416580 0x5e63ce20
told you there are 4 skin mesh in the container. (hkaAnimationContainer.m_skins)Let's open one of them , for example "LowBody", and expand them all. The structure should be pretty clear now. There are hkxVertexBuffer (with hkxVertexDescription as the description of the vertex buffer you might need to look deeply inside), hkxIndexBuffer,hkxMaterial("yellow")...

And also you can find the detail info about those classes in the documentations.
OK, let's back to the code.In NormalBlendingDemo.cpp

{

	const char* skinFile = "Resources/Animation/HavokGirl/hkLowResSkin.hkx";


	hkStringBuf assetFile(skinFile); hkAssetManagementUtil::getFilePath(assetFile);

	hkRootLevelContainer* container = m_loader->load( assetFile.cString() );

	HK_ASSERT2(0x27343437, container != HK_NULL , "Could not load asset");
	hkxScene* scene = reinterpret_cast( container->findObjectByType( hkxSceneClass.getName() ));

	HK_ASSERT2(0x27343435, scene , "No scene loaded");
	hkaAnimationContainer* ac = reinterpret_cast( container->findObjectByType( hkaAnimationContainerClass.getName() ));

	HK_ASSERT2(0x27343435, ac && (ac->m_skins.getSize() > 0), "No skins loaded");
	m_skinBindings = ac->m_skins;
	// Make graphics output buffers for the skins

	env->m_sceneConverter->convert( scene );
}
"ac" will have all the data you saw in thehkaAnimationContainer section inXML above, and also, m_skinBindings contained all those 4 skin meshes.
By the way, with "Write to Platform" filter, you also can output a XML format file can be opened in any text editor. Also, you can try to do the same thing forhkIdle.max ... to see the structure with no skined mesh, but animation datas.

I hope it can help you figure out the info what you want and try to use it for your own rendering pipeline.
Cheers.

Joe Developer Support Engineer Havok www.havok.com

Hi Joe,

There is a lot of really useful
info in those last 2 posts so thanks for the detailed explanation. Ill
continue to work through it all tonight.

Again, I still want to mentioned
that havok mesh data(hkxMesh, hkxMaterial...) is mostly used for havok
internalgraphic lib. If you want to use havok graphic (mesh,material)
data to pass into your own render pipeline, you need to convert them by yourself.

Surely, you have all necessary
raw data after loading, and if you can step into the ConvertMesh function, it
should be a very good reference.

Again, I still want to mentioned
that havok mesh data(hkxMesh, hkxMaterial...) is mostly used for havok internalgraphic
lib. If you want to use havok graphic (mesh,material) data to pass into your
own render pipeline, you need to convert
them by yourself
.Surely, you have all necessary raw data after
loading, and if you can step into the ConvertMesh function, it should be a very
good reference.

a.) I fully realize that most of the used methods and
implementations are for the Internal Havok graphics system which is primarily
just for the demo application.

In regards to you need to convert them by yourself, I
fully intend to do that, and am really trying to. I intend to implement all of
the relevant methods and rendering systems on my own. I just need to expose the lowest
level of primitive data in the exported havok files i.e. the vertex/index data.

The fact of the matter is that this base data does exist in the export files when you export
a mesh from Maya/3dsMax etc, and obviously the Havok team grabbed this vital
info from the files themselves and converted it into buffer streams using their
own(apparently locked down+hidden) method in hkgSceneDataMeshConverter.cpp.

Thats all Im trying to do myself (along with Aresthegod
and a host of other intrepid tinkerers).

Ill try playing with exposing each individual vertex&index
from the Mesh/Scene vertexbuffer object tonight and see if I can get any
further.

I refuse to export a whole separate file just for the
vertex&index info in parallel to the havok files especially when I KNOW
that that data is contained in the havok exports. Otherwise I might as well go
back to using my own .OBJ exported files or god forbid trying to write my own
implementation of an Assimp COLLADA animation file importer. If the export tool exports meshes - then by definition those must contain a simple vertex+index structure.

b.) Unfortunately you are providing some conflicting info
versus the documentation:

Tagfiles

Tagfiles are a platform
independent binary file format and are the default and recommended format to store
serialized objects and data.

Packfiles

Serialized data may also be
stored in packfiles. Packfiles, the default file format in previous versions,

have largely been superseded by
tagfiles.

Questions regarding this then

1.) If
I choose to use tagfiles going forward, will they be compatible with any method
starting with hkx as I havent seen any hkt specific methods anywhere yet ?

2.) Or
should I just ignore tagfiles ( even though Havok recommends them as the more
reliable, compact and modern approach) and go with the deprecated Packfiles
(which seems crazy based on everything already mentioned) ?

The more I try and work through this stuff, the more I find
loads of this kind of ambiguity and desyncs. I believe its just a result of a
very old public demo pack which the public is forced to trawl through. So much
could be resolved quickly by just releasing a more compact, tagfile focussed, modern
version of the demo download Animation+Physics packages.

Again, not hating and I do appreciate the time you guys
are devoting to supporting us non-paying Havok enthusiasts. Just really trying
to figure out if there is enough information and current support/examples
within the Havok community to continue focussing in this area.

Again, thank you for the rich answers.

Yet, we are looking for something like:
how to access the verices:
ac->potato[0] and explanation... you have to itarate through potato collect all the vertices from potato[i]->verticesVector[j] or... get the pointer to the vector pool with potato[i]->vertexpoolpointer()

This kind of information should be in the docs but is not... thanks

imagem de havok_joe
Best Reply

Hi aresthegod:
Please refer to
Reference Manual / hkSceneData / Class List
All scene data related class definations are listed here. It is pretty straightfoward if you follow the XML structure I mentioned before.For example, if you want to get vertices from mesh, you can call

hkaAnimationContainer::m_skins[...]::m_mesh[...]::m_sectrions[...]::m_vertexBuffer.getNumVertices() and getVertexDataPtr()

when you callgetVertexDataPtr(), you need to pass in the vertex description element about what kind of info you want to get. You can get those elements by calling hkxVertexDescription::getElementDecl() and pass in the correct enum,such asHKX_DU_POSITION for positions andHKX_DU_NORMAL for normals...

There is no detail docs about how havok organize the vertex buffer data and index buffer data and so on. But you defintely can grab every piece of data which you want through the free SDK. You just need to test it out a bit.
By the way, you also can look at these functions:
void hkxMesh::appendGeometry(...)
Made a triangle base geometry for you. And then you can callhkGeometry::getVertex() to get each vertex.

void hkxMesh::collectVertexPositions(...)Gathering all vertex position info for you.

Hope it is helpful and feel free to ask questions.

Cheers

Joe Developer Support Engineer Havok www.havok.com
imagem de havok_joe

Hi Dan:
For the question about how to load tagfiles / packfiles ,hkSerializeUtil will handle it for you. No matter what kind of file you passed in,hkSerializeUtil will figure it out by analyzing the header of the file and load it for you.

For deciding which file you should use. I still suggest you can use tagfile at first because it causes less problems. It's platform independent, so you can load the same tagfile on both 32bit PC , 64bit PC , or XBox... It's not versioned, which mean the SDK will always patch it to the newest version on run time.

For example, you export both tagfile and packfile for the same asset in SDK 2011 and compare them. You will find tagfile will be smaller than packfile. But tagfile will take more loading time in the game than packfile do (because system need to patch the tagfile but packfile need not). Later, you decided to upgrade to SDK 2012 and try to load those two files you made before in SDK 2011. You will find you still can load them. The tagfile will take almost the same time to load as it loaded in the old SDK, but the packfile will take much longer loading time than before. (because the system will also patch the packfile into the new version in run time) Unless you useAssetCC2 to manually patch the packfile offline.

Hope it is clear enough.

Cheers

Joe Developer Support Engineer Havok www.havok.com

Hi Joe,Thanks for the previous post about how to dig out the vertex and index data. Its proving to be very useful and I'll have a more concerted dig at it over this weekend.I'm still a little confused about the file formats and the reason why the documentation says they recommend the newer tagfile format - but doesnt mention at all that you should swap back to older packfile format for release builds etc.Annoyng really as I just wanted a clearcut answer, but the reasoning still seems a little muddy and unclear to me. Either way - I'm using the tagfiles at the moment and will continue to do so.Thanks for all the help!

Faça login para deixar um comentário.