Non-destructive export / dual-state

[met4citizen]

As I understand it, MPFB currently uses a single character state for both authoring (character design) and delivery (export to formats such as glTF/GLB).

The practical issue with this approach is that exporting requires destructive and irreversible transformations (e.g. baking modifiers, collapsing targets, simplifying materials, removing Blender-only constructs). Once these steps are applied, the character is no longer in a clean design state.

In practice this makes iteration costly: First, the user designs a character in MPFB. To preview the character in a target environment (e.g. game engine, WebGL), the user must now destructively convert the model into an exportable form. The user then typically applies additional delivery-specific customization (naming conventions, rig changes, final material adjustments, etc.). Only after this can the character be exported.

If, after testing in the target environment, the user wants to adjust the design (e.g. proportions, facial features), they must return to the first step and repeat both the destructive conversion and the delivery customization steps. With multiple iterations, this becomes repetitive and error-prone.

A possible solution might be to separate the character into two linked representations: A design version (Full MPFB rig; All modifiers, drivers, and parametric controls; Maximum editability) and an export version (Baked geometry; Mesh-specific shape keys; Simplified glTF-compatible materials; No Blender-only constructs). This separation is, I think, common in production pipelines, where a non-destructive source asset generates one or more delivery assets.

In this setup the design version remains authoritative. It could always be the same regardless of the export requirements, which could potentially simplify things a lot. The export version, on the other hand, is generated only once based on user-selected specs (topology, rig type, material system, triangle count, etc.) and automatically updated whenever the design version changes and/or is refit. Most delivery-specific customization (naming, extra bones, custom materials, etc) could persist on the export version across iterations instead of having to be redone each time.

From a user perspective, the export version would simply be another object or collection that can be shown/hidden. Switching between them would be easy. And, when the export version looks correct, it can be exported directly.

I want to emphasize that this is meant as a conceptual idea, not a concrete implementation proposal. Most likely this has been proposed/discussed at some point already. Also note that I am new to MPFB and do not have a deep understanding of its internal architecture. There may be design constraints that make this undesirable or infeasible to implement. However, from a user and pipeline perspective, a non-destructive export and dual-state workflow could significantly reduce friction when targeting glTF/GLB and similar runtime formats.

[joepal1976]

I think possibly some of this could be achieved by using a body proxy: https://static.makehumancommunity.org/mpfb/docs/assets/concept_proxymeshes.html

This proxy can have different materials and additional shape keys not related to the modeling. I tested now, and adding shape keys to the proxy does not prevent continued modelling and refitting.

It might not lead the whole way. For example, I guess one would want to exclude the base mesh in exports.

[joepal1976]

Thinking more about it, I think the following are a bare minimum of things which should make the problem solvable without adding a whole new concept into the core logic:

• A 1:1 body proxy which replicates the basemesh without the helpers. Possibly it should include the tongue and teeth helpers as a way to represent visemes (although one would have to paint new materials for these)
• Switches/settings which prevents automatic propagation of some blenderisms from the basemesh to the proxy (for example automatic delete groups with masks and skin material changes)
• A convenience operator which automatically assembles an exportable version where the base mesh is excluded.

Anyway, maybe I could ask you to see how far you get with one of the existing proxies and see where it starts to hurt? At least with FBX I think it should be possible to only export the selected objects, so selecting the character hierarchy and deselecting the base mesh should be a manual way of achieving what the convenience operator would do.

[met4citizen]

Thanks, Joel. I will do as you suggested. When I find some free time, I will try to find the best possible way to build a TalkingHead-compatible GLB file using the current MPFB version, proxies, customized Mixamo rig, ARKit/viseme targets, and so on. I will document the process and post a link to it here for review and further discussion. Might take a week or two.

[met4citizen]

As promised, here is the link to the description of my current MPFB workflow. You can also test my first attempt using the TalkingHead test app and choosing "MPFB" avatar.

I tried at least ten different approaches. The process as described now is a compromise between achieving the best results and minimizing the required effort. I still have a lot of work to do, but any questions, comments, or insights related to this would be appreciated.

I had to create a custom rig because the bone rolls in the existing MPFB Mixamo rig(s) were not exactly aligned with Mixamo's "Y-Bot" reference skeleton. This is primarily an issue for our pipeline, as Three.js applies animation rotations in the bone's local coordinate system exactly as stored in the glTF/GLB, and we do not perform any retargeting or axis correction in our class.

I also tried using a proxy, but the runtime quality with the altered topology and reduced triangle count was not as good as with the base mesh. Moreover, using a proxy did not actually save any effort. The required export tasks were practically the same with or without one.

Many of the export steps described are specific to our TalkingHead project, so I will need to handle those myself. I can probably consolidate most of them into a single script or add-on. That said, I can certainly see some of the more general parts being implemented in MPFB as well.

Regarding the original topic of this issue, if the export process could be made (more or less) one-click, similar to generating a reduced model for Mixamo, there would be little need to maintain two linked states. That is, a separate copy could be created to protect the design state, but the two representations would not need to be linked.

[joepal1976]

I've created a new branch at https://github.com/makehumancommunity/mpfb2/tree/feature_export_copy where I intend to add logic for working with non-destructive export copies. I haven't got very far, atm it is mostly place holders.

[joepal1976]

@met4citizen : When propagating shape keys to child meshes, would you say it'd make sense to limit it to only body parts and hair? I mean, visemes would not normally have an effect on shoes.

Downside would be fringe cases like a mask not getting shape keys, upside would be performance through not unneccessarily propagating lots of shape keys.

[joepal1976]

With the recent commit, at least some of the steps are now implemented. The "export copy" operations panel can be found under "operations".

There's a bunch of checkboxes left to implement, but at least it is now capable of creating an export copy with baked shapekeys, removed helpers and with meta visemes loaded.

[met4citizen]

When propagating shape keys to child meshes, would you say it'd make sense to limit it to only body parts and hair?

In my temporary add-on script (see the bake_target function), I make the user to choose the relevant meshes. However, I typically select all child meshes, as the limit_size parameter alone should rule out distant elements such as shoes.

Previously, in my script, I used a limit value of 1e-4 and ended up with too many shape keys on meshes where they did not make much sense. I then increased the limit to 1e-3. In other words, if none of the vertices in a given mesh move more than one millimeter when the shape key value is set to 1.0, the shape key is not created for that mesh. For many web app use cases, this threshold could likely be increased further, as a one millimeter change is generally too small to notice. As you correctly point out, fewer shape keys also result in better performance.

The "export copy" operations panel can be found under "operations".

I tried this, and the implemented parts seemed to work fine. I think this approach will be very helpful.

As a small side note, while browsing through the source code, I noticed that your META_VISEMES list actually includes the names of the Microsoft visemes.