Replace JSON with ProtoBuf

[nevyn]

The Flatbuffer refactor was a high priority ticket to get rid of JSON, our current biggest performance bottleneck, and replace it with something much more efficient both in terms of memory and CPU. (In short, our terrible performance on Meta Quest is attributable to JSON decoding, which due to JSON's dynamic nature is really hard to fix).

The Flatbuffer approach might still be possible, but it doesn't feel like the correct approach anymore. Alloverse is designed to have a flexible set of interactions and components, but flatbuffers are very heavily tailored to doing schemas at compile time. The fallback is flexbuffers, but then you lose the power of schemas.

It might be viable to use ProtoBuf instead. It has schemas, but also allows dynamic fields if configured correctly. It also has arena allocation, which might be an acceptable replacement

Prerequisites

• Do an experiment with a test project
  • ... that uses both schema-defined and flexibly-defined components.
  • It should use generic keypath based computation, to make sure that the animation system will survive the transition.
  • See if Arena allocated data can be memcpy'd for the "double buffering" optimization
  • Try out a Lua-compatible API with as little copying as possible (dynamic accessors?)
• Try to figure out if the whole refactor can be made on the main branch with ifdefs, so we don't get yet another months-long branch full of fixes that never get merged
• Stub out implementations in the protobuf fork
• Cherrypick to master any fixes in flatbuffer branch that isn't strictly flatbuffer related, since that branch will never be merged

Main work

• Make server run okay
  • Spawn entity
  • Change components
  • Remove entity
  • Property animations
  • List agents
• Simulation subsystem
• Animation subsystem
• Lua API
• Statediffs
• Bindiff for states over network (new delta algorhtim) with statehistory

Nice to haves

Funding