Add Examples and Guides

[pablomayobre]

Add examples on how to use the different parts of Concord. Some of these already exist in the README but could see some improvement with in depth explanations, for example, when to use them, or how they relate to other features.

Introduction

• Getting Started (Rewrite from README)
• Tips & Tricks (Utils, Assemblages, Method Chaining)
• Making a game with Concord and LÖVE
• Tooling? Once we have tooling

Basics

These following articles assume previously reading Getting Started
They go into the same topics but going a bit deeper into each topic and covering all the available methods. They would then link to more advanced topics when needed.

• Components
  • Flag Components
  • Adding methods
  • Do's and Don't's
• Entities
  • Give vs Ensure
  • Remove Components
  • Assemblages
  • Removing Entities (go into detail on safe ways to do it)
  • Entity's World
• Systems
  • Filters (including negation)
  • Pools (how to create many)
  • Enable/Disable
  • Callbacks
  • Initializing
  • Storing data
  • System's World
• Worlds
  • Events (basics)
• Why ECS? Why Concord?

Intermediate

These articles are for more specific uses and functionality not used by many but useful for those that would like to benefit of the feature complete benefits of Concord.

• Cloning a Component
• Entity Keys and Resources
• Serialization
• Loading Namespaces
• Sorting pools
• Component:removed
• onAdded/onRemoved
• onEmit

Advanced

Finally these articles go into some of the internals and are tailored to advanced users or people writing libraries that go alongside Concord.

• Tweens
• UIs
• Cloning an Entity
• Reusing Entities (mark as empty, find and reuse them)
• Before and After Events
• Pool Flushing and Timing
• Custom pools
• Finding the Entity and World in populate functions

[pablomayobre]

Pool Flushing and Timing

This article will help you understand when the Pools' Filters get synchronized with the latest changes to Entities in the World, the relationship with Events, how it affects your game and how you can avoid the default behaviors.

Synchronous Operations

Concord has a synchronous part and an asynchronous part. For the most part you'll be using the synchronous part, so whenever you make a change, most of the things update immediately, for example, giving a component, removing a component, getting a reference to an entity, sorting a pool and many others are synchronous operations, they execute immediately when you perform them and you can see the effects in real time.

entity:give("position", 10, 20)

entity.position -- { x = 10, y = 20 }

entity.position.x = 30

entity.position -- { x = 30, y = 20 }

world:emit("update") -- All Systems with an update listener are called immediately 
                     -- in the order you added them to the World

As you can see most operations are immediate and you can see the results right after

Asynchronous Operations

There is one place however where we can't have immediate updates without introducing other sets of issues. That is Pools and Filters.

As discussed in the in-depth Systems article, all Pools have an associated Filter, this Filter specified all the requirements Entities should have to be in the Pool. The one checking that Pools have all the Entities that meet their requirements is the World.

Syncing the Pools

In order to make sure that the Pools have the Entities they need, the World goes through all the necessary Entities and checks them against the filters of every Pool, one by one. This is a very expensive operation which we call Flush.

You can actually trigger a Flush manually through World:flush() and you may sometimes want to do so. We will discuss the use cases later on.

When do we sync?

Entities change a lot, we often change multiple Components one after the other, add Entities in bulk, apply Assemblages that give multiple Components at once, delete a bunch of Entities that died simultaneously and so on.

The World gets notified of ALL these changes, then keeps track of all the Entities that have been changed, removed, or added to the World, and waits until it's ready to perform a Flush.

Flushes can be manually triggered as we previously discussed, but Concord also has a built-in mechanism for flushes to happen automatically.

When can the sync cause issues?

A sync performs additions and removals on Pools, so the biggest issue it can create is if you were iterating over one Pool, and suddenly the Entities inside were to shift around. This is not a problem with Concord, its List implementation or flushes, this also happens in regular Lua:

for k, v in ipairs(t) do
  table.insert(t, 1, k) -- This has undefined behavior and can cause an infinite loop
  print(k, v)
end

So it's not a good idea to add or remove inside to the current Pool we are iterating on. So we should not flush while we iterate. These iterations generally happen inside event listeners inside your Systems, since that's where Pools live, so we shouldn't flush while a System's event listener is executing.

World:emit()

Concord tries to keep the list of Entities consistent through an entire emit event and won't automatically flush until the Event is over. This is so that if System A and System B handle the same event their Pools are consistent with each other, this assumption may be broken if World:flush() is called manually, but that's an explicit choice by the user.

The best time for the automatic flush then is before the Event is forwarded to the Systems, it's a spot where no updates are being made, no iteration is going on, the World has complete control of the main thread.

However World:emit() can be called from inside World:emit() so Concord makes sure that by default a nested World:emit() doesn't cause an automatic flush.

If you wanted to emit an Event without causing a flush at all you can use World:emitNoFlush() as a replacement to World:emit(). This can be used to gain more control on when flushes happen.

love.update = function (dt)
  World:emit("update", dt) -- Flushes before forwarding the event to the Systems
end

love.draw = function ()
  World:emitNoFlush("draw") -- Doesn't flush
end

function RenderSystem:update ()
  self:getWorld():emit("prepareForRender") -- Doesn't flush

  self:getWorld():flush() -- This is a manual flush
end

World:query()

This function is always executed against the most up to date version of the Entity list, it is not affected by flushes and it includes the latest updates at the time of executing the function.

local myEntity = World:newEntity():give("position", 10, 20)

local list = World:query({"position"}) -- list contains myEntity even though there was no flush

If you want to perform additions/removals of Entities, keep in mind you shouldn't use the onMatch function since that happens a loop. You can still use the list alternative.

-- Don't do this
World:query({"position"}, function (e)
  World:newEntity()
  e:remove()
end)

-- Do this instead
local list = World:query({"position"})
for _, e in ipairs(list) do
  -- This is safe because list is not mutated
  World:newEntity()
  e:remove()
end

Forbidden Flush

There are a few places where flushing is forbidden the list is short though:

• World:onEntityAdded / World:onEntityRemoved
• Pool:onAdded / Pool:onRemoved

These callbacks happen during a flush, you can't flush during a flush since the buffers are in use.

If your code performs a flush but may be called from inside of one of these callbacks you can use World:canFlush() to check if flushing is allowed, otherwise an error will be thrown.

if World:canFlush() then
  World:flush()
end

[pablomayobre]

Why ECS? Why Concord?

There are various reasons why people choose ECS for their projects, there is a very large FAQ for ECS made by the author of FLECS, but we can go through some of the reasons here as a starting point:

It's a trend

Yeah it's a trend, most game engines are moving towards ECS as their pattern of choice, big names in the industry are using the pattern and more people are following this trend (Unity, Unreal Engine, Overwatch (video), etc).

This is not a good reason to use Concord but it does help that more and more documentation is being written about ECS and how to use it to organize your game code.

But this is also a downside, ECS is NOW being used by more game engines, but OOP has been the king for way longer and is more prevalent and used in this market.

Concord isn't here as part of the trend, and doesn't intend to follow what other ECS frameworks do. It instead presents it's own interpretation of ECS and makes a heavy focus on the ease of use and development.

It doesn't rely on inheritance

So to follow this discussion, a reason people choose ECS is so that they can get away from the ugly parts of OOP.

The ugly part being inheritance and all the problems it comes with. A possible solution would be mixins or decorators, but most people agree that the problems are still there.

So that's when they start to consider composition over inheritance. This is a very common pattern in OOP where you put objects inside objects instead of trying to define their behavior in term of inheritance. And ECS is very close to this pattern, remember how we put Components inside of our Entities? That's composition.

But in "Composition over inheritance" we would still define our behaviors in our objects, and we would still need to forward our events to each component. So that's when people start to consider ECS, behaviors existing outside of the object/entity helps organize the code more neatly.

Concord does make use of OOP and the metaprogramming functionality in Lua, to help implements ECS, and does still allow you to work with Components and Systems as if they were objects. So if you are escaping from the "bad parts" of OOP, your OOP knowledge will not be wasted while using Concord.

It's faster

Well this is the argument most game engines make to sell you ECS as the best architecture.

Remember our behaviors and data are stored separately? This helps keep our data small.
Remember how we don't make distinction between entities and instead rely on filtering? This allows us to pack our entities close together in memory so that we don't have to jump around so much when looping through them in our systems.

Unfortunately this doesn't apply to Concord because in Lua land we can't benefit from all of this, our representation of Components, Entities and Pools don't provide an advantage towards objects. Yeah Concord is fast for the majority of use cases, but this is not its selling point. You can probably write faster code without a library, and there are plenty alternatives that focus on speed.

Lua although fast, is not used for optimization purposes, and the same applies to Concord. Concord tries to be reliable, easy to use and feature complete, it is not an optimization.

We actually made an experiment making a super efficient ECS library, but it was awful to code with and one line of code could throw all the optimizations away. In our opinion, although it's possible to make a super-fast ECS library and speed does make a huge difference, it's not worth it when working in Lua where JIT optimization is hard

Reminds me of Functional Programming

If you have tried "The Elm Architecture" or "Redux" you may find the unidirectional data flow in ECS familiar.

If we think about our Entities and Components as our Model or State, and Systems as our Update or Reducers.
The only way to mutate our Model/State is through an Event processed by our Update/Reducer

This unidirectional data flow and the concept of using reducer functions as a way to update state is a proven concept and widely used in Functional Programming.

This mental model has worked really well for Front-end Web developers, and is a very good way to reason about how our systems update over time while keeping our state as our only source of truth and purely as data. If you are already familiar with these architecture you'll be able to pick up Concord and ECS in general, with relative ease.

In addition to that, if we combine ECS with immutable data structures for Components and Entities, then we wouldn't be too far away from "The Elm Architecture" and all its benefits like Time Traveling debugging.

Unfortunately immutable data structures aren't cheap and the Lua garbage collector is not tuned for them.
Concord doesn't use immutable data structures, but if you treat your Components as just data, it does allow you to recover state through serialization and deserialization.

It's more organized

If you find this mental model intuitive and start to reason only through this paradigm you'll find yourself writing code in a more structured way. There are very specific ways to do things in ECS and Concord will push you towards these patterns.

This is however at the expense of a higher learning curve with a big payoff in maintainability and reusability of your code. Concord keeps this risk low by allowing you to write code outside of Concord without extra hurdles.

In the long run we hope you'll find yourself at home using Concord, without needing these escape hatches, in order to accomplish this we try to ease this learning curve through guides, documentation and tooling that will hopefully make working with Concord a great experience.

[pablomayobre]

Concord

Concord is a feature complete library that allows you to use Entity - Component - System (ECS) model to write and organize your Lua code.

Concord puts a lot of effort on being easy to use and provide a great developer experience, while trying to be competitive in performance with alternative libraries.

Getting Started

As a first step you will need to download this repository and add the concord folder to your project.

In that folder you will find all the necessary files to use Concord.

Once added you can import Concord in your code

local Concord = require("path.to.concord")

Note

Please note that the file imported should be concord/init.lua, some Lua runtimes will allow you to require concord but others will need concord.init

World

The first thing you will define with Concord is a World.

The World will provide a single interaction point for all our Systems, and will hold all our Entities.

We will explain those concepts later, but first we need a World:

local world = Concord.World()

Components

You can think of Components as the data and state that make up all of your game.
Components themselves don't exist as part of our World, they instead are the pieces that build up Entities.

Components have a name, and some arbitrary shape that you can define.
For example if we wanted to store the Position of a given Entity, we would define a component called position with a shape that could be something like {x: number, y: number}

In Concord this would look like this:

Concord.Component("position", function (component, x, y)
  component.x = x
  component.y = y
end)

The first argument is the name, and the second is what we call the populate function.
Components start as empty tables and this populate function assigns the values for each of the Component properties.

Sometimes you don't need to store data on the Component since the Component itself will act as a flag, in those cases you don't need to provide a populate function:

Concord.Component("terrestrial")

Note

All Components are registered to Concord itself, not to a specific World.
This is why all the Components you define, regardless of the World you use them in, MUST have different names, two different Components can't share a name.

Entities

In ECS everything that exists as part of the World is an Entity. An Entity could be a Player, an Enemy, a Crate, a Bullet, etc.

Entities by themselves are empty packages with nothing on them, and they would all behave the same if we left them like that. To tell them appart and give unique data to each of them, we need to give them some Components.

In Concord this is very simple:

local Cat = world:newEntity()
--local Cat = Concord.Entity(world) -- Alternative syntax

Cat:give("position", 50, 50)
Cat:give("terrestrial")

Once you give a Component to an Entity, the Component will become a part of it:

print(Cat.position.x, Cat.position.y) -- Prints: 50, 50

if Cat.terrestrial then print("Walks!") end -- Prints: Walks! 

Note

Any Component can be given to any Entity ONCE. You can't give a Component multiple times, giving the same Component one more time will override the previous instance of the Component.

Filters

The power of Concord and ECS in general comes from grouping a bunch of Entities based on rules. These rules acts as a Filter where we can ask the World for all Entities that have all the Components we required.

For example we could ask the World for a list of all the Entities that are terrestrial and have a position, and that are not a machine:

local animals = world:query({ "terrestrial", "position", "!machine" }) -- The ! in front of machine indicates negation

for i, entity in ipairs(animals) do
  print(entity.position.x, entity.position.y)
end

Side note on Filter rules

If we had another Entity like this one:

local Parrot = world:newEntity()

Parrot:give("position", 100, 100)

It would not be part of our list above, since the filter required:

terrestrial and position and not machine

And although Parrot has position and doesn't have machine, it lacks the terrestrial Component.

This is great but the World:query operation is rather expensive, so Concord instead provides Pools.
A Pool has an associated Filter and will be updated immediately when Entities are added or modified in the World.

In order to use pools however, we need to speak about Systems

Systems

In ECS, Components and Entities generally don't have behavior themselves, and instead rely on Systems that define how a group of Entities should behave.

For example if we want all terrestrial animals to move 5 pixels to the right every time we receive a move event we would first create a System like:

local Movement = Concord.System("Movement", {animals = { "terrestrial", "position" }})

This system is called Movement and has a Pool called animals that contains all the Entities with terrestrial and position Components.

We now want to receive all move events. To do that we define a callback in our System like so:

function Movement:move ()
  for i, entity in ipairs(self.animals) do
    entity.position.x = entity.position.x + 5
  end
end

Whenever we receive a move event in our World, this callback will execute, and the Movement System will iterate through the Pool of animals changing their position Components to move them 5 pixels to the right.

Events

In the previous section we defined a move callback, but we don't yet know how to execute that callback.

To do that first the System needs to be added to our World

world:addSystem(Movement)

Note

You can add each System once to a given World.
A System can be added to multiple Worlds (they will be different/separate instances of the System).

Finally we need to fire the move event, to do that, we don't call the System directly, instead we emit the event to the World

world:emit("move")

Once emitted, all the Systems that define a move callback will be called in the order they were added to the World.

You can also provide extra arguments to these events, for example if we had:

function Movement:update(dt)
  -- Do something here
end

You could provide the dt argument like so:

world:emit("update", 5) --dt = 5

Summary

We defined a few big terms and how they are used in Concord, but to summarize them:

• World: This is where Entities and Systems live, it receives external and internal Events.
• Components: Data and state, each with a name and a shape. Each Component describes a specific aspect of an Entity.
• Entities: Everything that exists in the World. They are bundles of Components.
• Filters: A way to group Entities based on the Components they have or don't have.
• Systems: Behaviors that apply to groups of Entities and react to Events received by the World.
• Events: Signals that tell the World something happened or needs to happen, the World delegates the Events to all the Services that can respond to it.

These are Concord definitions, and although they may apply to other ECS libraries or frameworks, some differences may exist.

With this we have covered the basics. With the API docs you would be ready to start working with Concord, but if you want to dig into more advanced concepts read some of our Guides instead.

[pablomayobre]

Custom Pools

This article details a escape hatch when the List used by Concord Pools is not enough for your System's need.

First we need to understand why this escape hatch is needed so we discuss some upsides and downsides of Lists, we then discuss the solution based on onAdded/onRemoved that Pools provide, and finally look at the escape hatch and its benefits.

Lists upsides and downsides

Pools in Concord by default use a List implementation that provides fast inserts, fast removals, fast search and fast iteration.

These are all useful for Pools:

• Pools need to check if an Entity is part of the List or not.
• Pools need to add and remove Entities quickly
• The user needs to be able to iterate through the Pool quickly.

However these Lists have a few downsides:

• They consume some extra memory because they need to keep a value -> key hashmap
• They can't guarantee the order of their members
• Sorting is expensive, since the hashmap needs to be regenerated

List Synchronization/Duplication

Sometimes you need to add your Entities to a different data structure like a Spatial Hash for Physics for example, when working with such a structure, you may end up with code that looks like this:

local Physics = Concord.System("Physics", {
  boxes = {"position", "size"} 
})

function Physics:init ()
  -- Create a new spatial hash
  self.spatialhash = spatialhash()

  -- Whenever there is a new box added to the boxes list
  -- Add the box to the spatial hash
  function self.boxes:onAdded (entity)
    self.spatialhash:add(entity)
  end

  -- Do the same when a box is removed
  function self.boxes:onRemoved (entity)
    self.spatialhash:remove(entity)
  end
end

Another fairly common example would be if you wanted a Sorted List for Rendering, and don't want the extra cost of sorting the List on each Event and instead would prefer to sort on insertion.

As you can see we have duplication of data, our Spatial Hash has all the same Entities that our boxes List. And now the code to sync the List with the Spatial Hash lives inside of the Physics System, which makes it hard to share or reuse.

So Concord offers an escape hatch out of this situation. If you only need the filtering part of a Pool and not the List part, you can bring your own data structure.

Bring your own data structure

This feature is called Custom Pools

The way to define a Custom Pool is fairly simple. You need to create a function that returns an object with 4 methods: has, add, remove and clear

We could redo our example above by creating a Custom Pool wrapper for our Spatial Hash (this is not always needed but may be helpful if the method names or their parameters don't match)

function SpatialHashPool ()
  local storage = {hash = spatialhash()}

  function storage:clear()
    self.hash = spatialhash()
  end

  function storage:add(entity)
    self.hash:add(entity)
  end

  function storage:remove(entity)
    self.hash:remove(entity)
  end

  function storage:has(entity)
    if self.hash:search(entity) > 0 then
      return true
    else
      return false
    end
  end

  return storage
end

Then we can use this in any System like so

local Physics = Concord.System("Physics", {
  boxes = {"position", "size", constructor = SpatialHashPool}
})

Please note that the code now lives outside of the Physics System and can now be reused by different Systems or even different projects. This abstraction helps us share and reuse code.

Additional Methods

The Storage we defined only exposes the methods needed by Concord which means that in order to iterate or perform other iterations you would need to use self.boxes.hash to access the Spatial Hash directly. However Concord does not enforce any limit on the methods exposed by your Pool other than those 4 required methods, so you could expose more utility functions that you need to access in your System's code.

For example we could add

function storage:query(x, y, w, h)
  return self.hash:query(x, y, w, h)
end

And use it in our System like so:

function Physics:update(dt)
  local visibleEntities = self.boxes:query(0, 0, 800, 600)
  
  for _, entity in visibleEntities do
    -- Do something
  end
end

Pool definition

One advantage Custom Pools have is that when they are instantiated you get the complete Filter definition for the Pool as argument to the constructor function:

function SpatialHashPool (def)
  print(def) -- {"position", "size}

  -- ...
end

This could be useful if you need some additional parameters to generate your storage for example

function SpatialHashPool (def)
  local storage = { hash = spatialhash(def.width, def.height) }
  
  -- ...

  return storage
end

-------

local Physics = Concord.System("Physics", {
  boxes = {"position", "size", constructor = SpatialHashPool, width = 800, height = 600}
})

[spritefullake]

Thank you for writing this. This helped me understand how to use Concord and ECS much better than other resources. Wish your docs could be the main docs shown on the github for Concord.

Thanks so much. :)