General Feedback / Plans for the Future (Overture?)

[Keyslam]

This repository has been in kind of a bad state, with poor (and sometimes wrong) documentation, a /dev and /experimental branch, each containing important features, and some questionable design choices.

This issue is to propose a set of changes and actions to take, to get Concord and it's repository back to a good state.

Logo

We have talked about a logo in the past, but never got around to doing it.
Perhaps a Concord plane?

Sumneko annotations

The Sumneko type system has become really good, and should be used in favor of the current LDoc annotations.
It would be absolutely fantastic if component data could be typed too, but we might need to wait for generics to be supported better.

Revamp syntax

In the Concord Discord I have often posted some proposals for alternative syntax, mainly for defining systems. The most recent proposal I feel is pretty good:

return ECS.defineSystem("My cool system", function(system, world, c)
    local query = world:createQuery({
        include = {c.position, c.velocity},
        exclude = {c.sleeping},
    }
   
    system:disable()

    system:on("update", function(dt)
        for e in world:query(query) do
            e.position.x = e.position.x + e.velocity.x * dt
            e.position.y = e.position.y + e.velocity.y * dt
        end
    end)

    system:onMatch(query, function(e)
        print("Entity matched", e)
    end)

    system:onUnmatch(query, function(e)
        print("Entity unmatched", e)
    end)
end)

In a nutshell:

• Queries replace filters, allowing for more complicated queries:
• • We could replace 'include' with 'read' and 'write' fields, for even more control over how components are accessed and modified
• • We could add 'any', for entities that match any of the listed components
• • We could add nested queries
• • Queries could be (more easily) created at runtime
• All components are passed in as c, so no more "components as strings" stuff
• system:on(event, cb) is to respond to emits. This way system events and system private functions dont get all mixed up.
• system:onMatch(query, cb) and system:onUnmatch(query, cb) replace pool:onEntityAdded and pool:onEntityRemoved, pushing it to systems instead allows for queries to be shared more easily.

Serialization

I've always felt like serialization is kind of shoehorned in. I also didn't use it when I needed serialization. It felt like a better approach to handle it in a saving and loading system respectively instead. I'd like to take a closer look at it.

Documentation

All documentation needs to be rewritten, period. The README should have some basic getting started info, the Wiki should be for indepth API info, and there should be practical examples in the repository.

Issues and Pull Requests

There are a bunch of open issues and pull requests, these should be looked at.

[Ulhar]

I think I agree with most of these proposals, although I do have a few thoughts:

• referring to components using their classes over strings seems a little odd (to me at least), especially since referring to components on entities using the strings (e.position etc) still works fine
• the de/serialization functions probably need to be optimized greatly (memory wise) if they are going to remain as library-provided functions, I think; I agree that their implementation may be better left to the user

(Also, a bit of a tangent, but I noticed the Discord isn't linked anywhere in the repo - should this be fixed?)

[pablomayobre]

Plans for Concord 4 will be moved to its own project probably called Overture.

This new project will reuse some of the ideas and discoveries made by Concord but will begin from a new canvas allowing it to break free from some of the constraints Concord has established. There have been some API proposals in Discord already and it's an entirely new approach.

Which means Concord will keep existing and being optimized, and making improvements to it's user experience and performance but trying to maintain the essence and some sort of compatibility or upgrade path.

Overture will take inspiration from Concord and Concord will take inspiration from Overture and hopefully in the future these two will meet in a single destination, which would hopefully be some sort of optimal approach to ECS or even a new paradigm entirely

For now I'll keep the issue around, we can see which of the bullet points on this issue can be addressed, open new issues and link to Overture once it's public and open source.

I'll be maintaining Concord and I have a few projects that I'll be slowly working towards, to keep improving the developer experience and usability of Concord

[oeN]

Sorry to bring up this old topic, is Overture still going to happen?

I've compared some ECS library for lua/Love2D and right now Concord is the one I like the most but I'd like to make some changes mostly how the systems are created and change a little the way Custom Pools #40 are created.

I was thinking in creating a fork but before doing that if a new library is on the horizon that maybe will solve my problems I'll wait (none of the mentioned "problems" is an hard block right now)

Thanks for the work on this project!

[Keyslam]

I happen to be getting back into Lua recently, also making a game with Concord, and I am noticing some pain points that really make me want to continue work on Overture.

Once I have an update on that I'll post it here.

For now: What are some of the changes you'd make? Perhaps I can keep them in mind while designing Overture.

[oeN]

These are good news 😄

for the changes I'd make (the first 2 point are already taken care of in the first post of this issue):

• systems: what's bothering me the most it the component as string, for my current project I've made a little utility to where I pass the queries as objects and it takes care of converting it to names
• queries: speaking of query, the utility I've mentioned take an object with the all/none keywords and covert it to the right "component"/"!component"
• systems ordering/dependency: I'm quite new to ECS but sometimes it seems I need to order the systems in a more complex way than it is now. I've been able to solve the issues I come across without a custom ordering so probably is my lack of experience with ECS that is talking here
• custom pools: simplify how to create "custom pools", before Concord I was using https://github.com/nidorx/ecs-lua and it had a nice Query.Filter(function(_e) return true; end) where you could easily define dynamic filtering in the query itself, I think under the hood the two library build the pools of entities in different way but having a more simpler way to inject dynamic filter in the current query would be great
• apsects: I've experimented a bit with Unity DOTS and one of the things I've liked about it were the Aspects I've ported the concept in my current project and in the utility that creates the query I can pass the BallAspect and the query will contain all the components that identify a ball:
  for example:

local BallAspect = require('aspects').BallAspect
-- BallAspect.components = { c.ball, c.position, c.collider, c.velocity }
-- the BallAspect will hold also a lot of methods, such as :move(), :render() etc. 
-- basically all the ball logic that is called by different systems
local BallRenderSystem = System({ query = my.Ecs.query({ all = { BallAspect } }) })
-- it will result in System({ query = { "ball", "position", "collider", "velocity" } })
...

[Keyslam]

I agree with all your points and foresee how to include them all in Overture elegantly :)
The only thing that I'm still unsure about is system ordering. I agree that there needs to be a better way to do that, but I haven't found any approach that I like thus far.

[pablomayobre]

custom pools: simplify how to create "custom pools", before Concord I was using https://github.com/nidorx/ecs-lua and it had a nice Query.Filter(function(_e) return true; end) where you could easily define dynamic filtering in the query itself, I think under the hood the two library build the pools of entities in different way but having a more simpler way to inject dynamic filter in the current query would be great

That's actually a fairly easy implementation:

local List = require("concord.list") -- Built-in List

-- Inherit from List
local FilteredList = setmetatable({}, { __index = List })
local meta = { __index = FilteredList }

-- Extend the :add method to filter based on layer
function FilteredList:add(e)
    -- Here we check that the entity matches our custom filter
    if self.filter(e) then
        List.add(self, e) -- if so add it
        return true
    end

    return false
end

--Export the FilteredList Constructor
return function(def)
    -- Create a new FilteredList
    local self = setmetatable(List(), meta)

    -- Save the filter in the FilteredList
    self.filter = def.filter

    -- And don't forget to return the new Pool
    return self
end

local LayerList = require("FilteredList")
local System = require("concord.system")

local RenderUI = System {
    uiLayer = {
        "render", "position"
        constructor = FilteredList,
        filter = function (e) e.position.x > 50 end,
    }
}

This is just the tip of the iceberg for Custom Pools and the idea of Custom Pools is that we can then create a repository with a bunch of them and share them across projects.

Custom Pools are not just limited to simple filters, it can be used alongside other data structures that provide alternative benefits than those of the List implementation used by default. There are not that many Custom Pools out in the wild but I plan to write some myself like the insertion-sort list, the spatial-hash pool, and the tree structure one.

systems: what's bothering me the most it the component as string, for my current project I've made a little utility to where I pass the queries as objects and it takes care of converting it to names
queries: speaking of query, the utility I've mentioned take an object with the all/none keywords and covert it to the right "component"/"!component"

Could you go into more detail on the issue you have with strings representing Components?

I think it has provided a lot of value to you since you were able to build all these nifty abstractions on top of Concord simple features.

The alternative syntax you are using does create a lot more garbage than simple strings with no visible benefit. I could be wrong but let me know if there are missing features here.

apsects: I've experimented a bit with Unity DOTS and one of the things I've liked about it were the Aspects I've ported the concept in my current project and in the utility that creates the query I can pass the BallAspect and the query will contain all the components that identify a ball:
for example:

I like the aspect idea and it feels like we could do it in Concord if we allowed you to nest tables like this:

local BallAspect = { "position", "collider", "velocity", "ball" }

local BallRenderSystem = System({ query = { BallAspect, "sprite", "size" } })

[oeN]

Custom Pools: I'm experimenting with Custom Pools so I don't have a full grasp on them yet, but I see that they're more than simple filters, and thanks for the example. It is pretty much what I've done but is good to see it confirmed 😄
What I meant with the comparison was just the simplicity of creating "dynamic" filters for the query, if I want to add another filter with the FilteredList constructor I'll have to change it and make it handle a table of filters (or create a new constructor) where on ecs-lua is just another function. I know I'm comparing two different features, so maybe is better if I get more experience with Concord before making any more assumptions.

Components as a string: you're right in saying that the simplicity of it made me able to make all those abstractions and I don't know if I can explain properly why I don't like using strings, probably is just an old habit of mine; I do gamedev as a hobby but I'm a professional developer, and I mostly work with python, typescript, and rust all languages with typing systems (some stronger than others).
A stupid example, if I change the name of a component if I use the "class/object" I can leverage the capabilities of my IDE to correctly replace all the instances of that "class/object" whereas with the strings I should find and replace all the occurrences but is possible that a word like "position" is used in some other place and not only for the component itself.

About the syntax, I think a table with { all = {...}, none = {...} } or { include = {...}, exclude = {...} } explains more than { "component", "!component" } even if the "!" is recognized enough as a negation indicator, is not immediately clear.
Having touched a lot of code bases started by other people I undoubtedly prefer a more verbose syntax that explains clearly what the code wants to do than one that is more beautiful, fancy, and compact but makes me think or, even, doubt what it wants to do.

Anyway, I understand that this could easily fall under the personal preference umbrella, probably if I spend another month or two using Concord I could ditch my syntax in favor of the string one, I'm also sure that years down the line in re-opening the code I will regret it but this it is programming itself and little can be done about it.

[Keyslam]

About components as strings:
I would be for using the objects/classes itself if those IDE tools actually worked propery. I've found that with Lua you can't really rename a symbol and have the change be appleid in the entire codebase. Combine that with the obligation of having to require the components (or letting the library pass in a container containing all components) and I feel like it's better to just keep it simple and use strings.

[pablomayobre]

Regarding component strings, the plan we had with @Keyslam was to try and make a nice plugin that integrates with the Lua LSP project so that we can provide autocomplete by building an internal "Component enum" that gets generated based on the Components you define. Still waaaay to go for that to be at a usable enough stage but it's generally the direction we are going. You can follow any updates on #71 but not much progress on that end just yet.

The reason I went with Components as strings was because then you don't need to require each component or a list of all Components everywhere. It also enabled the negation feature, which as you mentioned could be implemented with tables as well, but it's a bit more verbose and requires more tables. Could be doable but it could also be a plugin on top of Concord like the one you made, so not sure if worth changing.

The point @Keyslam made is one I mostly agree with. But I'm open for people to test all these assumptions we made ages ago and provide some good examples of things strings can't accomplish.

---

Custom Pools, definitely a lot of unexplored territory, they aren't paying off just yet. I have high hopes for them, we should start a repository of awesome-concord-pools and see what the community can come up with.

---

I'm really liking this discussion, please keep all this feedback comming, even if it doesn't make it into Concord it may push @Keyslam over the limit and make him finally build Overture lol

But it also helps me plan for the future of Concord and make a solid base for a stable project that can keep on growing.