MatterServer cannot be started more than once

[bkoblenz]

I am interested in the case where there is a (TBD) bug that requires me to shutdown and restart the MatterServer. I have certainly had plenty during my development. It seems that after the server.stop() and regeneration of the MatterServer object, there is a failure regenerating the chip stack. I have been able to verify this by editing scripts/example.py and putting a loop around the code and hitting ^C to cause the termination of the first iteration, stopping the server and then trying to start the second iteration. The log output shows close to where things run amiss (starting during the shutdown of the first iteration):

2024-12-22 13:00:14 Master-198 aiorun[25935] INFO Entering shutdown phase.
2024-12-22 13:00:14 Master-198 aiorun[25935] INFO Executing provided shutdown_callback.
2024-12-22 13:00:14 Master-198 matter_server.server.server[25935] INFO Stopping the Matter Server...
2024-12-22 13:00:14 Master-198 matter_server.server.stack[25935] INFO Shutting down the Matter stack...
2024-12-22 13:00:14 Master-198 chip.native.CTL[25935] Level 39 Shutting down the stack...
2024-12-22 13:00:14 Master-198 aiorun[25935] INFO Waiting for executor shutdown.
2024-12-22 13:00:14 Master-198 aiorun[25935] INFO Shutting down async generators
2024-12-22 13:00:14 Master-198 aiorun[25935] INFO Closing the loop.
2024-12-22 13:00:14 Master-198 aiorun[25935] INFO Leaving. Bye!
iteration 1
2024-12-22 13:00:14 Master-198 matter_server.server.stack[25935] INFO Initializing CHIP/Matter Controller Stack...
2024-12-22 13:00:14 Master-198 chip.native.DL[25935] Level 39 ChipLinuxStorage::Init: Attempt to re-initialize with KVS config file: /tmp/chip_kvs
2024-12-22 13:00:14 Master-198 chip.native.DL[25935] Level 39 ChipLinuxStorage::Init: Attempt to re-initialize with KVS config file: /data/chip_factory.ini
2024-12-22 13:00:14 Master-198 chip.native.DL[25935] Level 39 ChipLinuxStorage::Init: Attempt to re-initialize with KVS config file: /data/chip_config.ini
2024-12-22 13:00:14 Master-198 chip.native.DL[25935] Level 39 ChipLinuxStorage::Init: Attempt to re-initialize with KVS config file: /data/chip_counters.ini
2024-12-22 13:00:14 Master-198 chip.CertificateAuthority[25935] INFO Loading certificate authorities from storage...
Traceback (most recent call last):
File "/home/pi/Irricloud-substation-development/python-matter-server/scripts/example.py", line 78, in
server = MatterServer(
^^^^^^^^^^^^^
File "/home/pi/Irricloud-substation-development/python-matter-server/matter_server/server/server.py", line 140, in init
self.stack = MatterStack(self, bluetooth_adapter_id, enable_server_interactions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pi/Irricloud-substation-development/python-matter-server/matter_server/server/stack.py", line 130, in init
self.certificate_authority_manager.LoadAuthoritiesFromStorage()
File "/home/pi/.venv/lib/python3.12/site-packages/chip/CertificateAuthority.py", line 253, in LoadAuthoritiesFromStorage
caList = self._persistentStorage.GetReplKey('caList')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'GetReplKey'

[marcelveldt]

Please provide (a lot) more info because this is way too little for us to understand your goals and issue.
I have never ever seen this issue before so it must be some edge case you trigger.

What dev environment ?
Why are you trying to loop the code around ?
What are you trying to accomplish ?
etc

[bkoblenz]

This is definitely a dev environment setup and is not impacting me. I just wanted to understand failure cases with the server or client and making sure I could restart them in such a case. fwiw, the error is in the chip library and, in my case, I do not have that built directly but installed via snap. That may or may not be part of the issue and I could retry (in a while) with a clean build. But, I tried to give you an easy way to replicate the problem in your environment. In particular, I edited the scripts/example.py which starts the server and client, and when interrupted (ie, ^C) exits. I put that body (shown below) in a stupid loop so after the ^C, it would reestablish the server and client. It was during that reestablishment, that the error is shown. for i in range(2): print('iteration',i) # Init server server = MatterServer( args.storage_path, DEFAULT_VENDOR_ID, DEFAULT_FABRIC_ID, ... async def handle_stop(loop: asyncio.AbstractEventLoop): """Handle server stop.""" await server.stop() # run the server run(run_matter(), shutdown_callback=handle_stop) Does that make enough sense for you to reproduce?

…

On Tue, Dec 31, 2024 at 12:45 AM Marcel van der Veldt ***@***.***> wrote: Please provide (a lot) more info because this is way too little for us to understand your goals and issue. I have never ever seen this issue before so it must be some edge case you trigger. What dev environment ? Why are you trying to loop the code around ? What are you trying to accomplish ? etc — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[bkoblenz]

I definitely think this is a bug. Here is another manifestation.

The high level view is:

    while True:
       self.logger.info('matter run looptop')
       self.matter_server = MatterServer(...)
       loop = asyncio.new_event_loop()
       try:
           asyncio.set_event_loop(loop)
           loop.set_debug(True)
           self.logger.info('MatterController run server and client')
           loop.run_until_complete(self.start_server_and_client())
           self.logger.info('MatterController stop server')
           loop.run_until_complete(self.stop_server())
           self.logger.info('MatterController run finish')
       finally:
           self.logger.info('MatterController finally')
           asyncio.set_event_loop(None)
           loop.close()
           self.client = None

       time.sleep(30)

where start_server_and client looks runs for an hour then breaks out of its loop to force the reinstantiation of MatterServer above

      await self.matter_server.start()
      url = DEFAULT_URL
      async with aiohttp.ClientSession() as session:
          async with MatterClient(url, session) as client:
              self.client = client
              asyncio.create_task(client.start_listening())
              await asyncio.sleep(10)
              while True:
                  if an hour has gone by since start:
                      break # completes routine so new MatterServer is created
                  process client commands

Here is the log showing how the shutdown (and attempted restart) proceeded:

2025-02-02 12:21:22,447 - matter - INFO - MatterController stop server
2025-02-02 12:21:22,449 - matter_server.server.server - INFO - Stopping the Matter Server...
2025-02-02 12:21:22,512 - matter_server.server.stack - INFO - Shutting down the Matter stack...
2025-02-02 12:21:22,515 - chip.native.CTL - Level 39 - Shutting down the stack...
2025-02-02 12:21:22,534 - chip.native.DIS - Level 39 - Failed to advertise records: src/lib/dnssd/minimal_mdns/Server.cpp:344: CHIP Error 0x00000046: No endpoint was available to send the message
2025-02-02 12:21:22,537 - chip.native.DIS - Level 39 - Failed to advertise records: src/lib/dnssd/minimal_mdns/Server.cpp:344: CHIP Error 0x00000046: No endpoint was available to send the message
2025-02-02 12:21:22,573 - matter - INFO - MatterController run finish
2025-02-02 12:21:22,574 - matter - INFO - MatterController finally
2025-02-02 12:21:52,579 - matter - INFO - matter run looptop
2025-02-02 12:21:52,584 - matter_server.server.stack - INFO - Initializing CHIP/Matter Controller Stack...
2025-02-02 12:21:52,586 - chip.native.DL - Level 39 - ChipLinuxStorage::Init: Attempt to re-initialize with KVS config file: /tmp/chip_kvs
2025-02-02 12:21:52,587 - chip.native.DL - Level 39 - ChipLinuxStorage::Init: Attempt to re-initialize with KVS config file: /data/chip_factory.ini
2025-02-02 12:21:52,587 - chip.native.DL - Level 39 - ChipLinuxStorage::Init: Attempt to re-initialize with KVS config file: /data/chip_config.ini
2025-02-02 12:21:52,588 - chip.native.DL - Level 39 - ChipLinuxStorage::Init: Attempt to re-initialize with KVS config file: /data/chip_counters.ini
2025-02-02 12:21:52,599 - chip.native.DL - Level 39 - Failed to reset WiFi statistic counts
2025-02-02 12:21:52,600 - chip.CertificateAuthority - INFO - Loading certificate authorities from storage...
2025-02-02 12:21:52,601 - matter - ERROR - no MatterServer
Traceback (most recent call last):
File "/home/pi/Irricloud-substation-production/plugins/matter.py", line 289, in run
self.matter_server = MatterServer(
^^^^^^^^^^^^^
File "/home/pi/Irricloud-substation-production/python-matter-server/matter_server/server/server.py", line 140, in init
self.stack = MatterStack(self, bluetooth_adapter_id, enable_server_interactions)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pi/Irricloud-substation-production/python-matter-server/matter_server/server/stack.py", line 130, in init
self.certificate_authority_manager.LoadAuthoritiesFromStorage()
File "/home/pi/.venv/lib/python3.11/site-packages/chip/CertificateAuthority.py", line 253, in LoadAuthoritiesFromStorage
caList = self._persistentStorage.GetReplKey('caList')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'GetReplKey'

[bkoblenz]

fwiw, I also tried to stop the MatterServer and then restart it (without creating a new instance of MatterServer), but that too ran into reinitialization problems in the chip library.

[agners]

The Matter SDK Python bindings have quite some globals which do not allow to initialize the Matter stack twice.

Our MatterServer class initializes and holds the Matter stack, which can only be initialized once, so that limits things currently.

Technically a single MatterStack can handle multiple fabrics. So if your goal is to have multiple Matter fabrics, that should be possible. It probably needs some changes though. Specifically weprobably should make main() to initialize a single MatterStack, and pass that to the MatterServer. This would allow multiple MatterServer instances.

[bkoblenz]

This makes total sense and would be great.

…

On Tue, Feb 18, 2025 at 3:25 PM Stefan Agner ***@***.***> wrote: The Matter SDK Python bindings have quite some globals which do not allow to initialize the Matter stack twice. Our MatterServer class initializes and holds the Matter stack, which can only be initialized once, so that limits things currently. Technically a single MatterStack can handle multiple fabrics. So if your goal is to have multiple Matter fabrics, that is possible. We probably should make main() to initialize a single MatterStack, and pass that to the MatterServer. This would allow multiple MatterServer instances. — Reply to this email directly, view it on GitHub <#1009 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADHHYLIEVTRJO4M5YCZZPXT2QO6M5AVCNFSM6AAAAABUB72WRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRXGE2DQNJRGI> . You are receiving this because you authored the thread.Message ID: ***@***.*** com> [image: agners]*agners* left a comment (matter-js/python-matter-server#1009) <#1009 (comment)> The Matter SDK Python bindings have quite some globals which do not allow to initialize the Matter stack twice. Our MatterServer class initializes and holds the Matter stack, which can only be initialized once, so that limits things currently. Technically a single MatterStack can handle multiple fabrics. So if your goal is to have multiple Matter fabrics, that is possible. We probably should make main() to initialize a single MatterStack, and pass that to the MatterServer. This would allow multiple MatterServer instances. — Reply to this email directly, view it on GitHub <#1009 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADHHYLIEVTRJO4M5YCZZPXT2QO6M5AVCNFSM6AAAAABUB72WRGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRXGE2DQNJRGI> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[marcelveldt]

For now this is not a priority but PR's accepted.

[Apollon77]

Lets revisit this when the server is migrated to matter.js.