intermittent websockets server failure

[caver456]

sometimes the websockets server just stops sending updates. Not sure if this is because it has stopped listening, or because the server has stopped sending. Observed while walking a pairing through its statuses. Most recent code changes were to the handler for setting pairing status to 'DONE'. Local database and localhost database seem to be getting appropriate edits - it seems to just be a dead websocket server.

Noticed this sequence in the websocket log around that time, but only for one of the occurances of dropped connections:

09:09:42 : incoming message:
{"msg": "{\"teamsView\": [[\"101\", \"AC\", \"DEBRIEFING\", \"Ground Type 1\", \"--\"]], \"assignmentsView\": [[\"AA\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AB\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AC\", \"101\", \"DEBRIEFING\", \"Ground Type 1\"], [\"AD\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AE\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AF\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AG\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AH\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"]], \"teamsCount\": \"1 Assigned / 0 Unassigned\", \"assignmentsCount\": \"1 Assigned / 7 Unassigned\"}"}
message repeated to 2 registered listener(s)
unregister: <websockets.server.WebSocketServerProtocol object at 0x04513EB0>
register: <websockets.server.WebSocketServerProtocol object at 0x04513B70>
register: <websockets.server.WebSocketServerProtocol object at 0x04513910>
09:09:43 : incoming message:
{"msg": "{\"teamsView\": [[\"101\", \"--\", \"UNASSIGNED\", \"Ground Type 1\", \"AC\"]], \"assignmentsView\": [[\"AA\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AB\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AD\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AE\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AF\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AG\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AH\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AC\", \"101\", \"COMPLETED\", \"Ground Type 1\"]], \"teamsCount\": \"0 Assigned / 1 Unassigned\", \"assignmentsCount\": \"1 Assigned / 7 Unassigned\"}"}
09:09:43 : incoming message:
{"msg": "{\"teamsView\": [[\"101\", \"--\", \"UNASSIGNED\", \"Ground Type 1\", \"AC\"]], \"assignmentsView\": [[\"AA\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AB\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AD\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AE\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AF\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AG\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AH\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AC\", \"101\", \"COMPLETED\", \"Ground Type 1\"]], \"teamsCount\": \"0 Assigned / 1 Unassigned\", \"assignmentsCount\": \"1 Assigned / 7 Unassigned\"}"}
message repeated to 3 registered listener(s)
unregister: <websockets.server.WebSocketServerProtocol object at 0x04513B70>
unregister: <websockets.server.WebSocketServerProtocol object at 0x04513910>
Error in connection handler
Traceback (most recent call last):
  File "C:\Users\caver\AppData\Roaming\Python\Python37\site-packages\websockets\server.py", line 191, in handler
    await self.ws_handler(self, path)
  File ".\trackerWSServer.py", line 38, in trackerHandler
    await repeat(message)
  File ".\trackerWSServer.py", line 22, in repeat
    for user in USERS:
RuntimeError: Set changed size during iteration
register: <websockets.server.WebSocketServerProtocol object at 0x04513DF0>
09:09:43 : incoming message:
{"msg": "{\"teamsView\": [[\"101\", \"--\", \"UNASSIGNED\", \"Ground Type 1\", \"AC\"]], \"assignmentsView\": [[\"AA\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AB\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AD\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AE\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AF\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AG\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AH\", \"--\", \"UNASSIGNED\", \"Ground Type 2\"], [\"AC\", \"101\", \"COMPLETED\", \"Ground Type 1\"]], \"teamsCount\": \"0 Assigned / 1 Unassigned\", \"assignmentsCount\": \"1 Assigned / 7 Unassigned\"}"}
message repeated to 2 registered listener(s)
unregister: <websockets.server.WebSocketServerProtocol object at 0x04513DF0>

Maybe this would be simplified if the server kept its ws connection open while running, rather than open and close with every message?

[caver456]

other recent code changes involved avoiding multiple back-to-back websockets sends, to avoid race conditions and html flickers on listeners, by only pushing once per compound operation like new pairing, and pairing status change to DONE. Hopefully that's the cause; but, the server appears dead from that point on.