Failure: MasterCI #311- RegressionTestsRelease: S3 (aws_s3, 1), S3 (gcs, 1), S3 (minio, 1) - Code 76: Cannot lock /var/lib/clickhouse/status (CANNOT_OPEN_FILE)

[CarlosFelipeOR]

Description

All three S3 backends (AWS, GCS, MinIO) in MasterCI #311 are failing due to the same underlying ClickHouse restart error.

During steps that update storage.xml and trigger a ClickHouse restart, the server fails to start and logs:

Code: 76. DB::Exception: Cannot lock file /var/lib/clickhouse/status.
Another server instance in same directory is already running. (CANNOT_OPEN_FILE)

Because the server does not come back up, the tests waiting for config reload or health checks eventually hit ExpectTimeoutError, causing the scenarios to fail.

This issue is backend-agnostic and appears to be caused by an incomplete stop sequence leaving orphaned ClickHouse processes that still hold the status file lock.

Summary (Root Cause & Resolution)

The restart failures were not caused by orphaned ClickHouse processes as initially suspected.
The actual root cause was a regression introduced on 2025-10-17, when a new known issue (XFail) was added for >= 25.8.

This XFail caused the PID-termination retry loop to exit prematurely, leading the framework to:

• remove /tmp/clickhouse-server.pid while the server was still running, and
• attempt to start a new ClickHouse instance without killing the previous one.

This created a race condition that consistently produced:

Code: 76 — Cannot lock /var/lib/clickhouse/status
Another server instance in same directory is already running.

Additionally, we identified that overly broad XFail patterns (e.g., using /*) contributed to the problem.
Such patterns can unintentionally match retry iterations, preventing the retry loop from executing properly.
These broad patterns will be corrected in the S3 test suite, and a future cleanup will update all suites to use precise XFail paths to avoid similar issues.

The fix (PR #70) updates the restart logic to terminate all running ClickHouse server processes regardless of PID file state, fully resolving the issue.

A follow-up fix PR #72 was needed because PR #70 introduced a retry-loop syntax error.
This issue became visible in the /version test on ClickHouse 22.8, where the retry logic was actually triggered and failed.
PR #72 fixes the syntax and removes the obsolete XFail, ensuring the retry loop works correctly.

This was not an upstream ClickHouse problem.

---

CI Run

MasterCI #311

Artifacts and Reports

ClickHouse® CI Workflow Run Report

Suite Report:

• AWS
• GCS
• MiniO

[CarlosFelipeOR]

Last CI Run

MasterCI #313

Artifacts and Reports

ClickHouse® CI Workflow Run Report

Suite Report:

• AWS
• GCS
• MiniO

Analysis

In MasterCI #311, the failures occurred only in:

/s3/aws s3/part 1/disk/access
/s3/gcs/part 1/disk/access
/s3/minio/part 1/backup/local and s3 disk

In MasterCI #313, the same test failed again, but we now also see an additional failure in:

/s3/aws s3/part 1/table function
/s3/gcs/part 1/table function

Both failures show the same underlying error:

Code: 76. DB::Exception: Cannot lock file /var/lib/clickhouse/status.
Another server instance in same directory is already running. (CANNOT_OPEN_FILE)

This indicates that the issue is not tied to a specific scenario, but rather a systemic restart problem triggered whenever the test framework updates the configuration and attempts to restart ClickHouse. The failure pattern is consistent with the server being unable to acquire the status file lock due to leftover ClickHouse processes from the previous instance.

[CarlosFelipeOR]

Historical CI data confirms Code: 76 failures across multiple versions since 2025-10-17

I checked the Altinity internal test database to understand how far back this failure goes.

SELECT clickhouse_version, MAX(start_time), COUNT(start_time) FROM `gh-data`.clickhouse_regression_results
WHERE test_name LIKE '%/s3/aws s3/part 1/disk/access%'
AND result_message LIKE '%Code: 76%'
GROUP BY clickhouse_version
ORDER BY MAX(start_time) DESC
LIMIT 100

The query shows that Code: 76 (“Cannot lock /var/lib/clickhouse/status”) has been occurring since 2025-10-17, affecting multiple ClickHouse versions across both upstream and Antalya builds:

• 25.8.x (many builds)
• 25.9.x
• 25.10.x
• Both official upstream and Antalya variants

Here is the summarized result:

clickhouse_version	last_failure	count
25.8.12.129	2025-11-27 04:45:40	2
25.8.12.20134.altinityantalya	2025-11-27 03:39:51	2
(empty / N/A)	2025-11-27 01:00:38	42
25.10.2.65-alpine	2025-11-27 00:19:58	16
25.8.9.20496.altinityantalya	2025-11-26 22:06:56	4
... (many others antalya) ...	...	~100
25.9.3.48-alpine	2025-11-01 00:21:41	16
25.8.10.7-alpine	2025-10-28 08:08:45	12
25.8.9.20207	2025-11-10 11:55:47	30
25.8.9.20172	2025-10-17 11:47:50	2

This confirms that the restart failure (Code: 76) is not specific to a single version, backend (AWS/GCS/MinIO), or release cycle.
It has been consistently present across all S3-related suites for more than a month.

This aligns with the root cause already identified:
ClickHouse restart sometimes leaves child processes alive, which keep the /var/lib/clickhouse/status file locked, leading to repeated startup failures across scenarios.

[CarlosFelipeOR]

Update: Root Cause Identified and Fix Proposed

The intermittent failures in the S3 test suites (AWS, GCS, MinIO) have been traced to a ClickHouse restart issue triggered during config reload steps.

---

🔍 Root Cause Analysis

During tests that modify storage.xml and perform a ClickHouse restart, stop_clickhouse() was terminating only the main ClickHouse PID, while several internal ClickHouse child processes/threads remained alive (e.g., DNSResolver, SystemLog workers, system.part_log, TCPHandler, etc.).

These leftover processes continued to hold the lock on:

/var/lib/clickhouse/status

As a result, the next startup failed with:

Code: 76. DB::Exception: Cannot lock file /var/lib/clickhouse/status.
Another server instance in same directory is already running.

This explains all failures observed in:

• /s3/aws s3/part 1/disk/access
• /s3/aws s3/part 1/table function
• and the equivalent GCS/MinIO steps where the test waited for ClickHouse to restart and timed out.

---

Proposed Fix (Implemented in PR #70)

PR: Altinity/clickhouse-regression#70

The fix introduces a more robust shutdown procedure by enhancing stop_clickhouse() to:

• Detect any remaining ClickHouse processes after killing the main PID
• Repeatedly terminate them with the configured signal
• Fall back to SIGKILL if they persist after several attempts
• Guarantee that all ClickHouse processes are gone before calling start_clickhouse()

This ensures the status file lock is fully released and prevents orphaned processes from interfering with the next restart.

---

Results

• ClickHouse restarts become deterministic and clean
• The status file is no longer locked by remaining threads
• S3 test failures triggered by restart issues are resolved across:
  • AWS
  • GCS
  • MinIO

This fix stabilizes all S3 backends in the regression suite and eliminates the intermittent Code: 76 failures observed in MasterCI runs.

[CarlosFelipeOR]

PR #70 has been approved and merged.

This issue will remain open because we still need to verify whether this behavior is an upstream issue or expected in the newer versions.

[CarlosFelipeOR]

Detailed Investigation: Double-Start Behavior of clickhouse server --daemon

Below is a detailed description of the behavior we observed when the ClickHouse server is started multiple times consecutively using:

clickhouse server \
  --config-file=/etc/clickhouse-server/config.xml \
  --log-file=/var/log/clickhouse-server/clickhouse-server.log \
  --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log \
  --pidfile=/tmp/clickhouse-server.pid \
  --daemon

This behavior explains why the regression framework can enter a broken state during restart operations.

1. First start — normal behavior

When the command is executed for the first time:

• /tmp/clickhouse-server.pid is created
• The server starts correctly
• /var/lib/clickhouse/status is locked by the running server
• No errors appear in the log

Everything works as expected.

Second start — PID file gets deleted but the server keeps running

Running the same command again without killing the previous instance produces surprising behavior:

• The second invocation removes /tmp/clickhouse-server.pid
• The running server continues operating normally
• The log records:

Code: 76. DB::Exception: Cannot lock file /var/lib/clickhouse/status.
Another server instance in same directory is already running.

Important details:

• Only the second (new) instance shuts down
• The original server continues running in the background
• The PID file is gone (deleted by the second start), even though the server is still alive
  This means automation relying on the PID file becomes out-of-sync with reality.

Impact on regression restart logic

The regression framework follows this restart sequence:

• Read PID from /tmp/clickhouse-server.pid
• kill -TERM
• Start a fresh server instance
• Wait for “Loaded config” in logs

However, after the double-start behavior:

• /tmp/clickhouse-server.pid no longer exists
• Steps (1) and (2) are skipped
• The framework believes the server is already stopped
• It attempts to start a new server instance while the previous one is still running

This triggers:

Cannot lock file /var/lib/clickhouse/status

The framework then waits ~5 minutes for “Loaded config” and ultimately fails with a timeout.

Reproduction Steps

# 1) Confirm no server is running
ps -ef | grep 'clickhouse.*server'
cat /tmp/clickhouse-server.pid   # should not exist

# 2) Start ClickHouse once
clickhouse server \
  --config-file=/etc/clickhouse-server/config.xml \
  --log-file=/var/log/clickhouse-server/clickhouse-server.log \
  --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log \
  --pidfile=/tmp/clickhouse-server.pid \
  --daemon

# 3) Verify it started
cat /tmp/clickhouse-server.pid
cat /var/lib/clickhouse/status

# 4) Start ClickHouse a second time
clickhouse server \
  --config-file=/etc/clickhouse-server/config.xml \
  --log-file=/var/log/clickhouse-server/clickhouse-server.log \
  --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log \
  --pidfile=/tmp/clickhouse-server.pid \
  --daemon

# 5) Observe:
# - /tmp/clickhouse-server.pid is deleted
# - Error 76 appears in logs
# - Original server keeps running

# 6) Attempting a restart now fails:
clickhouse server \
  --config-file=/etc/clickhouse-server/config.xml \
  --log-file=/var/log/clickhouse-server/clickhouse-server.log \
  --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log \
  --pidfile=/tmp/clickhouse-server.pid \
  --daemon
# → Error 76: Cannot lock /var/lib/clickhouse/status

[CarlosFelipeOR]

1. Discarding ClickHouse Child Processes as the Cause

Initially we suspected that leftover ClickHouse child processes (e.g., DNSResolver, SystemLog workers, part log threads, etc.) might be holding the lock on:

/var/lib/clickhouse/status

However, deeper analysis showed that the processes still visible after shutdown were actually tail processes reading the status file, not leftover internal ClickHouse workers.
These were unrelated to the locking issue and therefore not the root cause.

---

2. Evidence of Multiple ClickHouse Starts Without a Prior Kill

In several test runs we observed that the regression framework attempted to start a new ClickHouse server instance without killing the previous one.

Manual reproduction confirmed a key behavior:

• Running clickhouse server --daemon twice deletes /tmp/clickhouse-server.pid of the existing instance.
• Running it a third time results in:

Code: 76. DB::Exception: Cannot lock file /var/lib/clickhouse/status.
Another server instance in same directory is already running.

This is the exact error pattern seen in the failing test suites.

---

3. Timeline Correlation With Issue ClickHouse#88699 and Known Issue Introduction

The problem aligns perfectly with the following sequence:

• 2025-10-16 — Issue Data mismatch when reading via s3Cluster table function ClickHouse/ClickHouse#88699 created

• 2025-10-17 at 08:35 — Known issue added:
  Altinity/clickhouse-regression@94623a5

  ":/part 1/table function/*": [
      (Fail, "https://github.com/ClickHouse/ClickHouse/issues/88699",
       check_clickhouse_version(">=25.8")),
  ]

• 2025-10-17 at 11:47 — The first occurrence of Error Code 76 appears in our regression logs.

The timestamps leave almost no doubt:
the issue was introduced immediately after the known issue/XFail entry was added.

---

4. How the XFail Introduces a Race Condition

The retry loop responsible for verifying that the ClickHouse PID has terminated normally works like this:

1. kill -TERM <pid>
2. Retry ps <pid> until the process disappears

But with the XFail active:

• The loop exits prematurely on the first retry
• The framework incorrectly assumes the server has terminated
• It proceeds to:

rm -rf /tmp/clickhouse-server.pid

even though the server is still alive

This creates a race condition:

• The process is still running
• The PID file is removed
• The next clickhouse server --daemon attempt starts while the previous instance is still running
• The second attempt removes the PID file again
• The third attempt hits Error Code 76

This behavior explains exactly why tests started to fail only after the new known issue was introduced.

---

5. Why Manual Reproduction is Difficult

Reproducing this manually is challenging because:

• ClickHouse often terminates quickly in interactive environments
• The race timing window is small
• Automation triggers steps significantly faster than a human can

Therefore the regression framework encounters the failure consistently, while manual attempts may not.

---

6. Fix Validation

The fix implemented in PR #70 addresses the core issue:

• Instead of relying on /tmp/clickhouse-server.pid,
• The framework now searches for all processes matching the ClickHouse server command,
• And terminates them explicitly before attempting to start a new instance.

This guarantees no orphaned server remains running, fully preventing Error Code 76.

The fix has been verified across multiple versions and behaves correctly in all scenarios tested.

---

7. Final Conclusion

• This is not a ClickHouse upstream issue
• The problem was introduced by the XFail/known-issue logic added on 2025-10-17
• It caused premature termination of the retry loop and forced deletion of the PID file while the server was still running
• PR #70 resolves the issue completely

We can safely close this issue.

Full Log

Note: The log below shows the PID file being removed while the original ClickHouse server process is still running.

⟥  When I close terminal to the node to be restarted
            2s 214ms                  [clickhouse1] 
                19ms                 ⟥⟤ OK I close terminal to the node to be restarted, /s3/aws s3/part 1/table function/remote host filter/remote_host_filter․xml/I wait for config to
 be reloaded/I close terminal to the node to be restarted
Nov 26,2025 17:04:36                 ⟥  And I stop ClickHouse to apply the config changes
Nov 26,2025 17:04:36                   ⟥  By capturing pid
                 3ms                        [clickhouse1] bash# cat /tmp/clickhouse-server.pid
                 4ms                        [clickhouse1] 2587
                 7ms                        [clickhouse1] bash# echo $?
                 8ms                        [clickhouse1] 0
                 9ms                        [clickhouse1] bash#
                 9ms                   ⟥⟤ OK capturing pid, /s3/aws s3/part 1/table function/remote host filter/remote_host_filter․xml/I wait for config to be reloaded/I stop ClickHous
e to apply the config changes/capturing pid
Nov 26,2025 17:04:36                   ⟥  And sending kill -TERM to ClickHouse server process on clickhouse1
                 3ms                        [clickhouse1] bash# kill -TERM 2587
                 6ms                        [clickhouse1] bash# echo $?
                 6ms                        [clickhouse1] 0
                 6ms                        [clickhouse1] bash#
                 6ms                   ⟥⟤ OK sending kill -TERM to ClickHouse server process on clickhouse1, /s3/aws s3/part 1/table function/remote host filter/remote_host_filter․xml/
I wait for config to be reloaded/I stop ClickHouse to apply the config changes/sending kill -TERM to ClickHouse server process on clickhouse1
Nov 26,2025 17:04:36                   ⟥  And checking pid does not exist
Nov 26,2025 17:04:36                     ⟥  Step try #0, flags:RETRY
                 3ms                          [clickhouse1] bash# ps 2587 | grep -v grep | grep ' clickhouse.server '
                 8ms                          [clickhouse1]    2587 ?        Ssl    1:00 clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhous
e-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --pidfile=/tmp/clickhouse-server.pid --daemon
                10ms                          [clickhouse1] bash# echo $?
                10ms                          [clickhouse1] 0
                11ms                          [clickhouse1] bash#
                11ms                     ⟥⟤ XFail try #0, /s3/aws s3/part 1/table function/remote host filter/remote_host_filter․xml/I wait for config to be reloaded/I stop ClickHouse 
to apply the config changes/checking pid does not exist/try #0, pid still alive, https://github.com/ClickHouse/ClickHouse/issues/88699
                12ms                   ⟥⟤ OK checking pid does not exist, /s3/aws s3/part 1/table function/remote host filter/remote_host_filter․xml/I wait for config to be reloaded/I 
stop ClickHouse to apply the config changes/checking pid does not exist
Nov 26,2025 17:04:36                   ⟥  By deleting ClickHouse server pid file
                 3ms                        [clickhouse1] bash# rm -rf /tmp/clickhouse-server.pid
                 6ms                        [clickhouse1] bash# echo $?
                 6ms                        [clickhouse1] 0
                 7ms                        [clickhouse1] bash#

[CarlosFelipeOR]

New issue observed in retry logic after PR #70

It looks like the fix introduced in PR Altinity/clickhouse-regression#70 also triggered a new issue.
It was observed on the version suite ('/version/altinity/error message') when testing ClickHouse 22.8.15.25.altinitystable.

Scheduled Altinity 22.8 run:
https://github.com/Altinity/clickhouse-regression/actions/runs/19774562166/job/56664977631

In other cases, after the first pkill -{signal} -f 'clickhouse.*server', the remaining ClickHouse processes usually exit quickly, allowing the code to reach:

result = self.command("pgrep -f 'clickhouse.*server' || true")
if not result.output.strip():
    break

Because of that, the retry loop never reaches the escalation condition.

But in this specific version, the server process takes longer to terminate, so the retry loop eventually evaluates:

if attempt.iteration >= 5:

RetryIteration has no iteration attribute, leading to:

AttributeError: 'RetryIteration' object has no attribute 'iteration'

So the retry logic itself is correct, the problem is simply the wrong attribute used to check the attempt counter.

[CarlosFelipeOR]

New proposal: Improve TestFlows behavior with xfail + retries

The function stop_clickhouse() in helpers/cluster.py is used by both the version tests and the S3 tests.
Both failure cases had something else in common: the failing step was marked as xfail (known issue).

This helped identify a specific TestFlows behavior when xfail is combined with retry logic.

---

🧠 Observation: xfail makes retries stop after the first failed attempt

When using Retrying Code inside a scenario/step that is marked xfail, the retry mechanism does not run all iterations.

Instead, it behaves like this:

• On the first failed attempt, TestFlows immediately marks the step as XFail
• It then skips all remaining retries
• Execution continues to the next step

This differs from the behavior when xfail is not present, where retries run normally until:

• the maximum number of iterations is reached, or
• the timeout is hit, or
• one attempt succeeds.

---

🔍 Example from the version test

With xfail

Only try #0 runs, and because it fails, the step is immediately marked:

⟥  And checking pid does not exist
Dec 01,2025 20:01:56             ⟥  Step try #0, flags:RETRY
                 2ms                  [clickhouse1] bash# ps 24 | grep -v grep | grep ' clickhouse.server '
                 4ms                  [clickhouse1]      24 ?        Ssl    0:00 clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --pidfile=/tmp/clickhouse-server.pid --daemon
                 6ms                  [clickhouse1] bash# echo $?
                 6ms                  [clickhouse1] 0
                 6ms                  [clickhouse1] bash#
                 6ms             ⟥⟤ XFail try #0, /version/altinity/error message/the ClickHouse process is restarted/checking pid does not exist/try #0, pid still alive, https://github.com/Altinity/ClickHouse/pull/899
                 6ms           ⟥⟤ OK checking pid does not exist, /version/altinity/error message/the ClickHouse process is restarted/checking pid does not exist
Dec 01,2025 20:01:56           ⟥  By killing any remaining ClickHouse child processes

Retries never happen.

Without xfail

Dec 01,2025 20:24:17           ⟥  And checking pid does not exist
Dec 01,2025 20:24:17             ⟥  Step try #0, flags:RETRY
                 2ms                  [clickhouse1] bash# ps 24 | grep -v grep | grep ' clickhouse.server '
                 4ms                  [clickhouse1]      24 ?        Ssl    0:00 clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server
/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --pidfile=/tmp/clickhouse-server.pid --daemon
                 6ms                  [clickhouse1] bash# echo $?
                 6ms                  [clickhouse1] 0
                 6ms                  [clickhouse1] bash#
                 6ms             ⟥⟤ Fail try #0, /version/altinity/error message/the ClickHouse process is restarted/checking pid does not exist/try #0, pid still alive
Dec 01,2025 20:24:20             ⟥  Step try #1, flags:RETRY
                 5ms                  [clickhouse1] bash# ps 24 | grep -v grep | grep ' clickhouse.server '
                10ms                  [clickhouse1]      24 ?        Ssl    0:00 clickhouse server --config-file=/etc/clickhouse-server/config.xml --log-file=/var/log/clickhouse-server
/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log --pidfile=/tmp/clickhouse-server.pid --daemon
                13ms                  [clickhouse1] bash# echo $?
                13ms                  [clickhouse1] 0
                13ms                  [clickhouse1] bash#
                14ms             ⟥⟤ Fail try #1, /version/altinity/error message/the ClickHouse process is restarted/checking pid does not exist/try #1, pid still alive
Dec 01,2025 20:24:23             ⟥  Step try #2, flags:RETRY
.....
Several Retries
.....
                12ms                  [clickhouse1] bash#
                12ms             ⟥⟤ Fail try #6, /version/altinity/error message/the ClickHouse process is restarted/checking pid does not exist/try #6, pid still alive
Dec 01,2025 20:24:38             ⟥  Step try #7, flags:RETRY
                 4ms                  [clickhouse1] bash# ps 24 | grep -v grep | grep ' clickhouse.server '
                13ms                  [clickhouse1] bash# echo $?
                13ms                  [clickhouse1] 1
                14ms                  [clickhouse1] bash#
                14ms             ⟥⟤ OK try #7, /version/altinity/error message/the ClickHouse process is restarted/checking pid does not exist/try #7
           21s 100ms           ⟥⟤ OK checking pid does not exist, /version/altinity/error message/the ClickHouse process is restarted/checking pid does not exist
Dec 01,2025 20:24:38           ⟥  By killing any remaining ClickHouse processes

Retry behavior details

The step only succeeds on try #7, as expected.

This proves the retry loop is correctly implemented and robust, it attempts the check up to 19 times, escalates with SIGKILL at iteration 20, and continues up to the configured timeout.

But with xfail applied, the loop breaks prematurely and skips directly to the next steps, which caused the failures we saw in both /version and /S3.

---

❗ Problem summary

• The retry loop in stop_clickhouse() is supposed to run many attempts (up to timeout).
• With xfail, TestFlows stops after the first failed attempt, even though that is not a real test failure.
• This prevents the cleanup logic from running and leads to cascading issues (leftover PIDs, restart errors, etc.).

Therefore, the failures in /version and /S3 were a side effect of xfail breaking retry behavior, not a problem with ClickHouse itself.

---

✅ Proposed actions

1. Quick fix

• Fix the syntax issue (attempt.iteration → correct attribute).
• Remove or adjust xfail entries in the affected tests.
• Ensure retries behave correctly (use common loop instead of Retries).

2. Propose an enhancement to TestFlows so that:

xfail does not override retry logic.
A retry should only produce an XFail if all attempts fail, not when the first attempt fails.

Failing one iteration ≠ failing the test.

This would make retry behavior consistent, predictable, and aligned with how retries are typically expected to work.

[CarlosFelipeOR]

ChatGPT said:

Update on actions taken

Following the proposed actions, here is what has been completed:

1. Fixed the retry loop syntax
   The incorrect attribute used inside stop_clickhouse() (attempt.iteration) was corrected.

2. Removed the obsolete xfail
   The xfail entry for /version/altinity/error message/* was removed after validating that the test passes on all Altinity Stable versions < 25.3.

3. Validated behavior across suites
   The /S3 and /version suites were executed in CI/CD and are now passing with the updated retry logic and without the incorrect xfail.

PR: Altinity/clickhouse-regression#72

These changes address the immediate failures caused by the retry + xfail interaction.

---

Additional findings about TestFlow + xFail

As discussed earlier, TestFlows' behavior is technically correct: an xfail is treated as a passing result, so retries do not continue after the first failed attempt.
In this case, the issue was not TestFlows itself, but the use of an xfail expression on a step that depends on retries for proper cleanup.

We also found that overly broad xfail patterns (e.g., using /*) can unintentionally match retry steps and stop the retry loop early.
This contributed to the recent S3 failures, as the retry logic never executed fully when the broad xfail pattern applied.

To avoid this, xfail patterns should be as specific as possible and target only the exact test case.

As next steps:

• I will clean up the xfail patterns in the S3 suite to make them more precise.
• A future review should verify that no other suites are using overly broad patterns that could interfere with retry behavior.