tlog cdc test AbsoluteDeadline more robust under load

[ploxiln]

Sometimes unit test /NativeCDC/TLogCommittedFrontier/AbsoluteDeadline failed in nightlys (mostly for release-8.0 branch):

Assertion result.minKnownCommittedVersion == 93 failed

This was probably due to system load, where the peek timed-out at 0.4 seconds, before the last intended fixture.advance(frontier) which should be around 0.3 seconds (but can be more delayed under load).

Fix by stopping the frontier advance loop if the peek already timed-out, and comparing to what (and when) the last frontier advance actually was.

[ploxiln]

How I tested this locally: First change the delay(0.1) to delay(0.132) to get the race very close, and then:

for i in $(seq 20); do yes > /dev/null & done
for i in $(seq 20); do nice ./bin/fdbserver_tlog_test -f /NativeCDC/TLogCommittedFrontier/AbsoluteDeadline | grep -E 'failed|Assert'; done
kill %9 %2 %3 %4 %5 %6 %7 %8 %10 %11 %12 %13 %14 %15 %16 %17 %18 %19 %20 %21

With main and release-8.0 this fails 50% of runs:

Assertion result.minKnownCommittedVersion == 93 failed @ /Users/pierce/scratch/okteto-rhel9/foundationdb/fdbserver/tlog/TLogServer.cpp 4750:
  atos -o /Users/pierce/scratch/buildfdb/bin/fdbserver_tlog_test -arch x86_64 -l 0x0 0x10592caa4 0x104f06a3c 0x10597f7a4 0x105a12cec 0x104c463f4 0x182030d7c
Test failed: /NativeCDC/TLogCommittedFrontier/AbsoluteDeadline: An internal error occurred
0 tests passed; 1 tests failed.
1 tests passed; 0 tests failed.
1 tests passed; 0 tests failed.
1 tests passed; 0 tests failed.
Assertion result.minKnownCommittedVersion == 93 failed @ /Users/pierce/scratch/okteto-rhel9/foundationdb/fdbserver/tlog/TLogServer.cpp 4750:
  atos -o /Users/pierce/scratch/buildfdb/bin/fdbserver_tlog_test -arch x86_64 -l 0x0 0x105224aa4 0x1047fea3c 0x1052777a4 0x10530acec 0x10453e3f4 0x182030d7c
Test failed: /NativeCDC/TLogCommittedFrontier/AbsoluteDeadline: An internal error occurred
0 tests passed; 1 tests failed.

... etc

and with this branch it always passes

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: 3803dce
• Duration 0:21:47
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 3803dce
• Duration 0:47:58
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux RHEL 9

• Commit ID: 3803dce
• Duration 0:48:43
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 3803dce
• Duration 0:50:27
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 3803dce
• Duration 1:32:12
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 3803dce
• Duration 1:37:59
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 3803dce
• Duration 2:14:30
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)