traceevents for additional info on recruitment failover to remoteDC

[akshay-v-kaushik]

Summary

When the primary region can't satisfy a recruitment (for example, all satellite tlog workers sit in one zone),
the cluster controller silently falls back to the other region. The only log was
AttemptingRecruitmentInRemoteDc with Error=no_more_servers, which doesn't say what failed or why.

This PR adds a RecruitmentFailed trace event at each place that throws no_more_servers inside the
primary recruitment pass. No behavior change.

What's logged

A new RecruitmentFailed trace event (SevWarn) is emitted just before each existing throw no_more_servers()
in the primary recruitment pass. RecruitingRole says what was being recruited, and the other fields show why it failed.

Test plan

The first commit adds a temporary unit test, tempRecruitmentFailureTraces, to trigger each trace.
The second commit removes it, so the PR's final diff contains only the trace changes.

Scenarios in the temporary test:

1. Primary tlogs, 12 workers in one zone, policy needs 3 zones (Simple helper)
2. Primary tlogs, only one data hall qualifies (Complex helper)
3. Primary tlogs, custom policy data_hall^3 x 1 (Backup helper)
4. Satellite DC with 12 workers in one zone, primary tlogs fine
5. Seed storage with 12 workers in one zone
6. First commit proxy with no workers

Each scenario asserts that findWorkersForConfigurationFromDC still throws no_more_servers.

Run:
bin/fdbserver -r unittests -f /fdbserver/clustercontroller/tempRecruitmentFailureTraces --logdir <dir>
grep -h RecruitmentFailed <tracefile>

[saintstack]

Looks good.

Small detail below.

Thanks.

[saintstack]

You do this twice. Want to make a getFitnessWorkersCandidates local helper method?

[saintstack]

Again!

[akshay-v-kaushik]

@saintstack added a helper

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux RHEL 9

• Commit ID: 04ee310
• Duration 0:35:34
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux RHEL 9

• Commit ID: 04ee310
• Duration 1:00:50
• Result: ❌ FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux RHEL 9

• Commit ID: 04ee310
• Duration 1:03:54
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux RHEL 9

• Commit ID: 04ee310
• Duration 1:04:41
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

• Commit ID: 04ee310
• Duration 1:51:23
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS 14.x

• Commit ID: 04ee310
• Duration 1:02:24
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS 14.x

• Commit ID: 04ee310
• Duration 2:19:19
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[saintstack]

nit: could just call countCandidateWorkers here and save the local variable allocation... no biggie.