[BUG] Tables below version 9 lack hoodie.table.precombine.field, so 0.x and 1.0.x see no ordering field #20345

Bug Description

What happened:

Since 1.1.0, HoodieTableMetaClient.TableBuilder records the ordering fields of a table only as hoodie.table.ordering.fields, at every table version. Readers of a table below version 9 only know hoodie.table.precombine.field: Hudi 0.x for version 6 and Hudi 1.0.x for version 8. So a table that a 1.x writer creates with hoodie.write.table.version 6 or 8 (Spark datasource or SQL, Hudi Streamer, Flink, Kafka Connect) has no ordering field for those readers. The 9 to 8 downgrade writes hoodie.table.precombine.field for exactly that reason, but table creation does not.

ALTER TABLE ... RENAME TO does the same to tables that carry only the legacy key, such as tables written by 0.x. It re-creates hoodie.properties from the table's own properties through TableBuilder.fromProperties, which reads hoodie.table.precombine.field through the alternative key and writes it back only as hoodie.table.ordering.fields.

The key generator class has the same problem below version 8 (#20275).

What you expected:

A table below version 9 records its ordering fields under hoodie.table.precombine.field as well, as tables written by 0.x and 1.0.x do, and keeps that key when it is re-created from its own properties.

Steps to reproduce:

1. Create a table below version 9, for example with Spark SQL: create table t (id int, name string, ts long) using hudi tblproperties (primaryKey = 'id', orderingFields = 'ts', hoodie.write.table.version = '6').
2. .hoodie/hoodie.properties holds hoodie.table.ordering.fields=ts and no hoodie.table.precombine.field.
3. Loading that file into the HoodieTableConfig of Hudi 0.15.0 gives getPreCombineField() = null. For a version 8 table, Hudi 1.0.2 gives the same.

Reproducing tests in TestHoodieTableMetaClient, which fail on master for versions 6 and 8:

@ParameterizedTest
@EnumSource(value = HoodieTableVersion.class, names = {"SIX", "EIGHT", "NINE", "TEN"})
void testOrderingFieldsKeysFollowTableVersion(HoodieTableVersion version) throws IOException {
  HoodieTableMetaClient created = orderingFieldsTableBuilder(version)
      .setOrderingFields("ts")
      .initTable(this.metaClient.getStorageConf(), tempDir.toAbsolutePath() + Path.SEPARATOR + "ordering-v" + version.versionCode());
  assertOrderingFieldsKeys(version, created.getTableConfig());
}

@ParameterizedTest
@EnumSource(value = HoodieTableVersion.class, names = {"SIX", "EIGHT"})
void testReInitFromPropertiesKeepsLegacyOrderingFieldsKey(HoodieTableVersion version) throws IOException {
  String path = tempDir.toAbsolutePath() + Path.SEPARATOR + "legacy-ordering-v" + version.versionCode();
  HoodieTableMetaClient created = orderingFieldsTableBuilder(version).initTable(this.metaClient.getStorageConf(), path);
  Properties legacyOrderingFields = new Properties();
  legacyOrderingFields.setProperty(HoodieTableConfig.PRECOMBINE_FIELD.key(), "ts");
  HoodieTableConfig.update(created.getStorage(), created.getMetaPath(), legacyOrderingFields);
  created.reloadTableConfig();
  assertFalse(created.getTableConfig().getProps().containsKey(HoodieTableConfig.ORDERING_FIELDS.key()));

  HoodieTableMetaClient reInitialised = HoodieTableMetaClient.newTableBuilder()
      .fromProperties(created.getTableConfig().getProps())
      .initTable(this.metaClient.getStorageConf(), path);
  assertOrderingFieldsKeys(version, reInitialised.getTableConfig());
}

private static HoodieTableMetaClient.TableBuilder orderingFieldsTableBuilder(HoodieTableVersion version) {
  return HoodieTableMetaClient.newTableBuilder()
      .setTableType(HoodieTableType.MERGE_ON_READ.name())
      .setTableName("table")
      .setTableVersion(version)
      .setRecordKeyFields("id");
}

private static void assertOrderingFieldsKeys(HoodieTableVersion version, HoodieTableConfig tableConfig) {
  assertEquals("ts", tableConfig.getProps().getProperty(HoodieTableConfig.ORDERING_FIELDS.key()));
  assertEquals(version.lesserThan(HoodieTableVersion.NINE) ? "ts" : null,
      tableConfig.getProps().getProperty(HoodieTableConfig.PRECOMBINE_FIELD.key()),
      "precombine key at version " + version.versionCode());
}

Environment

Hudi version: master (1.3.0-SNAPSHOT), and every release since 1.1.0
Query engine: any writer that creates a table below version 9 (Spark, Flink, Hudi Streamer, Kafka Connect)
Relevant configs: hoodie.write.table.version = 6 or 8 with ordering fields set

Logs and Stack Trace

The reproducing tests on master:

[ERROR]   Run 1: TestHoodieTableMetaClient.testOrderingFieldsKeysFollowTableVersion:527->assertOrderingFieldsKeys:562 precombine key at version 6 ==> expected: <ts> but was: <null>
[ERROR]   Run 1: TestHoodieTableMetaClient.testOrderingFieldsKeysFollowTableVersion:527->assertOrderingFieldsKeys:562 precombine key at version 8 ==> expected: <ts> but was: <null>
[ERROR]   Run 1: TestHoodieTableMetaClient.testReInitFromPropertiesKeepsLegacyOrderingFieldsKey:549->assertOrderingFieldsKeys:562 precombine key at version 6 ==> expected: <ts> but was: <null>
[ERROR]   Run 1: TestHoodieTableMetaClient.testReInitFromPropertiesKeepsLegacyOrderingFieldsKey:549->assertOrderingFieldsKeys:562 precombine key at version 8 ==> expected: <ts> but was: <null>

hoodie.properties written by the table builder on master, loaded into the HoodieTableConfig of the released readers. reinit-v6 is a table that carried hoodie.table.precombine.field=ts, re-created from its own properties the way ALTER TABLE RENAME does:

== Hudi 0.15.0 (table version 6 reader)
tables-master/created-v6/.hoodie/hoodie.properties -> hoodie.table.version=6, getPreCombineField()=null
tables-master/reinit-v6/.hoodie/hoodie.properties -> hoodie.table.version=6, getPreCombineField()=null
== Hudi 1.0.2 (table version 8 reader)
tables-master/created-v8/.hoodie/hoodie.properties -> hoodie.table.version=8, getPreCombineField()=null