TTL Export partition

[arthurpassos]

Implement a new TTL syntax, TTL export.

TTL export shall look at expired partitions (partition is an indivisible unit) and export them using the built in partition export. The system.replicated_partition_exports shall be used a history so partitions that have already been exported but not deleted are skipped. We shall export task n + 1 only once task n has completed. Failures shall be re-started agressively, the backoff policy should be implemented inside the export task.

[mkmkme]

Discussed this issue privately and here's the summary of the changes we're making (rendering #1792 obsolete).

(I actually used markdown formatting here, no AI involved)

Actual changes in system.replicated_partition_exports

The table gains a new field called source_of_export. The possible values are: alter, ttl. The field represents whether the export task was caused by ALTER TABLE ... EXPORT PARTITION command or by TTL EXPORT respectively.
The table entries with source=alter are not subject of change. The entries with source=ttl should be excluded from TTL eviction of the table since they're going to be the source of truth for TTL EXPORT.

For each pair of (source_table, dest_table) there will be only one entry in system.replicated_partition_exports with source=ttl.

ALTER TABLE ... EXPORT PARTITION changes

ALTER TABLE ... EXPORT PARTITION is going to respect SETTINGS export_merge_tree_partition_mark_as_ttl=1. In that case, the entry will be added with source=ttl instead of source=alter. Effectively this changes the marker for TTL EXPORT scheduler and should be used by DBA only for error recovery.

An attempt of exporting the partition that is older than the one currently in system.replicated_partition_exports should produce error stating that this is a dangerous operation and thus requires export_merge_tree_partition_force_export=1 to be set.

TTL EXPORT changes

CREATE TABLE / ALTER TABLE with TTL EXPORT should perform the same schema compatibility check as ALTER TABLE ... EXPORT PARTITION.

The flow of the TTL EXPORT scheduler can be described as following:

• Check the partition needed to export by checking the marker in the table
  • If no entry, grab the first partition eligible for exporting by TTL
  • If there's a succeeded entry, grab the next one that is eligible for exporting by TTL and replace the entry in system.replicated_partition_exports
  • If there's a failed entry, reschedule it agressively
  • If there's a pending entry, do nothing