Atomic commit of ingest/import jobs

### Background

An ingest or bulk import job can run multiple times, either because SQS delivered it twice, or because it failed, potentially after adding some data. Every run of the job will add the same data to the state store, resulting in duplication.

During standard ingest in an ingest task, files are written to S3 any time the ingest coordinator fills up a record batch. These are added to the state store immediately after they are created. This means at a non-deterministic point partway through an ingest, some of the data appears in the Sleeper table.

An ingest or bulk import job can also run multiple times, especially if it fails after adding some data. Every run of the job will add the same data to the state store, resulting in duplication.

Some implementations of the state store can also fail part way through adding files, in which case some will be added and some will not. This will be retried and this may also result in duplicated data.

For some use cases this sort of duplication is fine. A table iterator can filter out duplicates during queries and compaction.

For some use cases it may be a problem that we don't have a way to recover the state of the system before the failure. A job may add some data to a table, then fail in an unrecoverable way so that the only way to get the rest of the data in is to make a new job after removing some bad data.

Related to:
- https://github.com/gchq/sleeper/issues/2688
- https://github.com/gchq/sleeper/issues/5731

### Description

We'd like to add the capability to avoid the same data being added more than once to a Sleeper table in the same ingest or bulk import job.

### Analysis

We could wait until a job is finished before adding any files to the state store. We could track the IDs of jobs that have been committed, so that each job can only commit to the state store once. This seems like the simplest solution, though it may not be desirable in all cases. We could make it an option, potentially configured on each job, with a default in a table property.

We could provide some lock on the job, where once a process adds files, it takes ownership of the job. That could prevent other runs of the job from updating the state store. That may not be desirable because we still need to be able to retry the job.

We could track which run of an ingest job each file was added against. We could potentially ensure that a file will not be compacted until the whole ingest job is finished. We could replace any files from a previous run of the job when a new run adds files. We would need to decide which run of the job is allowed to add more files.

It may be difficult to make the whole update to the state store atomic because for some implementations (e.g. DynamoDB), there can be too many files to do it in one transaction. We could have a separate record for the job as a whole, so the files will only really be added once the whole job is committed.

#### Scenario to duplicate data with standard ingest

1. Create an ingest job that is big enough that the record batch will fill up halfway through
2. Wait for the first files to be added as the job runs in an ingest task
3. Terminate the ingest task while it's processing the rest of the job
4. See partial commit of data from the ingest job in the Sleeper table
5. Notice there's no way to recover the Sleeper table to a state where the data is tracked deterministically
6. See the ingest job runs again on a new task, and creates duplicate data in the Sleeper table