Page History
Experimental Overlapped File Transfer
Wiki page not yet completed, and has not been reviewed for correctness.
This experimental feature introduced in HTCondor version 8.1.6 allows a pipelined use of an execute slot by overlapping the execution of one job with the transferring of output from the previous job. This work is detailed in #4291. The motivation behind this feature is that the amount of CPU time used for transferring output files back to the submit host can be significant, while the CPU usage during the transfer is minimal. The goal is a more productive use of the CPU during this transfer time, by allowing it to start on a new job.
In implementation, an execute slot is paired with minimal-resource slot by configuration, and both slots are claimed together. These minimal-resource slots are called a transfer slots or buddy slots.
Implemented only for static slots.
Configuration
The metaknob that configures all static slots on a machine to use this experimental feature is
USE EXPERIMENTAL : Async_Stageout