Ticket #2330: support pool defragmentation

Step 1: Add basic mechanisms for draining partitionable slots.

Step 2: Create a central daemon that implements a simple but useful pool defragmentation policy.


  1. [DONE] 2011-07-29 - Initial draft of plan with details of protocols, policies, and metrics.

  2. [DONE] 2011-08-12 - Initial draft of development plan (i.e. steps needing to be done and a time estimate).

  3. [DONE] 2011-08-26 Revision 2 of startd drain plan addressing issues in the remark below from tannenba aug-24-2011.

  4. [DONE] 2011-09-09 Revision 3 of startd drain plan to match Miron's requirements

  5. [DONE] 2011-09-20 Revision 4 of startd drain plan. The upper bound on eviction time is now explicit. Also, eviction of jobs is delayed in a coordinated way in order to reduce idle time.

  6. [DONE] 2011-09-22 Make a development plan for startd draining.

  7. [DONE] 2011-09-29 Add new draining commands to the startd and create a command-line tool that calls them. The commands are no-ops for now.

  8. [DONE] Make upper bound on time spent in Preempting/Vacate explicit. See #2536.

  9. [DONE] 2011-10-27 Implement transition to draining and cancellation of draining. No new jobs accepted (i.e. START=False), but no eviction yet.

  10. [DONE] 2011-10-31 Implement eviction. The graceful draining case must coordinate eviction to reduce idle time.

  11. [DONE] 2011-11-03 Implement Drained state.

  12. [DONE] 2011-11-03 Add support for auto-resume after defragmentation.

  13. [DONE] 2011-11-04 Add startd attributes that advertise draining badput and idle time and estimates of time to completion.

  14. [DONE] 2011-11-07 Add an automated test of draining.

  15. Draining mechanism is code complete. Checked in for 7.7.5.

  16. [DONE] 2011-11-09 Finish updating documentation.

  17. [DONE] 2011-11-15 Finish implementing and documenting condor_defrag.
[Append remarks]


2011-Aug-24 12:18:35 by tannenba:
For the record, here's what I think we concluded today in our chat on draining.

During draining, we still need most of the existing state machine, so it is awkward to introduce "draining" as an activity or a state. Therefore, we propose to add a 3rd independent state variable that indicates whether the startd is draining or not. It could be published as a boolean attributed named "Draining".

We talked about the semantics of the draining deadline. In the existing document, this was proposed to be the time after which claims transition to the retiring activity (i.e. the normal process by which the startd closes a claim). This means draining could take an arbitrary amount of time past the deadline, depending on the startd preemption policy. Not good.

We agreed that it would be more useful for the draining deadline to be the time at which we will finish vacating jobs. To accomplish this, the startd must advertise the earliest draining deadline that its policy will permit, and it will reject requests with shorter deadlines. The earliest possible draining deadline depends on MaxJobRetirementTime. I propose that the startd advertise CurrentMaxJobRetirementTime, just like it advertises CurrentRank. Once the startd has a draining deadline, it must modify its advertised MaxJobRetirementTime expression so that it never promises more retirement time than what remains before the deadline. This will prevent backfill jobs that require more retirement time from matching.

Currently, once MaxJobRetirementTime expires, the Preempting state is entered. How long preemption takes depends on WANT_VACATE and KILL. Again, this could be an arbitrary amount of time, so this could conflict with the draining deadline. The idea we talked about for addressing this was to change the semantics of MaxJobRetirementTime. Instead of this being the time at which preemption begins, it should be the time at which hard-killing begins.

2011-Aug-24 12:19:31 by danb:
I thought a bit more about this. I am concerned, because the proposed change provides a way to promise the job a certain total runtime, but it does not provide a way to promise a certain amount of vacate time. Perhaps we need both MaxJobRetirementTime and MaxJobVacateTime.

With MaxJobRetirementTime, the job can say that it wants less than what the machine is willing to offer (never mind about incentives for now). The same could be true for MaxJobVacateTime: e.g. if the job's SoftKillTime is less than the machine's MaxJobVacateTime. (If we followed the same convention as MaxJobRetirementTime, then instead of naming the job attribute SoftKillTime, it would be named the same as the machine attribute, MaxJobVacateTime.)

So under this proposal, I think we would get rid of WANT_VACATE and KILL. The function of these would be taken over by MaxJobVacateTime. We may not need/want to change the semantics of MaxJobRetirement time after all. Still need to think through some of the details.

2011-Sep-02 11:06:47 by danb:
After further discussion, Todd and I were less comfortable with revamping the preempting state in order to be able to predict an upper bound on how long it will take. We agreed to treat eviction time as negligible for now. So the draining deadline is the time at which eviction begins. In the future, this could be revisited.

I should also note that after our discussion, I realized that job suspension makes time of completion of MaxJobRetirementTime unknown in advance. I therefore adopted the stance that there should be a "graceful" deadline semantic that automatically extends to meet MaxJobRetirementTime promises. It seemed convenient to me for this behavior to apply both at the time of the request and later when/if job suspension takes place. Stricter deadline semantics could be added, but I thought we should at least start with "graceful".

2011-Sep-06 18:20:48 by danb:
Todd, Greg, and I (Dan) discussed the 2nd draft of the draining plan today.

We concluded that the proposed backfill toggle should be replaced with two things:

  1. a toggle that says whether early termination of draining should be accepted or not (i.e. what to do when the slots all happen to be idle). For the defragmentation daemon, we would expect early termination to be a good thing, whereas in the case of a coordinated pool-wide maintenance time, early termination might not be desired.

  2. a capability to specify arbitrary DrainingAttr attributes in the draining request

2011-Sep-12 10:31:11 by danb:
Miron wanted a different design. A new version of the draining design doc has been attached that attempts to achieve this.

Ideas to consider for the future:

2011-Sep-12 14:48:28 by danb:
Another discussion with Miron:

[Append remarks]


Type: enhance           Last Change: 2011-Nov-21 16:12
Status: resolved          Created: 2011-Jul-22 17:35
Fixed Version: v070705           Broken Version: v070000 
Priority:          Subsystem:  
Assigned To: danb           Derived From:  
Creator: danb  Rust:  
Customer Group: cms  Visibility: public 
Notify: dan@hep.wisc.edu,gthain@cs.wisc.edu,tannenba@cs.wisc.edu,tstclair@redhat.com,eje@cs.wisc.edu  Due Date:  

Derived Tickets:

#2536   make upper bound on time spent in Preempting/Vacate explicit

Related Check-ins:

2011-Dec-28 10:46   Check-in [28931]: last details for documenting condor_defrag, including some index entry changes ===GT=== #2330 (By Karen Miller )
2011-Dec-27 14:24   Check-in [28924]: A lot of minor edits and LaTeX changes to documentation about the new Drained state and the condor_defrag daemon (and associated ClassAd attributes and configuration variables). ===GT=== #2330 (By Karen Miller )
2011-Nov-30 13:56   Check-in [28504]: Fixed typos in defrag documentation. #2330 (By Dan Bradley )
2011-Nov-21 16:09   Check-in [28379]: editing changes associated with new condor_defrag work ===GT=== #2330 (By Karen Miller )
2011-Nov-21 14:58   Check-in [28376]: Added missing defragad.tex #2330 (By Dan Bradley )
2011-Nov-16 16:34   Check-in [28283]: Documented state and activity transitions relating to draining. #2330 (By Dan Bradley )
2011-Nov-16 14:34   Check-in [28279]: new diagram for machine activity/state transitions, given new Drained state ===GT=== #2330 (By Karen Miller )
2011-Nov-15 18:24   Check-in [28268]: Documented LastDrainStartTime. #2330 (By Dan Bradley )
2011-Nov-15 18:24   Check-in [28269]: Documented effect of suspension during draining on completion estimate. #2330 (By Dan Bradley )
2011-Nov-15 18:24   Check-in [28267]: Added LastDrainStartTime. #2330 (By Dan Bradley )
2011-Nov-15 18:05   Check-in [28266]: Documented state transitions related to the new Drained state. #2330 (By Dan Bradley )
2011-Nov-15 17:51   Check-in [28265]: Documented condor_defrag. #2330 Added section to give high-level view and configuration examples. Added section describing attributes in the daemon ad. (By Dan Bradley )
2011-Nov-15 13:54   Check-in [28263]: Added daemon ad for condor_defrag. #2330 Added some draining-related stats to this daemon ad. (By Dan Bradley )
2011-Nov-14 12:22   Check-in [28242]: Document updated condor_defrag defaults. #2330 (By Dan Bradley )
2011-Nov-14 12:22   Check-in [28241]: Filter out offline ads in condor_defrag. #2330 (By Dan Bradley )
2011-Nov-14 12:02   Check-in [28239]: forgot to commit .dia format version of diagram. ===GT=== #2330 (By Karen Miller )
2011-Nov-14 11:55   Check-in [28238]: Redraw new states diagram for figure 3.3. Remove old, and useless versions of the diagram. ===GT=== #2330 (By Karen Miller )
2011-Nov-11 17:26   Check-in [28228]: Documented condor_defrag. #2330 (By Dan Bradley )
2011-Nov-11 17:26   Check-in [28229]: Improved defaults and fixed a bug in condor_defrag. #2330 (By Dan Bradley )
2011-Nov-11 10:06   Check-in [28221]: Improve comments in defrag test code to make it easier to understand. #2330 (By Dan Bradley )
2011-Nov-11 09:54   Check-in [28220]: Fix defrag test for windows. #2330 It is simpler to use multiple personal condors than to safely pack multiple startds into one personal condor. (By Dan Bradley )
2011-Nov-10 19:42   Check-in [28216]: Added condor_defrag. #2330 (By Dan Bradley )
2011-Nov-10 19:42   Check-in [28217]: Added test of condor_defrag. #2330 (By Dan Bradley )
2011-Nov-09 11:11   Check-in [28189]: Documented new draining-related machine ad attributes. #2330 (By Dan Bradley )
2011-Nov-09 10:47   Check-in [28188]: Improvement to condor_drain docs. #2330 (By Dan Bradley )
2011-Nov-08 17:57   Check-in [28183]: Documented condor_drain. ===GT=== #2330 ===VersionHistory:Complete=== (By Dan Bradley )
2011-Nov-08 12:14   Check-in [28168]: fix typo ===GT=== #2330 (By Karen Miller )
2011-Nov-08 10:40   Check-in [28164]: Added cmd_drain test to list of windows tests. #2330 (By Dan Bradley )
2011-Nov-07 16:54   Check-in [28162]: Added test of condor_drain. #2330 (By Dan Bradley )
2011-Nov-07 16:54   Check-in [28160]: Documented Drained state. #2330 (By Dan Bradley )
2011-Nov-07 16:54   Check-in [28161]: Added support for draining the startd. #2330 (By Dan Bradley )
2011-Oct-17 19:00   Check-in [27871]: Added MachineMaxVacateTime and JobMaxVacateTime and WantGracefulRemoval. #2536 (parent ticket #2330)  [...] (By Dan Bradley )