*:Proposal: Add a new machine: "Packager" which makes read-only disk space available. A submitted job is done on the packager. The packager collects all of the input into the archive location, then submits the "real" jobs the NMI submit point. Initially the packager would expose the archive location as NFS or similar to the submit point, otherwise looking like a "normal" build. This needlessly routes data through the submit node, but should work without Metronome changes. As Condor's capability to draw files from web or other inputs grows, individual execute nodes could pull files directly from the packager. *:Todd M notes, "If having Metronome track what's fetched on the execute hosts would actually be helpful (note that we wouldn't want to manage the cache, since that's really Condor's job (Condor's, not Flightworthy), I could probably add a 'remote_fetch' step..." + +*: Another note from Todd M: +*:: - Talked to Nick about long-term reproducibility. Because of compiler + drift, even extraordinary efforts to maintain source code repositories + wouldn't necessarily suffice. Likewise, hardware drift could easily + make it hard if not impossible to install any copy of an old OS we + may have kept. It seems like the best bet in terms of duplicating + old bits is to use virtual machines, archiving them every so often, and + forward-converting (and testing) them at the same time against the + latest version of the virtual machine software. However, we may be + better served by (automatically, machine-readable) maintaining knowledge + about what used to work; then, at some point in the future, we have a + place to start from that we can be certain /used/ to work. Depending + on exactly what the goal is, a plan that quickly adapts old releases + to the present day may be of more use than a plan that quickly cranks + out (presumably patched) binaries for systems that nobody uses with + dependencies that are obsolete. (In fact, it occurs to me as I write + this up that there's no reason Flightworthy couldn't continue to submit + builds and tests for stable releases long after it's stopped working on + them. Even if you don't fix them when the rot into brokenness, at least + you can keep the record of what broke them, and have a good place to + start, later.) + +*: Other than matching up, fetching and building externals, I think the main issue is that Condor is currently stored in a git repository. We have 2 alternatives here: + +1:: Keep a copy of the current git source code and a snapshot of the git repo +1:: Copy the source code of the build out, and modify the glue logic to handle the source tarball. This should be similar to a workspace build, I think.
