Page History
Lark
Introduction
Lark is a NSF-funded project for adding network-awareness to HTCondor's High Throughput Computing approach.
Broadly, it can be split into three major areas:
- Advanced Network Testbed: A small testbed for HTCondor networking technologies. Consists of dedicated HTCondor pools at Wisconsin and Nebraska. This will serve as a "launch point" for Lark technologies onto the production clusters at the sites.
- Network Monitoring: Integrating existing network monitoring tools (particularly, perfSONAR) into the HTCondor ecosystem. This will provide various HTCondor daemons with the ability to make decisions based on the observed network state.
- Network Management: Have HTCondor actively alter the network layer based on its internal policies.
Lark started on October 1 and will last for approximately two years. More information will be added to this wiki page as the project progresses.
Advanced Network Testbed (ANT)
The ANT consists of small HTCondor pools at Nebraska and Wisconsin. These are meant to test Lark technologies and harden existing advanced HTCondor network technologies.
Example ANT use cases include:
- Verification of IPv6 support, especially of the flocking use case.
- Testing of network accounting at several sites.
- Testing dynamic VLAN creation between Nebraska and UW using the technology provided by the DYNES project.
Personnel responsible: Garhan Attebury, Alan DeSmet
Network Monitoring
HTCondor is relatively uninformed of the underlying network conditions. For example, its queuing is based on the number of concurrent file transfers, regardless of whether the underlying network is 100Mbps or 100 Gbps.
The network monitoring task will work on gathering data from the perfSONAR boxes in the DYNES project and pushing them into a HTCondor collector.
From there, the schedd would be able to utilize this information to better adjust its concurrent file transfers.
Personnel responsible: Dan Bradley (schedd changes), Unassigned (perfSONAR integration)
Network Management
Personnel responsible: Zhe Zhang, Brian Bockelman
Development Documentation
Attachments:
- Lark_proposal_scrubbed.pdf 357185 bytes added by bbockelm on 2012-Nov-13 17:06:27 UTC.
A shortened version of the original Lark proposal.
- LarkNSFAnnualReportYr1-scrubbed.pdf 133082 bytes added by tannenba on 2013-Oct-03 17:13:13 UTC.
A shortened version of the Lark Year 1 Annual Report