The Intelligence Advanced Research Projects Activity (IARPA) is seeking information on automated methods to detect, quantify, and correct model drift. This request for information (RFI) is issued solely for information gathering and planning purposes; this RFI does not constitute a formal solicitation for proposals. The following sections of this announcement contain details of the scope of technical efforts of interest, along with instructions for the submission of responses.
Background & Scope
IARPA develops technologies to forecast a broad set of events relevant to national security. Once these technologies are deployed in an operational environment, the Intelligence Community needs automated methods to detect model drift, including: • When algorithms require re-training (e.g. to include more historical data or significant recent events), or • When the assumptions (overt or implied) underlying a model do not apply (e.g. mobility assumptions underlying an epidemiological model), or • When the input data are significantly different from the data on which the model was trained (e.g. applying a model trained on one part of the world to forecast events in another part of the world), or • When the context in which the events are being forecasted has changed significantly (e.g. forecasting societal events before and after a coup), or, more generally, • When confidence in the output of a model has been diminished enough to require model retraining, updating, forking for specific conditions, or even withholding.
When model drift is detected, it is important to determine whether correction is required. Correction can be resource-intensive, so must be done only when needed. For example, a small degradation of performance may be acceptable to some users and applications. This is a risk-management problem that requires quantifying the drift. If model drift causes decrements in performance, a model may need to be corrected, through retraining, updating, or complete rebuilding. The purpose of this RFI is to identify existing automated methods to detect, quantify, and correct model drift. These methods will likely include an efficient instrumentation of models, and computational techniques to process the data generated by that instrumentation. This RFI also seeks to identify innovative approaches to long-term model maintenance that would advance the state-of-the-art and improve greatly the long-term viability of technologies transitioned by IARPA.
Responses to this RFI should answer any or all of the following questions: 1) What are existing automated methods for detecting model drift? At what frequency and with what protocol are these methods applied? Please provide instantiations of such methods for specific models or domains. 2) What are existing automated methods for quantifying model drift? Please provide instantiations of such methods for specific models or domains. 3) What are existing automated methods for correcting model drift? Please provide instantiations of such methods for specific models or domains. 4) Can these existing automated methods be applied to systems by measuring the input/output characteristics, or do systems that employ these methods require instruments built into the system itself? 5) What are metrics and protocols to assess the performance of such methods? 6) What novel approaches to long-term model maintenance do you propose that would advance the state-of-the-art as described in your answers to questions 1-3? 7) What are the key, cross-domain challenges or limiting factors in the development of methods to detect, quantify, and correct model drift?
Preparation Instructions to Respondents
IARPA requests that respondents submit ideas related to this topic for use by the Government in formulating a potential program. IARPA requests that submittals briefly and clearly describe the potential approach or concept, outline critical technical issues/obstacles, describe how the approach may address those issues/obstacles and comment on the expected performance and robustness of the proposed approach. If appropriate, respondents may also choose to provide a non-proprietary rough order of magnitude (ROM) regarding what such approaches might require in terms of funding and other resources for one or more years. This announcement contains all of the information required to submit a response. No additional forms, kits, or other materials are needed.
IARPA appreciates responses from all capable and qualified sources from within and outside of the US. Because IARPA is interested in an integrated approach, responses from teams with complementary areas of expertise are encouraged.
Responses have the following formatting requirements:
1. A one page cover sheet that identifies the title, organization(s), respondent's technical and administrative points of contact - including names, addresses, phone and fax numbers, and email addresses of all co-authors, and clearly indicating its association with RFI-14-02; 2. A substantive, focused, one-half page executive summary; 3. A description (limited to 5 pages in minimum 12 point Times New Roman font, appropriate for single-sided, single-spaced 8.5 by 11 inch paper, with 1-inch margins) of the technical challenges and suggested approach(es); 4. A list of citations (any significant claims or reports of success must be accompanied by citations, and reference material MUST be attached); 5. Optionally, a single overview briefing chart graphically depicting the key ideas.
Submission Instructions to Respondents
Responses to this RFI are due no later than 4:00pm, Local Time, College Park, MD on March 14, 2014. All submissions must be electronically submitted to email@example.com as a PDF document. Inquiries to this RFI must be submitted to firstname.lastname@example.org. Do not send questions with proprietary content. No telephone inquiries will be accepted.
DISCLAIMERS AND IMPORTANT NOTES
This is an RFI issued solely for information and planning purposes and does not constitute a solicitation. Respondents are advised that IARPA is under no obligation to acknowledge receipt of the information received, or provide feedback to respondents with respect to any information submitted under this RFI.
Responses to this notice are not offers and cannot be accepted by the Government to form a binding contract. Respondents are solely responsible for all expenses associated with responding to this RFI. IARPA will not provide reimbursement for costs incurred in responding to this RFI. It is the respondent's responsibility to ensure that the submitted material has been approved for public release by the information owner.
The Government does not intend to award a contract on the basis of this RFI or to otherwise pay for the information solicited, nor is the Government obligated to issue a solicitation based on responses received. Neither proprietary nor classified concepts or information should be included in the submittal. Input on technical aspects of the responses may be solicited by IARPA from non-Government consultants/experts who are bound by appropriate non-disclosure requirements.
Contracting Office Address:
Office of the Director of National Intelligence Intelligence Advanced Research Projects Activity Washington, District of Columbia 20511 United States
Primary Point of Contact:
Dewey Murdick Intelligence Advanced Research Projects Activity email@example.com