This is a static copy of the ActEV Prize Challenge page as of 21 December 2018. The authoritative website is offline during the US Federal Government shutdown; once it is active again, links to this static copy will revert to the authoritative site.

Activities in Extended Videos Prize Challenge sponsored by IARPA



Overview

Activities in Extended Videos Prize Challenge (ActEV-PC)

Activity Annotated Video (4x speed and bounding boxes added for clarity).
Summary
The Activities in Extended Videos Prize Challenge (ActEV-PC) seeks to encourage the development of robust automatic activity detection algorithms for an extended video. ActEV-PC is operated by NIST and sponsored by IARPA. Challenge participants will develop algorithms that address the "Activity Detection Task" which requires systems to detect and temporally localize the activity instance for 18 activities that are to be found in extended videos. Extended videos contain significant spans without any activities and intervals with potentially multiple concurrent activities. ActEV-PC will contain two phases - the ActEV-PC Open Leaderboard evaluation (Phase 1) and the ActEV-PC Independent evaluation (Phase 2).

ActEV-PC Open Leaderboard Evaluation (Phase 1)
For the ActEV-PC Open Leaderboard evaluation, challenge participants will run their activity detection software on their compute hardware and submit system output defined by the Evaluation Plan to the NIST ActEV Scoring Server. This phase will serve as a qualifying stage where the top 8 participants will proceed to phase 2.

ActEV-PC Independent Evaluation (Phase 2)
For the ActEV-PC Independent evaluation, invited challenge participants will submit their runnable activity detection software to NIST using the forthcoming Evaluation Commandline Interface Submission Instructions. NIST will then evaluate system performance on sequestered data using NIST hardware.

Prizes
The developers of the most accurate activity detection algorithms will be eligible to win one of three cash prizes from a total prize purse of $50,000 (USD).
  • First Place Prize - $25,000
  • Second Place Prize - $15,000
  • Third Place Prize - $10,000

To receive the full award payment, each winning team must send at least one representative to CVPR'19 ActivityNet workshop for the challenge results announcement and present the winning team's results at the workshop. The challenge will be one of the tasks of CVPR'19 ActivityNet workshop .

See the complete eligibility requirements in Evaluation and Rules tab.

What Activities?
For ActEV, an activity is defined to be "one or more people performing a specified movement or interacting with an object or group of objects". The activities that systems are required to detect in this challenge are defined in on the Activities tab and Section 2.5 of the Evaluation Plan

How to Participate
To take part in the ActEV-PC evaluation you need to click link currently down https://actev.nist.gov/users/sign_up Register above and then acknowledge that you have read and accepted the terms and conditions from the ActEV-PC Evaluation Plan and the VIRAT data license PDF, by clicking on the two checkboxes to agree.

NIST will then send you information on how to download data. After signing in (using the link above) with your registered account, you will be able to upload your JSON formatted results (see Section 5 in the Evaluation Plan for format details) to the ActEV Scoring Server using the "Instructions" link (available on this web site after Sign In). The challenge will then proceed with the selection of 8 participants, who will deliver their activity detection algorithm to NIST.

Judging Criteria and Metrics
The main scoring metrics will be based activity detection using the weighted probability of missed detection at 0.15 false alarms per minute averaged over activity types. A complete description of the metrics is provided in the Evaluation Plan. NIST also provides the Scoring software for ActEV-PC on GitHub.

Task coordinator
ActEV NIST team (actev-nist@nist.gov )

Calendar
12Dec
Start of the ActEV-PC Open Leaderboard evaluation
28Feb
End of the ActEV-PC Open Leaderboard evaluation.
02Mar
Start of ActEV-PC Independent evaluation
15May
End of ActEV-PC Independent evaluation

Schedule

Activities Detection in Videos Prize Challenge (ActEV-PC) Schedule

Phase 1 - ActEV-PC Open Leaderboard evaluation:

Nov 08, 2018: link currently down https://actev.nist.gov/users/sign_up Account registration opens, evaluation plan released and encrypted evaluation data available for download
Nov 12, 2018: Evaluation data unlocked (decryption key published)
Dec 12, 2018: Leaderboard open for submissions
Feb, 28, 2019: Top eight teams on the Leaderboard selected for participation in Phase 2

Phase 2 - ActEV-PC Independent evaluation:

March, 01, 2019: Challenge participants selected for Phase 2 deliver software conforming to the ActEV Evaluation CLI
March, 02, 2019: NIST evaluates challenge participant's code performance on sequestered data
May, 15, 2019: NIST reports results to IARPA
May, 30, 2019: Challenge winners announced
June, 2x, 2019: Challenge Workshop/Presentation of winning results (tentatively the ActivityNet workshop at CVPR'19)

Data

Data Licensing for the ActEV Prize Challenge Evaluation
New ActEV Teams:
Data Website URL: (requires a NIST-provided user and password)

Existing ActEV Participants:


Leaderboard

ActEV-PC Leaderboard for Phase 1

PC Leaderboard

Updated: 2018-12-21T18:54:01+00:00
RANK SUBMISSION_ID EVALUATION_NAME TRACK_NAME TEAM_NAME SYSTEM_NAME METRIC_NAME METRIC_VALUE
1 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00189_STARK_20181221-134529-1523.sr-20181221-134529-2194 ActEV-2018 AD STARK k1 mean-w_p_miss@0.15rfa 0.757797169525
2 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00162_UCF_20181217-110256-6113.sr-20181217-110256-6485 ActEV-2018 AD UCF UCF-PC mean-w_p_miss@0.15rfa 0.7925697933220001
3 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00188_MUDSML_20181219-015959-3718.sr-20181219-015959-4227 ActEV-2018 AD MUDSML MUDSML_MAIN mean-w_p_miss@0.15rfa 0.800768761454
4 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00185_INF_20181217-193445-6747.sr-20181217-193445-7336 ActEV-2018 AD INF tst mean-w_p_miss@0.15rfa 0.809781472746
5 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00170_SRI_20181217-094621-1223.sr-20181217-094621-1972 ActEV-2018 AD SRI TardisVP mean-w_p_miss@0.15rfa 0.836086777577
6 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00186_INF_20181217-201008-7475.sr-20181217-201008-8019 ActEV-2018 AD INF tst2 mean-w_p_miss@0.15rfa 0.8584239656820001
7 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00168_JHUDIVATeam_20181214-181646-1458.sr-20181214-181646-3054 ActEV-2018 AD JHUDIVATeam Structured_Genralization_AD mean-w_p_miss@0.15rfa 0.8642026560780001
8 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00169_JHUDIVATeam_20181214-181827-2859.sr-20181214-181827-3922 ActEV-2018 AD JHUDIVATeam CLI_Indepedent_Eval mean-w_p_miss@0.15rfa 0.8723423127590001
9 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00171_SRI_20181217-103113-8180.sr-20181217-103114-0595 ActEV-2018 AD SRI TardisVS mean-w_p_miss@0.15rfa 0.8768933866609999
10 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00166_DIVA-TE-Baseline_20181213-083956-4174.sr-20181213-083956-4501 ActEV-2018 AD DIVA TE Baseline baselineACT_1_AD mean-w_p_miss@0.15rfa 0.9072321154299999
11 ActEV-2018_AD_Prize-Challenge-Leader_SYS-00133_DIVA-TE-Baseline_20181212-084002-2087.sr-20181212-084002-2555 ActEV-2018 AD DIVA TE Baseline baselineRC3D_1_AD mean-w_p_miss@0.15rfa 0.9129740244110001

Activities

Activities and the Activity Detection Task
An ActEV activity is defined to be "one or more people performing a specified movement or interacting with an object or group of objects". Activities are annotated by humans using a set of annotation guidelines that specify how to perform the annotation and the criteria to determine if the activity occurred. Each activity is formally defined by five elements:

For example:


Activity name Description and Example Chip Videos
Closing
  • Description: A person closing the door to a vehicle or facility.
  • Start: The event begins 1 s before the door starts to move.
  • End: The event ends after the door stops moving. People in cars who close the car door from within is a closing event if you can still see the person within the car. If the person is not visible once they are in the car, then the closing should not be annotated as an event.
  • Objects associated with the activity : Person; and Door or Vehicle

Vehicle_turning_left
  • Description: A vehicle turning left or right is determined from the POV of the driver of the vehicle. The vehicle may not stop for more than 10 s during the turn.
  • Start: Annotation begins 1 s before vehicle has noticeably changed direction.
  • End: Annotation ends 1 s after the vehicle is no longer changing direction and linear motion has resumed. Note: This event is determined after a reasonable interpretation of the video.
  • Objects associated with the activity : Vehicle

Loading
  • Description: An object moving from person to vehicle.
  • Start: The event begins 2 s before the cargo to be loaded is extended toward the vehicle (i.e., before a person's posture changes from one of "carrying" to one of "loading").
  • End: The event ends after the cargo is placed into the vehicle and the person-cargo contact is lost. In the event of occlusion, it ends when the loss of contact is visible.
  • Objects associated with the activity: Person; and Vehicle

List of Activities

The target activities are listed in the table below. The detailed definitions are in the Evaluation Plan.
List of Activities
Closing
Closing_trunk
Entering
Exiting
Loading
Open_Trunk
Opening
Transport_HeavyCarry
Unloading
Vehicle_turning_left
Vehicle_turning_right
Vehicle_u_turn
Pull
Riding
Talking
activity_carrying
specialized_talking_phone
specialized_texting_phone

The Activity Detection (AD) task

Given a target activity, a system automatically detects and temporally localizes all instances of the activity. For a system-identified activity instance to be evaluated as correct, the type of activity must be correct and the temporal overlap must fall within a minimal requirement as described in the Evaluation Plan

Rules

ActEV-PC Evaluation Plan and Participation Rules

Evaluation Plan
The Evaluation Plan describes the evaluation task, metrics, and scoring protocols, and system inputs and outputs.
Who is Eligible to Participate?
To be eligible to win a prize under this competition, an individual or entity
  1. Must have completed and submitted a registration form on link currently down https://actev.nist.gov/prizechallenge prizechallenge
  2. Must meet any other account creation or registration requirements, such as creating an account on challenge.gov, etc.;
  3. Must have complied with all the requirements stated in these Rules and Terms/Conditions;
  4. Must agree to abide by the decisions of ODNI, NIST, and/or the individual judges, which shall be final and binding in all respects;
  5. Must agree to follow all applicable local, state, federal and country of residence laws and regulations.
  6. Must be (1) an individual or team of individuals each of whom are 18 years of age and over, or (2) a for-profit or non-profit entity organized or incorporated under law;
  7. May not be a federal entity or federal employee acting within the scope of their employment;
  8. Shall not be deemed ineligible because the individual or entity used Federal facilities or consulted with federal employees during a competition if the facilities and employees are made available to all individuals and entities participating in the competition on an equitable basis;
  9. In the case of federal grantees may not use federal funds to develop challenge applications unless consistent with the purpose of their grant award;
  10. In the case of federal contractors, may not use federal funds from a contract to develop challenge applications or to fund efforts in support of a challenge submission;
  11. May not be employees of NIST or ODNI and/or any other individual or entity associated with the development, evaluation, or administration of the competition, as well as members of such persons’ immediate families (spouses, children, siblings, parents), and persons living in the same household as such persons, whether or not related, are not eligible to participate in the competition;
  12. May not be prime contractors or subcontractors and their employees of the IARPA DIVA program, due to its similarity to the ActEV prize challenge (however, they may still register under the “forgoing prizes” option, in order to compete and have their solutions posted on the leaderboard without the possibility of a cash prize); and
  13. Must not be currently on the Excluded Parties List ( link currently down https://www.epls.gov).

Warranties
By submitting an Entry, you represent and warrant that all information you submit is true and complete to the best of your knowledge, that you have the right and authority to submit the Entry on your own behalf or on behalf of the persons and entities that you specify within the Entry, and that your Entry (both the information and software submitted in the Entry and the underlying technologies or concepts described in the Entry):
  1. Is your own original work, or is submitted by permission with full and proper credit given within your Entry;
  2. does not contain confidential information or trade secrets (yours or anyone else's);
  3. does not knowingly, after due inquiry (including, by way of example only and without limitation, reviewing the records of the United States Patent and Trademark Office and inquiring of any employees and other professionals retained with respect to such matters), violate or infringe upon the patent rights, industrial design rights, copyrights, trademarks, rights in technical data, rights of privacy, publicity or other intellectual property or other rights of any person or entity;
  4. does not contain malicious code, such as viruses, malware, timebombs, cancelbots, worms, Trojan horses or other potentially harmful programs or other material or information;
  5. does not and will not violate any applicable law, statute, ordinance, rule or regulation, including, without limitation, United States export laws and regulations, including, but not limited to, the International Traffic in Arms Regulations and the Department of Commerce Export Regulations; and
  6. does not trigger any reporting or royalty or other obligation to any third party.
If the Submission includes any third party works (such as third party content, equipment, or open source code), entrant must be able to provide, upon the request of ODNI or NIST, documentation of all appropriate licenses and releases for such third party works. If entrant cannot provide documentation of all required licenses and releases, ODNI and NIST reserves the right to disqualify the applicable Submission, or seek to secure the licenses and releases for the benefit of ODNI and NIST, and allow the applicable Submission to remain in the Competition. ODNI and NIST also reserve all rights with respect to claims based on any damages caused by participant's failure to obtain such licenses and releases.

Limitation of Liability
By participating in the Competition, you agree to assume any and all risks and to release, indemnify and hold harmless ODNI, NIST, and Engility Corporation, each of the Judges, and Subject Matter Experts, from and against any injuries, losses, damages, claims, actions and any liability of any kind (including attorneys' fees) resulting from or arising out of your participation in, association with or submission to the Competition (including any claims alleging that your Entry infringes, misappropriates or violates any third party's intellectual property rights). Participant agrees that it will not seek compensation for any equipment, materials, supplies, information, travel, labor and/or other Participant-provided services. In addition, you agree to waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from your participation in this Competition, whether the injury, death, damage, or loss arises through negligence or otherwise. Entrants are not required to obtain liability insurance or demonstrate financial responsibility in order to participate in the competition. ODNI, NIST, and Engility Corporation are not responsible for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete or late Entries. ODNI and NIST are not responsible for:
  1. Any incorrect or inaccurate information, whether caused by a Participant, printing errors, or by any of the equipment or programming associated with or used in the Competition;
  2. unauthorized human intervention in any part of the Entry process for the Competition;
  3. technical or human error that may occur in the administration of the Competition or the processing of Entries; or
  4. any injury or damage to persons or property that may be caused, directly or indirectly, in whole or in part, from a Participant's participation in the Competition or receipt or use or misuse of an Award.

If for any reason an Entry is confirmed to have been deleted erroneously, lost, or otherwise destroyed or corrupted, the Participant’s sole remedy is to submit another Entry in the Competition.

Additional Information
These rules cannot be modified except by ODNI or NIST. All decisions by ODNI or NIST regarding adherence to these rules are final. The invalidity or unenforceability of any provision of these rules shall not affect the validity or enforceability of any other provision. In the event that any provision is determined to be invalid or otherwise unenforceable or illegal, these rules shall otherwise remain in effect and shall be construed in accordance with their terms as if the invalid or illegal provision were not contained herein. ODNI and NIST reserve the right in their sole discretion to amend these rules throughout the duration of the contest should extenuating circumstances arise, to extend or modify the dates of the Competition, and to change the terms set forth herein governing any phases taking place after the effective date of any such change.

Payment Terms
Prize payments will be provided by Engility Corporation. Prize winners need to submit a W-9 tax form, or a W8-BEN form in order to receive payment. Participants are responsible for all taxes incurred from the acceptance of Prize funds. No payments will be made to participants where US sanctions preclude it.

Cancelation

IARPA reserves the right to cancel the ActEV prize challenge due to low participation or indications from early results that insufficient progress is being make toward challenge goals. It is anticipated that at least five participants will be necessary to award any prize challenge payment.

Intellectual Property

Each entrant retains full ownership and title in and to their submission and expressly reserve all intellectual property rights not expressly granted under the challenge agreement. By participating in the challenge, each entrant hereby irrevocably grants to the Office of the Director of National Intelligence (ODNI ), of which IARPA is a component, and NIST a limited, non-exclusive, royalty-free, worldwide license and right to reproduce, publically perform, publically display, and use the submission for internal ODNI and NIST business and to the extent necessary to administer the challenge. By submitting an Entry, you grant a non-exclusive right and license to ODNI and NIST to use your name, likeness, biographical information, image, any other personal data submitted with your Entry and the contents in your Entry (including any created works, such as YouTube® videos, but not including any App software submitted with or as part of your Entry), in connection with the Competition. You also agree that this license is perpetual and irrevocable.

You agree that nothing in this Notice grants you a right or license to use any names or logos of ODNI, IARPA, or NIST or any other intellectual property or proprietary rights of ODNI, IARPA, and NIST. You grant to ODNI and NIST the right to include your company or institution name and logo (if your Entry is from a company or institution) as a Participant on the Event Web site and in materials from IARPA announcing winners of or Participants in the Competition. Other than these uses or as otherwise set forth herein, you are not granting ODNI or NIST any rights to your trademarks.

ODNI and NIST will retain and use the system for a period of no longer than two years after submission or when the Participant ceases to participate in ActEV evaluations, whichever is later.

Any data or documentation that qualifies as business proprietary information, as defined by the Freedom of Information Act (5 USC Section 552), and is properly marked as such will be treated as confidential and will only be used for the purposes of the ActEV test.




If you have any question, please send an email to actev-nist@nist.gov


Updates

ActEV-PC Updates

During this Prize Challenge changes and updates will be posted here



This is a static copy of the ActEV Prize Challenge page as of 21 December 2018. The authoritative website is offline during the US Federal Government shutdown; once it is active again, links to this static copy will revert to the authoritative site.