URF Reviewer Guidelines

Description

The University Research Foundation is a campus program dedicated to fulfilling the mission of the Penn Compact. The Penn Compact motivates community members to innovate, be radically inclusive, and positively impact their local, national, and global communities.

All grants are reviewed by volunteers who serve on the University Research Foundation (URF) review panel. To both the new and returning members, your time and effort is sincerely appreciated.

Evaluation Criteria

Reviewers should evaluate the proposals as rigorously as they would if reviewing for an external sponsor.

Your score for each application should be based upon Significance, Investigator(s), Innovation, Approach, and Environment.

Other Evaluation criteria:

• Scholarly merit, creativity, impact, and innovation
• Investigator’s career trajectory
• Feasibility
• Significance of the research
• Prospects for future extramural funding
• Matching support from other sources
• Career development at early stages of career
• Advancement of school or institutional objectives, such as interdisciplinary research
• Overall impact to investigator’s research program
• Undergraduate participation

RODG: A major consideration for the Opportunity Development Grants will be alignment with the specific goals of the grant program.

Scoring

Scoring is a scale of 1-9.

Critiques

Please provide brief but constructive critiques.  Do not include funding recommendations.

All information you put in the “critique/summary” text area will be visible by your fellow panel members.  Additionally, it will be available to the applicant.  Reviewers will remain anonymous to the applicant, and the score will not be visible to the applicant.

We would appreciate any constructive feedback for the applicants in the event that they would like to submit the proposals to external sponsors for further consideration. In addition, feedback would be helpful to applicants whose proposals are unfunded and would like to resubmit to the next URF cycle.

Additional Biomed Processes

• Scores should be spread over the entire range (1-9).
• Reviewers should “spread” their scores, using the full range of the scoring system (1-9). One suggested approach is that each application may start as a “5”, a good application with medium impact, and then move up or down after the reviewer has reviewed it. It is most likely that each reviewer will be assigned one application which ends up scoring in the higher range (1-3) and one or more in the middle (4-6) or lower (7-9) ranges, although the distribution may vary. No two applications should be assigned the same score by the same reviewer in the initial review.
• Only those proposals above a certain threshold will be discussed at the meeting.
• Proposals that rank below the threshold will not be discussed at the meeting. In advance of the scheduled meetings, the URF administrative staff will notify the committee members of the list of proposals that will be discussed at the meeting. A list of proposals that are below the threshold (being triaged) will also be provided to the committee. Unless an individual reviewer has a valid and convincing reason to elevate a case that does not meet the threshold, only proposals above the threshold will be brought to discussion and ranked by the entire panel.
• At the review meeting, the primary reviewer will provide a very brief overview of the significance, investigator, innovation, approach, and other factors. After the primary reviewer provides these score-driving factors, the secondary reviewer should discuss distinct score-driving factors or issues which justified their providing a higher or lower score. There will then be a general discussion of the application, including budget recommendations, current support, and regulatory considerations (if any). All committee members will then enter their scores on-line. The applications will then be ranked based on the average scores.
• A sample critique can be found here.
• Applicants will be able to read the critiques, however, they will not be able to identify who has written them. Comments should be brief but constructive in order to maximize the value of the review process for the applicants.
• Final score will be polled (ALL members of the panel will score the grant in the online URF system).
• NOTE: ALL PRESENT AT THE DISCUSSION WILL VOTE SO THAT THE TWO REVIEWERS (PRIMARY AND SECONDARY) WILL NOT BE THE SOLE DECIDERS OF THE DISPOSITION OF THE GRANT. Reviewers will be asked to consider the full range of scores, allowing the funder to identify the most meritorious applications amongst the group of strong applications expected from the Penn faculty.  Using the full scoring range makes less frequent instances of grants having identical scores, leading to more difficult funding decisions based on lower-fidelity scoring. Similarly, a reviewer who scores all of their assigned applications or discussed applications with the same score (high, medium, or low) has less of an impact on the ultimate decision of which grants are funded.

Reviewer Eligibility

Reviewers with a conflict of interest on a specific grant must identify themselves and abstain from completing an individual review and participating in the panelist discussion and scoring of that grant. A conflict of interest is defined by the presence of a secondary interest (e.g., being a friend or competitor) which could impact on the primary interest (scoring the proposals based on the investigator and the project ). Conflict of interest  is a circumstance, rather than an action. Examples would include being a mentor or close collaborator of the applicant or being based in the same research group, academic division, or department. This may differ for a large department vs a small department. Potential conflicts should be discussed with the chairs in advance of the study section or at the study section before discussion of an application.

Applicants for the current cycle may not serve on the review panel.  Previous applicants and awardees are still eligible to serve on the review panel.

Online Review System

Please use our online system (SurveyMonkey) to review the proposals.  You can download the applications; however, they are easier to read online.

There is an option to save your drafted critique as you work on them within the system.  Mark it as “complete” when you are done.

You can download a 2-page “How-To Quick Guide” here. It explains how to login (with Pennkey), see which grants to evaluate, review and score an application, add download the proposals.