Developing Common Metrics for Clinical and Translational Awards

ann dozier

Dr. Ann Dozier, Lead Evaluator for the University of Rochester Clinical and Translational Science Institute (CTSI).

Dr. Ann Dozier, lead evaluator for the University of Rochester Clinical and Translational Science Institute, contributed to a recent publication in the Clinical and Translational Science Journal entitled “Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned.” The paper describes an effort by the national CTSA Evaluation Key Function Committee to develop and test a methodology for identifying common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics.

Dr. Dozier offered this comment: “This project further reinforced the importance of engaging stakeholders (potential end users) in the process of common metric development and how different the CTSAs are in what and how their internal data are gathered and maintained.”

The project involved more than one-fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.

Click here to access a PDF of the article.



Your Feedback from the CTSI Retreat

The CTSI’s latest retreat, held on November 11, concentrated on the theme of “renewing our strategic focus; preparing for action.” Participants were generally people with formal roles in the CTSI and effort support.

First of all, thanks to everyone who took part. We invited 85 people, 70 responded that they would come, and all but two showed up. This, and all the written feedback we received, demonstrates real commitment, and we thank you for it.

I have a thick stack of paper on my desk which contains input from many of you on the questions we asked you to think about. Here’s a review and summary of your input:

We asked retreat participants to provide their thoughts on concepts, ideas or just words that should be included in statements of CTSI mission, vision and values. I read each and every written response provided, whether we got it before, during or after the retreat. I actually read each one several times (and special thanks to all of you who listened to your 3rd grade teachers regarding good handwriting).

I’ve created three summaries of your input. For the first, I reviewed input regarding the CTSI mission statement. I recorded each suggested concept, and counted up the number of times that concept appeared across all of the feedback provided. Then, I grouped the concepts into categories. I admit that this analysis is subjective, but it seems illuminating nonetheless. Here’s the result:

mission concepts

In words:  in your feedback, you mentioned resources that the CTSI provides or could provide 26 times. You mentioned various high level goals that the CTSI could pursue about 24 times. And so on. For all of you data nerds, I’ve provided all the details in a separate post (click here to take a look).

After our new Senior Project Research Associate Dr. Kathleen Holt took a look at this first graph, she noted that I appeared to focus on nouns, and suggested that I go back and take a look at the verbs you used in your input. So I did that, and the graph below was the result:

mission concepts - verbs

Also very interesting perspectives: somewhat different than the first look, but similar in some ways as well. Again, you’ll find the details in a separate post (here).

Finally, I repeated the process with the concepts, ideas and words you suggested should be part of a CTSI statement of values. Here’s the result:


If you’d like to review the full compilation of responses, click here.

So – what do you think? You can use the “leave a comment” feature to share your thoughts, or email me directly.

What’s next? Over the next month or so the Operations Committee will review this information and their own thoughts, and work toward new or revised statements of mission, vision and values for the CTSI. But this is only the beginning of the strategic planning process. After mission, vision and values comes the development of a customer value proposition, strategic themes and results, objectives, related performance measures and targets, strategic initiatives and execution. You’ll be hearing more!

Thanks for your support for the CTSI!

For data nerds: retreat feedback details

In a separate post (here), I provide a summary of your feedback from the November 11, 2014 CTSI retreat. This post contains details, for those of you that want to ponder the issues a bit more deeply.

The table below summarizes feedback regarding a CTSI mission statement. The first column is the category I assigned; the second is the concept or idea as extracted from your feedback, and the third columns shows how many times that concept was mentioned across all feedback received.

Mission Concepts – Details

Category Concept Mentions
Collaboration connection 8
Collaboration team science 6
Collaboration collaboration 4
Collaboration multidisciplinary 3
Collaboration synergy 2
Community community engagement 6
Community integration 3
Community outreach 2
Community community centered 1
Population Health community health 3
Population Health clinic 2
Population Health health 2
Population Health patients 2
Population Health population health 1
Population Health prevention 1
Population Health population 1
Education training/education 12
Education mentoring 2
Education knowledge 2
Help facilitate 6
Help help/assistance 4
Help guidance 3
Help enable 2
Help accelerate 2
Help access 2
Help catalyst 1
Resources expertise 7
Resources funding 6
Resources resources 6
Resources service 2
Resources infrastructure 2
Resources solutions 1
Resources tools 1
Resources support 1
Efficiency efficiency 5
Efficiency relevance 2
Efficiency quality 2
Communication reputation 2
Communication awareness 2
Communication communication 2
Communication dissemination 1
Character trust 3
Character compassion 2
Character commitment 1
Goals translation 8
Goals innovation 7
Goals discovery 2
Goals success 2
Goals opportunity 2
Goals science 1
Goals evolution 1
Goals enhance 1

Another way to look at this information is through a Wordle. In the image below, each concept shows up. The size of the concept is proportional to the number of times it was mentioned. Click on the image to get a larger view.

mission wordle 3

The following table is similar to the previous one, but focuses on the verbs used in your feedback to describe the CTSI’s mission.

Mission Concepts (verbs) – Details

Category Concept Mentions
Advance advance translational science 2
Advance advance innovation 1
Advance improve research 1
Advance enhance 1
Advance accelerate 1
Partner establish connections with collaborators 2
Partner enhance culture of team science 1
Partner work with competing health systems 1
Partner work with competitors 1
Partner facilitate connections among basic science/clinical research/community engagement 1
Partner merge population health and team science 1
Connect translate research into clinical 2
Connect link research and clinical missions 1
Connect connect faculty to resources 1
Connect connect 1
Help facilitate 2
Help enable 2
Help fill gaps 1
Help help investigators 1
Help facilitate multidisciplinary researchers 1
Help support 1
Help provide answers to regulatory and technical questions 1
Help provide funds/resources/infrastructure 1
Train train 1
Train support new generation of scientists 1

A Wordle would not be terribly helpful in this case, since all the concepts only showed up once or twice.

This table shows the details regarding values.

Values – Details

Category Concept Mentions
Collaboration collaboration 13
Collaboration team work 6
Collaboration team science 5
Collaboration connection 2
Collaboration partnerships 1
Collaboration multidisciplinarity 2
Collaboration interdisciplinarity 2
Collaboration transdisciplinarity 1
Community community engagement 4
Community community 3
Community health disparities 3
Community health 3
Community translation to community 2
Community practical application 1
Efficiency efficiency 3
Efficiency excellence 3
Efficiency sustainability 3
Efficiency productivity 2
Efficiency effectiveness 1
Efficiency return on investment 1
Equity inclusive 2
Equity diversity 2
Equity accessible 1
Meliora ever better 5
Meliora innovation 4
Meliora passion 3
Meliora synergy 3
Meliora meliora 1
Meliora persistence 1
Meliora catalyst 1
Meliora leadership 1
Meliora opportunity 1
Meliora evolution 1
Education education 3
Education mentoring 1
I-CARE and other values integrity 7
I-CARE and other values respect 2
I-CARE and other values accountability 1
I-CARE and other values research 1
I-CARE and other values incubator 1
I-CARE and other values ethical 1
I-CARE and other values I-CARE 3

And here’s the associated Wordle:

values wordle 2

Why are you harassing me Nicole O’Dell?

By Nicole L. O’Dell, MLS31739_1507832462226_2941135_n

You see my name in your inbox, and you think to yourself, “Not her again!!!! What does she want now?!?!” I understand your pain, but I admit it: I am a harasser. That is my job. As the Senior Information Analyst for the Clinical and Translational Science Institute, I am charged with the daunting task of collecting evaluation data for a multi-million dollar Federal grant. It is not easy, and it requires information to be collected from many different sources, most likely including you.

Don’t despair! If I contact you for information or to complete a survey there is probably a GREAT reason behind it. spam0607Here is a list of the most common reasons that I will be in your inbox:

  1. You’re a faculty member: Congratulations!! All that hard work and schooling paid off and now you are officially a member of the faculty at highly respected University. The CTSI highly regards the opinions of the University of Rochester faculty and often surveys them for awareness of and satisfaction with our services and programs.
  2. You are a funded investigator: Congratulations!! You have successfully obtained a grant, which is not easy feat! The CTSI is interested in the research climate at the University and periodically collects data from you about your projects and research collaborations.
  3. You received funding from the CTSI:  Congratulations!! You were picked out of a group of very strong candidates to receive support through our pilot funding or educational programs! The CTSI surveys awardees and trainees twice per year about outcomes that resulted from the support and any success stories you would like to share.
  4. You utilized CTSI programs or services: The CTSI offers a vast array of services and programs that assist investigators at the University in conducting clinical and translational science. Our offerings include expert consultation services, the Clinical Research Center, the Center for Community Health, the Research Navigator Program, and others. The CTSI collects information about your use of these services and the benefits that result, such as publications and grants.

In closing, I would like to say that I am a human being, not an evil robot. I am actually quite nice in real life. If you would like me to stop harassing you, the solution is quite simple. Just send me the data.  And thank you for helping us improve the CTSI’s programs and services and show that we’re using federal funds wisely.