How risky are different sources of program information?

Introduction to this Series

This series of posts frames its analysis through the unusual lens of risk, rather than the conventional provider-program lens of potential or promises. This series is intended for those responsible for program selection or renewal, primarily at the district level. Your actual use and results happen in your real world, which can be quite different from the ‘lab’ conditions studied. Consider that no matter the ‘idealized’ potential of a program, if there are significant risks to actually achieving that potential for your students and teachers, it’s incumbent on you to become familiar with the risks and if need be, risk tradeoffs and risk mitigation to achieve results.

Related Posts:

  • Lens: Why the lens of “risk”?
  • Implementation: Implementation Risk – is that a thing?
  • Studies : How risky is your replicating results?

This second post lays out some typical channels through which a buyer garners information about an edtech program, and asks questions or provides remarks on aspects of the risks inherent in each of the channels. My goal is to get you to question, more, the applicability, suitability and the comprehensiveness of your information source.

Let’s contextually define information source risk: the chance that any given aspect of a program won’t happen as well as the information source implies.

7 Sources of Program Information: what are some risks?

Source #1: Program marketing
What’s the probability that you don’t realize that glossy experience the program describes itself with? Programs are advertised with many claims and explicit or implied promises of value-added results. What math provider is not going to call out “higher scores” and “better understanding” – and very confidently at that? But there is some risk, too. Does the marketing mention the required implementation levels? Impact at the lowest implementation level? Minimum training needs to get a level of implementation? Applicability and engagement for student subgroups; learning impacts broken out by student subgroup? How much information is in, or left out of, the glossies? Note: you can ask the provider for what’s left out.

Source #2: Your own experience
Drawing from your own experience with program X is true and real world conditions evidence about its use and effectiveness. For any “X” there were very likely some X factors that you liked. First question is for yourself: did you also learn from X some challenges or limitations or downsides? Were any significant enough that they must be addressed for ongoing success at scale, with real world use of program X? A couple of potential limitations to review: insightful enough implementation feedback to reach productive usage at scale across schools, classrooms. Any study of outcomes impact?

Second question for yourself: was your prior situation in using X similar to the scenario you are currently addressing? In grade-levels, demographics, district culture? In related content and instructional practices? If materially different, do any of those differences need to be considered on the way to to predicting the same or better from the program?

Source #3: Friends
Discussing program experiences with colleagues uncovers recommendations and new programs, providing additional perspective on what’s valued by users. It’s seems to me natural to share positives generally, and particularly regarding program decisions personally made. When hearing a program experience, are you asking for sharing of challenges also. A few potential learnings to prod for: robust adoption at scale, success rates in breadth and intensity of usage across classrooms or schools, applicability to all types of students. Are the experiences shared backed up by usage or outcomes reports, vs recalled anecdotes.

Source #4: The Market
What’s this year’s apparent consensus about the new tech, or the new method or approach, or the new feature or product? What’s getting repeated attention in Education Week and EdSurge and District Administration and Tech&Learning; at conferences and tradeshows; in articles and posts and ads and awards? Then: how reliable have those indicators been in your past, in giving you the insightful guidance you needed to turn into the results you’d planned for? Does the annual buzz provide enough types of information to comprehensively connect a program to your specific situation? Do the articles, ads, awards, posts and talks include enough mention of nuances of usage and application at scale, of what it takes in the real world, to obtain what outcomes for whom?

Source #5: Your own team’s research:
Conducting your own thorough research on the program being considered, piloting its actual use by your own people for your own students in your own scenario – a pilot – is the best right? However, very challenging indeed. Pilots are typically too small for statistical power and rigor, and are thus subject to picking desired findings. Program data collection for research, then analysis and modeling and review is complex, deep, time-consuming and expensive, if only in opportunity costs for your researcher and analytical talent. And, I’m here to tell you as a program provider having experienced many arms-length evaluations, there is a 50/50 chance that an external researcher isn’t evaluating any given program’s theory of action correctly; the program data elements, the outcomes measure constructs, and the logic models being used are insufficiently understood. A 50/50 chance of drawing some opposite conclusions from the truth. My advice is to be sure to ask the provider, up front and during, for guidance on how to evaluate.

Source #6: Independent teams’ research
That coin flip chance of errors by external researchers above isn’t limited to district or state users; 3rd party researchers comprise the vast majority of my experience of errors. In other words, it’s not because a SEA or LEA team “isn’t knowledgeable enough” somehow. The 50/50 risk applies to highly experienced university evaluators also. I know from having had to work long and hard to turn around multiple erroneous findings and results. Qualified researchers’ rigorous studies absolutely should play the crucial role in evaluating the risks and rewards associated with educational programs. But again my advice is to be sure to ask the provider, up front and during, for guidance on how to evaluate. For a further unpack of study risks for any researcher to consider, see my Studies Risk blog post.

Source #7: Vendor research
Sometimes the vendor provides their own research. And yes, having spent 2 decades with a data&eval team, I have a lot of experience and a bias towards the schemas I’ve built. That said, my personal opinion is that “vendor” studies can be, if they pass requirements for study rigor and reporting completeness and transparency, at least as good as a “vendor paid” researcher study. The product’s creators have this huge information advantage: they know their product front and back, and they know its theory of action – what’s importantly related to what. Yes a vendor will naturally be unable to fully resist a reporting-bias towards the positive (similar to a paid external researcher’s incentives) which is why rigorous methods, transparency and comprehensiveness are crucial. Every vendor-marketing claim should be backed up with data or research. Data provenance, methods and analysis should be transparent. The IES’ What Works Clearinghouse provides an excellent reference1 for judging study rigor. I also highly recommend Empirical Education’s tech evaluation guide, developed in concert with the SIIA edtech trade group, Guidelines for Conducting and Reporting EdTech Impact Research in U.S. K-12 Schools. A breakdown of evaluating rigor of any study, including vendor studies, and the risks they must address is found in my Studies Risk blog post.

This post is intended to prompt questions to ask of yourself or of others about sources of risk. Their responses may help you find and then unpack some risk areas, and that process will suggest mitigations. In course of these inquiries, you will learn more deeply how any program must be studied and applied, in order to achieve the real results you are seeking.

Next post:

Implementation: Implementation Risk – is that a thing?

  1. What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0 ↩︎