Recruiting for Your Pragmatic Clinical Study
6. Maximize Your Return on Recruiting for Your Pragmatic Study, Part 2
The widely-distributed pragmatic study is an emerging model for gaining insights into how a therapy performs in the real-world. The WDPS comes with a few unique caveats, but understanding how to address them is key to a fruitful pragmatic study.
Published on
January 11, 2019
Healthplan data can be a valuable source for recruiting for your pragmatic study If you know how to use it wisely and skirt some traps along the way.
“The significant problems we face cannot be solved at the same level of thinking we were at when we created them.” -- Albert Einstein (1)

“How do I find primary care physicians and endocrinologists who are willing to put a total of 1,000 type 2 diabetics, who are not well-controlled on metformin, for a pragmatic study of a recently FDA-approved extended-release metformin patch, and report their data so that we can find out how the patch is used and its outcomes in clinical practice?”

Let’s use this question, posed in Episode 5, to illustrate recruiting for a particular type of widely-distributed pragmatic study (WDPS), so that we can understand (and act on) how it is used, and how well it performs, with the wide variety of practice environments and patients in the real world.

Cut to the punchline: In a real-world pragmatic study--especially the widely-distributed kind--we want to study the use and outcomes of treatments over a wide spectrum of physicians, practices and patients. We engage physicians who in turn select patients who meet the study criteria and recruit them to be randomized or to offer them a clinically-appropriate treatment. Today we explore this process--from specifying qualifying-patient criteria through using data for screening, engaging physicians, and enrolling patients. We note potential sticking points and foreshadow considerations of process and technology platform… topic of our next post.

Recruiting for a WDPS differs from prospective randomized controlled trials (PRCTs) in certain respects. Most prominently, WDPS are not conducted solely in well-circumscribed centers; rather they’re conducted by physicians (and their patients) in the gamut of community settings. Still, like PRCTs, pragmatic studies need at least one intervention (study) group, and a comparison group (who received an alternate treatment or “usual care”).

With a WDPS, are patients actually “recruited?” Of course they are if the protocol involves randomization. If instead the physician determines whether to do a test or give an approved treatment to their appropriate patients, is the patient really being ‘recruited?’ From both the physician and patient perspective, the physician is offering them an approved clinically appropriate treatment, discussing its advantages and potential drawbacks, and helping them decide whether to take it. This is what good medical care looks like. So in this scenario, it’s the physician who is ‘recruited’ to a study in which they will offer their patients who meet criteria an already-approved treatment (or test), and likely do some extra work relating to background education and documentation.

If this non-randomized scenario doesn’t involve true patient recruitment--does it require informed consent? Or just informed decision-making?

This is currently a fuzzy area; we strongly recommend consulting your IRB or CRO.

Now let’s consider the WDPS ‘recruitment architecture’ or scenarios The doctor is recruited because he or she is the type of doctor who treats patients who meet the study criteria (and the available data say he or she has lots of patients who could be candidates); is asked to identify qualifying patients in their practice; and to offer them test or treatment.

The widely-distributed pragmatic study recruitment architecture
‍The widely-distributed pragmatic study recruitment architecture

Step 1: Specify patient and doctor criteria. We start off with a set of inclusion and exclusion criteria for patients qualified to be studied, and for physician specialties, if the protocol calls for that. The criteria are converted to algorithms (rules) and translated into queries against the available data (claims and/or electronic health records).

Step 2: Apply the algorithms or rules. The rules are about both doctors (right specialties?) and patients (meet inclusion and exclusion criteria?). The patient rules should be considered screens, especially with healthplan data, which will be used to determine which physicians are outreached-to.

Step 3: Outreach to physicians. We compile a list of physicians to invite to the study, based on who treats the most patients that met the screening criteria (Pareto analysis). We reach out to these physicians.  After fully informing them about all aspects of the study (including what we’ll ask them and their patients to do), those physicians who enroll will identify qualified patients from their practice (note that membership in one or more specific healthplans may be a qualifying criterion; and that additional data may need to be collected to support qualification (2)).

Step 4: Physicians recruit patients. Physicians enroll qualified patients into the study. If patients are assigned to treatment by randomization, the physician discusses how the study works and enrolls consenting patients to be randomized. If the design specifies that all clinically-appropriate patients are offered the treatment (3), an informed decision is made by physician and patient (just like in the rest of healthcare, though the study IRB will determine whether informed consent is needed).

Step 5: Follow patients through the protocol-specified period. The study protocol will specify what data to collect and when; a follow-up visit (or telemedicine) schedule. For randomized studies, these data will be collected on both study and comparison group patients. For nonrandomized studies, protocol determines what data to collect. Note, however, that for nonrandomized studies, untreated patients may come from those in the enrolled physicians’ practices who were clinically-appropriate for the study but declined the treatment, and/or from matched patients in non-enrolled practices.

Step 6: Analysis and reporting. Most protocols specify interim reporting to determine whether the study should be stopped prematurely because of safety issues or because primary endpoints were met, or to surface unanticipated design issues. All protocols specify what analyses must be done at study completion. Proper analysis of the various types of clinical trials and studies is beyond the scope of this series, but we’ve hinted at several considerations (and will, no doubt, continue to do so).

Information platform is an important factor to the success of your WDPS! You want to make it easy for physicians to qualify (apply study criteria, including collecting any additional data needed), inform, enroll (with consent if required) and track patients in your study. We’ll cover platform later in this series.

Data considerations for recruiting: Inclusion and exclusion criteria must be converted from plain English to algorithms in the ‘language’ of the data you’re using to recruit (e.g. healthplan, clinical or patient-reported), bearing in mind where you want to be on the sensitivity-specificity spectrum. Let’s illustrate this using healthplan data for our opening question; we’ll need:

  • Physicians with specialty types in primary care or endocrinology
  • Patients with Type 2 diabetes
  • Taking metformin but no other oral or injected meds for diabetes, including the metformin patch
  • Markers of control
  • Clinical exclusions (we’ll assume creatinine >= 2.5, heart failure, and estimated glomerular filtration rate less than 60, equivalent to Stage III chronic renal failure)

For now, we’ll ignore qualifiers such as:

  • How to determine which physician to attribute a patient to
  • How long a patient must have had diabetes
  • How far back in patient’s history to look for disqualifying evidence of other oral or injected diabetes meds or use of the metformin patch
  • Timeframe in which to look for markers of control

The protocol will specify all of the above in detail with code tables, algorithms and any timeframe requirements. Here’s a partial illustration using healthplan data (medical and pharmacy claims plus lab results data (4):

An example of precisely specifying code tables in your protocol
‍This simplified example illustrates the importance of precisely specifying code tables in-protocol, and reminds us that while healthplan data is very useful for screening for potential candidates, the physician will use or gather complete data to determine which of their patients to recruit.

I understand that studies can fail or be delayed because of poor accrual (enrollment). How can we make it easier? This is a matter of process-and-technology platform, and we look forward to your joining us to dive into it next time. In the meantime, we invite your comments (below), or to reach out to us directly, HERE.

Closing thoughts | Summary

  • One strategy for recruiting patients for a pragmatic study is to first recruit physicians who treat patients likely to qualify and ask them to recruit patients who qualify (meet the study’s inclusion and exclusion criteria).
  • Depending on the study design, patients may be randomized to different treatments, or the physician may offer the study treatment to appropriate patients, who then—after informed discussion—decide which of the treatment alternatives to embark upon.
  • Randomization always involves ‘informed consent.’ Choosing a treatment may involve formal consent, but always involves informed decision-making. Your CRO and IRB will make that determination.
  • Translating the study’s inclusion and exclusion criteria into algorithms that you can use to query your data involves a deep understanding of how your data types (claims, lab results, EHR, or patient-supplied data) works.


  1. Possibly related, Einstein also said, “Do not worry about your difficulties in Mathematics. I can assure you that mine are still greater.”
  2. For example, the study protocol may specify that lab tests, imaging, tissue sample (including genomics or microbiomics), or patient-reported data (e.g. symptoms, quality of life or functional status) be collected.
  3. Because pragmatic studies use approved treatments or tests--and specify that  the treatment/ test is to be offered only to clinically-appropriate patients--your IRB may determine that formal consent is not required. However, they may require it anyway, because of the additional data collection imposed by the protocol.
  4. Even though this is a simplified example, it illustrates the need to precisely specify code tables and reminds us that healthplan data is very useful for screening for potential candidates; the physician will use complete data (perhaps after gathering additional) for final screening. Lab results data are available in healthplan data sets when performed in a lab with which the plan has a contract to submit such data.
  5. Protocol will memorialize decisions about whether to exclude individuals taking metformin for other indications, e.g. pre-diabetes or polycystic ovary syndrome. For example, all such individuals may be excluded, or individuals with only one claim for other indications may be included if they meet all other criteria.
  6. Most healthplan data warehouses incorporate software to aggregate drugs into clinically (or chemically) meaningful groups - for example, oral hypoglycemic agents, insulins, metformin, or sulfonylureas. Note the variation in granularity of these examples! The protocol will often specify only generic/brand drug names, recognizing that the implementing organization’s informatics will translate well-specified drug names and categories into NDCs using the warehouse’s drug grouper software.
  7. CPT category II codes are used on claims to indicate that the clinician performed a procedure, ordered a test or drug, or that the patient had a lab test result within some range. These codes are often used in pay-for-performance quality management systems. Note that code 3046F is used for A1c greater than 9.0%. Alternately, the code for A1c between 7 and 9 could be added to improve sensitivity (while sacrificing specificity), since the intent here is for primary screening
  8. ICD codes for clinically-significant kidney failure can miss cases (for example, there are codes that specify stage I-IV; if only the stage V and end-stage codes are used, as in this example, patients with stages III and IV will be missed unless caught by other codes.
  9. We recognize the inherent false-positive rate of identifying heart failure with ICD codes; however, your protocol may choose to sweep in such patients, understanding that their clinicians will have or order definitive tests, such as ejection fraction
Recruiting for Your Pragmatic Clinical Study