By James Denmark, Founder and CEO
From our webinar: Uncover Study Compliance Blind Spots with the myClin Compliance Score
Compliance Blind Spots
Our objective today is to understand the full compliance disposition of your study, to show everybody has the latest information and to track users readership of this information and lastly to identify the blind spots and measure those to be prepared for an inspection.
To get started, let’s talk about some definitions of compliance and blindspots. Turning to the Merriam Webster’s dictionary, compliance is defined as conformity in fulfilling official requirements.
So in our context, that means how can you assure that at least as far as it affects quality, every site is using the latest documents and procedures according to the study protocol.
A blindspot, is defined as a portion of a field that cannot be seen or inspected with available equipment. So today, if we think of our available equipment as our systems and our monitoring staff in the field, then we need to define what they are measuring and inspecting that are not available to us in time to base decisions on.
So to tackle our first objective of understanding the full compliance disposition of your study, as an industry, we focus on metrics.
Most clinical operations teams have the benefit of having EDC and IRT systems deployed on many of their studies, they produce easy to measure metrics that are objective, simple to track and reliable. Such as site activations, subject enrollment, drug supply, CRF completion, et cetera. And of late, teams have begun to introduce risk management assessments at the beginning of their program and they attempt to measure that also. So you may have a risk log, a set of mitigations and a recurring set of trip report findings and potentially audit findings from proactive QA audits by sponsor or worst case scenario, actual inspection findings from a regulatory agency inspector.
So let’s look at the metrics we use.
We are going to look at a number of the metrics I just described, their source, whether they are leading or trailing, whether they are detectable, how timely they are and the impact of them being off track. So the first batch were the ones I described before as easy to track. Site selection, activation, enrollment, drug supply, lab tracking and even safety. I’ve classified these as leading indicators because they are both detectable insofar as there are frequently systems and very robust processes used to track these. They are coming in near real time, certainly less than 24 hours for site activation and enrollment information, drug inventory at the point of dispensing. Lab samples when kits show up at the Central Lab and for Adverse Events, your medical monitor knows about it within the business day, and that will be distributed within the prescribed process within 48 hours. So these are what we qualify as objective and simple leading indicators that are very useful to determine your study performance.
The next category I classified as trailing indicators. Those include case report form completion, query rates and response times. These are objective and imminently detectable since the metric was produced by the EDC system, but they tend to lag by about 21 days on average. Three of more weeks behind the actual activity at the site, but they are critical indicators of database completeness and your data quality.
The last category of trailing metrics are the ones that address risk. These are the ones that often fall into the blindspot. Trip report findings, audit findings, TMF completeness and ISF completeness are only detectable when you go on site and then only detectable when the contents of a trip report are then disseminated to decision makers. The impact of failing to respond to them is severe. So let’s look at some examples.
The metrics champion consortium developed a scorecard of ten critical site quality metrics. Of those, four are focused on compliance.
Particularly on protocol deviations for screened to randomized patients and on major audit findings. Coming back to our previous example, where these types of metrics are both trailing and somewhere between 21 to 90 days out of date. They hardly represent a meaningful metric that you can use to make decisions.
And wouldn’t it be great if you could identify these issues before an inspector records them in a warning letter.
So let’s quantify that. This chart represents the top 10 warning letter classifications over the last five years. And as you can see, it’s been a net decline of warning letters issued. But the overall distribution in terms of the category and classification remains consistent. So let’s make this a little simpler. Our focus in just on 2019.
So last year you can see that there were twice as many compliance citations. That’s the far right column, as there were findings around clinical data and was four to five times more frequently cited than or other findings combined. So clearly, this sector of the clinical compliance process has room for improvement and represents a blindspot for clinical teams today.
So coming back to our objective. Strategy number one was to understand the full compliance disposition of your study. Our conclusion is that as an industry we are pretty good at quantitatively measuring data quality, completeness, but we have significant room to improve in the investigator and protocol compliance arena. Under the current tools of monitoring and audits are really a rearview mirror with serious blindspots.
As we focus in to the root cause of many clinical compliance issues it often comes down to documentation, training, using the right versions of materials. So let’s begin to get to grips with that. So how do we ensure that all parties have correct and up to date study information all the time?
So digital material – the documents, that are distributed in a study become this whirlwind of electronic assets typically distributed by e-mail and ultimately boil down to a large volume of electronic documents that have information that must be tracked manually.
So the DIA-TMF lists 200 core documents are expected on every study.
An average study has about 300 central documents.
Each investigator site produces at least 100 of his own site documents.
So a thirty five site study is going to produce about thirty eight hundred individual documents and a 100 sites study produces over 10000 documents.
Now, in the scheme of TMF archiving, 10000 documents is not a great challenge.
The way you begin to multiply all of the different interactions with those documents during the study and inspectors begins to question you about who saw what, when and how often and what did they do with it? This becomes an exponentially larger problem to solve.
Predominantly today, this material is distributed via email, either directly from the sponsor or via the monitor.
And distribution via e-mail presents clear dangers. It’s dangerous because it cannot be deleted or retracted. It can be forwarded without your control.
That the use of the blind copy feature erodes trust with the recipients of your emails. It’s not private, but we act like it is. There really is no such thing as secure email. It’s very easy to cut recipients out of a chain and you cannot see the wood for the trees. There are so many e-mails we receive on a daily basis that really expecting our message to get through is unrealistic.
So when it comes to ensuring that all parties have the correct up to date study information at any time, our recommendation is that you develop a policy that prohibits the use of email to distribute study documents and use a central sharing a repository like myClin, but there are others out there too. It becomes the single source of truth for the sites and the study team. The system should be cloud-based, available globally and able to comply with all applicable regulatory requirements.
So once you get your documents out there, tracking readership or the user interaction with study information is how you can shine a light into the dark corners of the study and becomes an essential tool to subsequent inspection readiness.
So as you start to distribute information using a central platform, you are now able to track what happens to it after you hit the send button. Unlike email, this is no longer a black hole and you get to leverage readership information that you previously never had access to.
So as you distribute documents wouldn’t it be great if you could know exactly who interacted with it? I have a short example here using myClin where we’ve distributed a safety report, an inspector asked me a question about a particular site’s access to it. I can quickly search for the med watch report. Pull it up. Take a look at it and do this all while they the inspector is watching, go to the compliance view.
And here I can see all of the sites which access they had to it and sort by whether they have looked at the item and filed it or not. Let’s look at the site in Denmark. The inspectors asking about. And here you can see every interaction that site had with the document, which version they viewed both by the P.I. and the coordinator and that the fact that they downloaded and filed it. And lastly, if I need to produce this for the inspector, I can download that to PDF.
Now, on a single document, that means that you’ve got quite a lot of information just on a single instance of a document.
So now imagine when you’ve got hundreds or thousands of documents that are all being viewed and tracked by hundreds of thousands of users. Those actions viewing, filing, acknowledging, printing, downloading all add up to generate your final summary scores.
So tracking readership and checking site interactions is relatively easy with a tool like myClin, but without it practically impossible. And so it becomes immensely critical to track the user interaction with study information. Archiving documents is easy. Tracking site interactions with it is incredibly hard. Automated readership and receipt tracking is feasible, scalable, cost effective, reliable and most importantly, defensible.
So how does one score compliance using this information?
So a compliance score looks a lot like this, where you have an overall score for a study. Individual scores by site and region and access to the actual content of the study and other metrics.
During study execution. Absolute compliance scores are less important than identifying outliers and diverting resources to assist them.
And so looking at this example of a dashboard of for single study, you can see that the most important question is not why did the John Radcliffe Hospital still has 8 percent of their files un-acknowledged, but more importantly, why the Royal Marsden and the University of Pennsylvania have more than two thirds of their documents out of compliance.
Drilling down into an individual site such as this one in Denmark. You can see which documents that site still needs to review or acknowledge. It’s a strategy for exposing today’s compliance blindspots.
Our recommendation is that you should develop an objective approach to assessing compliance. It can no longer be a subjective assessment of just the monitor.
You should be transparent with these metrics. Share these scores with your investigators and their staff in real time. So they can see them and react to them before your monitor has to ask the question. You should focus on outliers during the study execution on worry less about the absolute numbers.
Lastly, how does this afford you the ability to be ready for an inspection at any point during the study? So when you receive an inspection, they may focus on a particular document such as safety distributions and look at this across the study. So you need to be able to look at that for all sites for a particular safety report across the study. Or they may look at a particular document in which case you then need to be in a produce an audit trial of an individual site’s access to a document.
In this example, you can see. Sophie Pedersen, the coordinator at this Danish site, viewed and downloaded these versions.
So recording evidence of good study execution is easy to do and with this accumulating over time you can easily project confidence in the integrity of the core clinical data being collected by sites.
In addition, continual access to these documents by your study team, including the clinical operations manager and other members of the CRO or study team also register views, download and prints and will prove your routine oversight without the need for additional extra documentation.
So today we discussed 5 strategies to uncover study compliance blind spots with the myClin Compliance score and we would be happy to take questions from the audience now.
Watch the webinar for the audience questions and answers!