Insights

From our roots in academia to today’s technology, groundbreaking and peer-reviewed research provides the foundation for all that we do.

The 5% Problem: What Random Sampling Misses in Quality Oversight

Quality oversight has long been a cornerstone of behavioral health and human services. Supervisors review recorded interactions to coach staff, monitor fidelity to evidence-based practices, and identify opportunities for improvement. But behind nearly every quality assurance program is a practical limitation that often goes unnoticed: only a small fraction of conversations are ever reviewed.

A 5% sample can sound reassuring. But for a caseworker who conducts 50 interactions in a month, that amounts to just two or three recorded conversations being reviewed.

The question is no longer whether 5% was a reasonable standard. It’s whether 5% is enough to confidently understand workforce performance today.

How 5% sampling became the standard

Quality oversight has always operated within practical constraints. Supervisors balance coaching, case consultation, documentation, staff development, administrative work, and the day-to-day needs of their teams. Listening to and evaluating every interaction has never been realistic through manual review alone.

Research in community mental health has documented the same challenge: evaluating fidelity to evidence-based practices has traditionally depended on trained human reviewers, making the process labor-intensive, costly, and difficult to scale.

Random sampling gave organizations a practical solution. Reviewing a small percentage of interactions made quality oversight manageable. The challenge comes when that operational limit is treated as a complete measure of quality.

What a 5% sample can miss

A small sample doesn’t just limit visibility. It introduces variability into quality decisions. The interactions selected for review may reflect an unusually strong day, an unusually difficult case, or simply chance.

Three limitations become especially important:

  • Sampling variance: A worker may appear stronger or weaker depending on which interactions are selected for review.
  • Inconsistent scoring: Supervisors may interpret the same evaluation rubric differently, creating variation across reviewers, teams, and offices.
  • Practice drift: Skills learned through training may gradually weaken without regular observation, feedback, and reinforcement.

Together, these limitations can affect how accurately quality review reflects day-to-day practice.

Sampling variance can distort quality review

Even experienced professionals have difficult days. A family visit may involve unusual circumstances or heightened emotions that make the interaction particularly challenging. When that visit becomes one of only two reviewed interactions, it may carry more weight than it deserves.

The reverse can happen just as easily. A random sample may capture two routine, high-performing sessions while missing the interaction that would have revealed a meaningful coaching opportunity.

When those reviews influence coaching priorities, professional development, or performance conversations, sampling variance can affect how organizations allocate time, training, and support. Leaders are often making decisions about quality improvement using only a fraction of the available information.

Quality review scores can vary across supervisors

Evaluation rubrics bring structure to quality review, but they still require human judgment. One supervisor may view a question as collaborative and open-ended, while another sees it as leading. A response one reviewer considers empathic may receive a different score from someone else.

This issue, known as inter-rater reliability, can create inconsistencies even when supervisors are reviewing the same type of work against the same standards. Training and calibration can improve agreement, but maintaining that consistency takes time — particularly across larger organizations with multiple supervisors, locations, and service lines.

Practice drift can be difficult to detect

Practice drift may be the most consequential blind spot created by limited sampling.
Following training, staff often begin using new skills consistently. Over time, those skills can gradually fade as caseloads grow, administrative demands increase, and daily pressures compete for attention.

Practice drift often looks like:

  • Fewer open-ended questions
  • Less reflective listening
  • More directive advice
  • Less collaborative goal-setting

In one study of counselors and supervisors at an opioid treatment program, participants described the limitations of one-time training and emphasized the value of timely, structured feedback to reinforce new skills.

When quality review relies on periodic snapshots, changes in practice may go unnoticed between reviews. By the time those changes appear in a random sample, new habits may already be well established. Limited sampling can also miss strong performance, making it harder to recognize high-performing staff and replicate effective practices across teams.

How to expand visibility across the workforce

Advances in AI make it possible to evaluate a much larger share of recorded interactions than manual review allows. Rather than relying on isolated snapshots, supervisors can begin to understand patterns that emerge across weeks, months, and even entire teams. That broader view helps supervisors see patterns earlier and place individual sessions in context.

Organizations can also measure adoption of new practice models, evaluate training effectiveness, and identify emerging workforce trends with greater confidence. For example, a supervisor could identify:

  • A gradual increase in advice-giving
  • A decline in reflective listening
  • Recurring difficulty with open-ended questions
  • Strong examples of empathy or collaborative goal-setting
  • Differences in skill use across teams, roles, or locations

Instead of waiting for practice drift to appear in a random sample, supervisors can proactively identify patterns — such as increases in advice-giving or decreases in reflective listening — and provide targeted coaching while those skills are still developing. These patterns give supervisors a clearer starting point for coaching. They can focus on specific behaviors, review relevant examples with the worker, and track improvement over time. They can also distinguish an isolated difficult interaction from a recurring concern, making it easier to reinforce strong performance and address emerging coaching needs before they become entrenched. AI can also apply a more consistent standard to key behaviors, such as empathy or reflective listening, helping reduce the inter-rater variability that can occur when different supervisors apply the same evaluation rubric.

Context still matters. Supervisors understand the worker, the family, the case history, and the conditions surrounding an interaction. They bring the professional judgment needed to interpret what the data shows and decide how to respond. More complete visibility gives supervisors better information for coaching, helping them focus their time where it can have the greatest impact.

With AI supporting review across a broader set of interactions, quality oversight can become a more continuous process that supports learning, coaching, and ongoing improvement rather than periodic review cycles.

More visibility should come with higher expectations

Expanding quality oversight also raises an important question: How do organizations know whether an AI tool can be trusted?

Child welfare and human services involve complex and often high-stakes circumstances. Inaccurate or biased results can affect workers, families, and communities, making reliability especially important when AI is used to inform supervision and quality decisions.

When evaluating AI-supported quality tools, key questions include:

  • Was the technology trained using relevant child welfare, behavioral health, or human services data?
  • Was it validated against established measures by qualified experts?
  • Has it been evaluated for bias across different demographic groups?
  • Is the provider transparent about performance, validation, and limitations?

The answers can help determine whether the technology is appropriate for quality oversight and capable of providing insights that strengthen professional judgment. That distinction is especially important when comparing general-purpose AI with technology developed specifically for health and human services. Quality oversight requires specialized data, validated measures, and expert human review.

Lyssn’s approach to clinically validated AI is grounded in data labeled by human clinical experts and tested against established measures of practice. Any AI used in these settings should demonstrate that it is safe, valid, robust, assessed for bias, and designed for the environment in which it will be used.

Conclusion: Looking beyond the next 5%

Random sampling has played an important role in quality oversight because it matched the realities of manual review. Today’s technology creates an opportunity to expand visibility and better understand how practice changes over time.

With a more complete picture, organizations can evaluate whether training is improving practice, measure fidelity over time, and identify where additional support is needed. Supervisors and quality leaders can make more informed decisions about coaching, reinforce evidence-based practices, and gain a clearer view of workforce strengths and emerging challenges.

More complete visibility can strengthen supervision, continuous improvement, and organizational learning across the workforce. And when AI is grounded in human clinical expertise and validated for health and human services, it can provide reliable insights that support professional judgment.

Frequently asked questions about AI and quality oversight in behavioral health and human services

What are the limitations of reviewing only 5% of interactions?
A 5% sample may provide useful observations, but it can also miss recurring patterns, gradual changes in practice, and interactions where coaching would be most valuable. At the individual level, a few selected sessions may place too much weight on an unusually strong or difficult interaction.

What is practice drift, and why can it be difficult to detect?
Practice drift occurs when skills learned through training gradually weaken over time. It can be difficult to detect because the change often happens slowly, and a small random sample may not capture the pattern until new habits are already established.

How can broader visibility support better coaching?
Reviewing a larger share of interactions can help supervisors identify recurring behaviors, distinguish isolated events from broader patterns, and find specific examples to use in coaching. This gives supervisors a clearer basis for deciding where support is needed and where strong practice should be reinforced.

What should human services organizations look for in an AI-supported quality tool?
Organizations should look for evidence that the tool was developed using relevant human services or behavioral health data, trained and assessed by qualified experts, validated against established measures, and evaluated for bias. Providers should also be transparent about performance, limitations, privacy, and security.

Proven science. Powerful AI.
Profound improvement.

Let Lyssn reduce burnout and transform the way you implement and model fidelity to evidence-based practices.