Skip to main content
search

Kyruus Health empowers healthcare organizations to connect patients with the right care through its industry-leading provider search and scheduling platform. At the core is robust provider search functionality that unifies and curates data from across the enterprise to deliver accurate, comprehensive provider profiles. Patients, call center agents, and care navigators can quickly match individuals to the most appropriate provider based on factors such as clinical expertise, availability, insurance coverage, and patient preferences. This improves access, reduces friction, and drives higher patient satisfaction and retention.

Problem with too many results

When our users are searching for care, their search typically falls into one of five categories: provider speciality, medical condition, a provider’s name, a practice location, or a primary care provider. Searching across all categories causes two issues: The broader the search, the slower the response was from the API and returning results from all categories meant that close to 80% of the results were irrelevant to that user.  What we found was that it took our users a long time to find care that was relevant to them.

Matching Results to Users’ Intent

To address this problem of diluted relevancy, we leverage our existing data to train a machine learning text classifier using AWS Comprehend. AWS Comprehend uses natural language processing to classify a string into our five categories accompanied by a level of confidence like:

Using these classifications, we are able to tailor our search to match our users’ most likely intent, removing irrelevant results making it easier to find the right provider. 

We recognize that our model may be incorrect from time to time; a user may actually be looking for treatment for Wilson’s disease even though most users would be looking for a provider name. That’s why we implemented a feature that allows users to correct the intent:

Any category that scores above a given threshold gets included in the results, and any category that scores below the threshold but above zero, is included in the suggestions. 

Measuring Success

Product Metrics

To ensure that we captured enough feedback to validate our assumption that fewer more relevant results would improve our users’ experience, we defined a set of metrics and recruited a handful of customers to beta test our new feature:

  • Increased Scheduled Appointment Rate by 38%
  • Increased Ranking Position of scheduled appointment by 33%
  • Decreased time to schedule by 98%
  • Did not harm No Result Search Rate
  • Achieved Statistical Significance

System Metrics

In addition to the primary product metrics, we defined a set of measurements that would tell us how our system was performing:

    • A little more than half of our Classifications are for locations or provider specialties.
    • 90% of search classifications are retrieved from a Cache mitigating scaling and cost issues with Comprehend
  • Only 0.07% of searches Fallback To Legacy Behavior indicating that the vast majority of users are getting a better experience.

 

These metrics gave us confidence to release our feature to our entire customer base which presented one other problem: Scaling.

Scaling

Our Intent Matching feature utilizes AWS Comprehend’s custom classifier model with a real time endpoint to classify each search string as they are submitted. The endpoint has a configured number of Inference Units (IUs) that determine how many characters it can process at a time (One IU can process 100 characters per second). IUs have two challenges that we had to overcome: they’re expensive and slow to scale up. 

Caching Classifications

We have a number of common searches across our platform which allows us to cache the classifications for a given search term in a Redis instance. With around 90% of searches utilizing this cache, it provides significant performance improvements as well as cost savings since we’re not hitting Comprehend as much. Each entry expires every few days so when we retrain and release a new model, new classifications slowly roll out over time and don’t affect performance.

Auto Scaling

The first thing we did to address our scaling problem was to implement target tracking so when our Inference Unit Utilization goes above a given threshold, we add more IUs to handle the load. In addition to traditional target tracking, we also implemented anomaly detection that enables AWS to predict increases in demand and proactively scale up before it is needed. This achieved the goals of cost optimization and maintaining good user experience.

Adjust User Experience Instead of Throwing Errors

There are two scenarios where our auto scaling doesn’t handle increased demand: when we hit our upper limit of Inference Units or when we are scaling up but the new capacity is not available yet. When either of these scenarios is detected, we fall back to the legacy behavior of searching across all categories instead of presenting the user with an error. This allows our user to accomplish their task even though the experience may not be as good as we want it.

 

Next Steps

Our next step is to operationalize the retraining process so our Comprehend model gets even better at classifying text. Using usage data from users clicking on the suggestions, publicly available medical datasets, and customer feedback, we can start to refine the model so it meets the needs of both users and customers. [/vc_column_text][/vc_column][/vc_row]

Close Menu