USING ARTIFICIAL INTELLIGENCE TO PREDICT RADIOLOGY WAIT TIMES

Participated in the operational data challenge posed by the Medical Analytics Group at MGH. The challenge consists of building machine-learning models to predict the wait time for four radiology departments at separate facilities.

This includes robust datasets spanning multiple years. If you are interested in learning more about this challenge click here or the button below.

The following is an app we built to run on an iPhone or iPad using the iOS Core ML framework.

The first step in the process is to review the features available in the data and to develop an optimal feature subset selection (OSS) to build the models.

OPTIMAL FEATURE SUBSET SELECTION - OSS

The datasets provided by the Medical Analytics Group included eighty-five features for facilities 1 thru 3 and fifty-seven features for facility 4. Facility 4 is a walk-in facility and does not provide the same exams as the other 3 facilities thus it will have less features.

Features are also sometimes referred to as “variables” or “attributes and represent a column in the tabular data or csv files. Each feature, or column, represents a measurable piece of data that can be used for analysis: DayOfYear, ArrivalTime, ScheduleTime, and so on. 

There is no right or wrong answer when selecting features. Part of Data Challenge was to reduce the number of features to build the model. Selecting features is a trial and error to select the appropriate number to obtain an accurate prediction.

I eliminated features that were derived from averages or sums of other features because I believe they dilute the magnitude of measured real-time variables. The csv files I used to build the models for facilities 1 thru 3 contained forty-four features to predict wait time. Facility 4, which is a walk-in facility with less exam types are performed, I used twenty-five features to predict wait-time.

PATIENT WAIT TIME MACHINE-LEARNING MODEL DEMO

I built a single view iOS app using iOS Charts created by Daniel Gindi to graph the output of each of the facilities models predictions based on the test data. 

FACILITY 1 THRU 4 MACHINE-LEARNING MODEL RESULTS

I selected data from the same day-of-the-year as the current date and included all the radiology exams performed prior to the current time as test data.

It's best to test models with data it has not seen so I used the data from the other facilities as test data to test each model:

• Facility 3 data was used to test facility 1's model.
• Facility 3 data was used to test facility 2's model.
• Facility 2 data was used to test facility 3's model.
• Facility 1 data was used to test facility 4's model.

Facility 4 is a walk-in radiology center and does not perform neuro, abdominal, vascular or cardiac exams. Therefore only thoracic, pediatric and muscular skeleton exams were extracted from the test data to test facility 4's model.

The blue line is the actual wait time while the gray line is the predicted wait time in minutes. Anything below zero represents no-wait. 

The yellow line represents the models confidence as a percentage and varies with each prediction. 

FACILITY 1

Facility 1's  model predicted there is no wait at 11:15 AM. The actual wait time is less than zero which indicates the patient showed up early for the appointment and taken early. The model confirm this with 99.7% confidence.

FACILITY 2

Facility 2's model predicted there is a 4 minute wait at 11:15 AM. The actual wait time is less than zero which indicates the patient showed up early for the appointment and was taken early. The model does not have a high degree of confidence in the prediction at ~24%.

FACILITY 3

Facility 3's model predicted there is a 6 minute wait at 10:30 AM. The actual wait time is about 20 minutes. The model does not have a high degree of confidence in the prediction at ~14%.

FACILITY 4

Facility 4's model predicted there is a 2 minute wait at 10:30 AM. The actual wait time is 2 minutes. The model is confident in this prediction at ~78%.

If you are interested in developing a machine-learning model to predict wait time for your organization or are interested in learning more. Please contact us by email at support@bizzlesoft.com.