Cube Long Arrow Right External Link angle-right Search Times Spinner angle-left

7.3 Patterns and Priorities of Success Insights

The  Patterns and Priorities of Success (PPS) Insights identifies an agency’s patterns and priorities of success, as demonstrated in Figure 7.3.a. This dashboard helps determine which assessment items, when resolved, are most strongly associated with success in care. It helps determine which people experience successful outcomes and which people more often do not experience successful outcomes, according your agency’s own definition of success.

Figure 7.3.a Patterns and Priorities of Success Example

The definition of success is determined by the you “on-the-fly” by selecting from filters:  the proportion of identified needs which were resolved, the number of strengths built and the proportion of natural support areas addressed. You can select different definitions of success to see who is served successfully for each type of outcome. This report helps an agency begin a conversation about which persons are being served effectively, and which persons may need additional services and supports. This report can be used to help generate ideas about what changes an agency could make to staff training, its service continuum, or to support implementation of evidence-based programs. This builds organizational competency, staff confidence and wins for future work.

Success is defined by you, and is set using the filters at the top of the report.

  • Filter options for Success Definitions Include:
    • Success_Needs_Resolved: This applies to the Patterns and Priorities of Success Insights. This will identify the minimum proportion of needs for focus that were identified during care that were resolved by the last assessment to meet your definition of success. The default setting is 50%.
    • Success_Strengths_Built: This applies to the Patterns and Priorities of Success Insights. This will identify the minimum number of strengths that were identified to build during care that were strengths to use by the last assessment to meet your definition of success.
    • Success_Support_Needs_Resolved: This applies to the Patterns and Priorities of Success Insights. This will identify the minimum proportion of support needs for focus that were identified during care that were resolved by the last assessment to meet your definition of success.

Any of these filters can be used alone or combined with the others.

In the example in  Figure 7.3.a, the definition of success is 50% needs addressed and 1 strength built. If a person has at least 50% of their indicated needs addressed, and at least one strength built by the most recent assessment, that is success.

Population filters for the Dashboard are located here as well, as described in 7.1 Insight Filters. There are three standard success filters, but additional criteria for success might be desired. For those agencies who want to customize the definitions of success further, our custom analytics team can help through upgrade options.

The  People Matching Filters table identifies the number of total people available and the number of people matching the filters and represented in the insights. A person must have been assessed at least twice to be represented in these insights, because these insights look at change from a first assessment to one or more subsequent assessments.

Throughout the dashboard you will see the information icon highlighting informative text related to the chart or table. For example, the  People Represented table states: “This table shows the number and demographic of people represented in this report.” You are reminded that “Sex and Race Criteria can be selected using the report filter dropdowns”.

The Group Characteristics table compares the group of youth who met the definition of success to the group who did not. The columns are described below.

  • Group Characteristics Table Columns:
    • SUCCESS: The group described by the row, including those who met the definition of success in the first row, and those who did not in the second row.
    • N: The number of people in each group.
    • M: Proportion of people in the group who were male (vs female or other).
    • F:The proportion of people in the group who were female (vs. male or other).
    • OTH:  The proportion of people in the group who identified with another gender (vs. female or male).
    • AGE: The average age at first assessment.
    • EXPS:  The average number of exposures indicated, per person. This identifies whether one group may have been exposed to more traumatic events than the other. It is a potential indicator of how trauma relates to success for your agency.
    • LOS: The average length of service per person, in days.
    • ASMT: The average number of times people in the group were assessed during care.

Looking at the example in Figure 7.3.a, the following exemplifies a possible interpretation of the results.

  • Group Characteristics Interpreted from Example in Figure 7.3.a:
    • N: There were 18 collaborations in the success group and 34 in the not success group
    • M: Males were slightly over represented in the not success group (70.6% not successful versus 50.0% successful)
    • F: Females were slightly over represented in the success group (38.9% successful versus 23.5% not successful)
    • OTH:  Other genders were slightly over represented in the success group (11.1% successful versus 5.9% not successful)
    • AGE: The average age was slightly younger in the success versus the non-success group (11.6 vs 12.9).
    • EXPS:  The people in the success group had 2.9 exposures on average compared to 2.4 in the not success group.
    • LOS: The people in the success group spent 424 days in care on average compared to 312 days in care for the not success group.
    • ASMT: The people in the success group were assessed 8 times on average compared to 4 time on average for those in not success group.

The Proportion Who Met Definition of Success or Not pie shows the proportion of people who met the definition of success versus those who did not.

So in the example in Figure 7.3.a, 35% of people met our definition of success and 65% of people did not. So 35% of people, by their last available assessment, had resolved at least 50% of their indicated needs and built at least 1 strength.

The next two line graphs represent Total Needs to Address over Time and Total Strengths to Address over Time. There is a line on each graph which represents the Success Group (blue) versus the non-successful group (yellow).

Looking at the Total Needs to Address over Time graph blue line, it represents the number of indicated needs that people in the ‘Success’ group had on their first assessment (First), ever during care (Ever) and on their last assessment (Last). This is then compared to the people in the ‘Not Success’ group in the  yellow line. In the example in Figure 7.3.a , you can see that people in the success group had 16 needs indicated on their first assessment, on average compared to 13 needs for people in the not success group. People in the success group had 21 needs ever indicated in care, on average compared to 20 needs for people in the not success group. People in the success group had 6 needs indicated on their last assessment, on average compared to 16 needs for people in the not success group.

Generally, a common pattern for needs is to see a slight increase in indicated items from First to Ever, because as people become more comfortable in care and as they address some needs, other needs will be revealed. This is very typical, and an increase in needs from First to Ever is generally expected. The change from Ever to Last indicates the total number of needs addressed in care, and this change is generally larger for the Success group. As we see with the blue line in Figure 7.3.a, a typical pattern of success in this graph looks somewhat like a hockey stick if held over your head. In comparison, yellow line for the Not Success group looks much more like a bat.

Looking at the Total Strengths to Build over Time graph, you can see that the graph compares the number of strengths identified to build people in the Success and Not Success groups have at First, Ever, and at the Last assessment. In the example in Figure 7.3.a, you can see that people in the success group had 5 strengths to build indicated on their first assessment, on average compared to 6 strengths to build for people in the not success group. People in the success group had 6 strengths ever indicated to build, on average compared to 7 strengths to build for people in the not success group. People in the success group had 3 strengths to build indicated on their last assessment, on average compared to 6 strengths to build for people in the not success group. Therefore, people in the success group built 3 strengths on average compared to 1 built by the not success group.

The Top Items for Success table ranks the assessment items most strongly associated with success. While this is not a causal association, this does give justification for further investigation into these areas. It could be possible that the top items, when resolved, are drivers of success.

In the Items column, you will see a list of the names or labels for each question on your agency’s assessments. The names of Items use the abbreviations of the instrument, category and question/item, separated by underscores, as:  INSTRUMENT_CATEGORY_ITEM. You’ll see that most Items represent questions from assessments, but there are a few additional measures our analytics team determined were important to consider in the model, and these will be ranked in the list when significant. The additional items are listed in the following.

yellow line. In the example in Figure 7.3.a, you can see that people in the success group had 16 needs indicated on their first assessment, on average compared to 13 needs for people in the not success group. People in the success group had 21 needs ever indicated in care, on average compared to 20 needs for people in the not success group. People in the success group had 6 needs indicated on their last assessment, on average compared to 16 needs for people in the not success group.

Generally, a common pattern for needs is to see a slight increase in indicated items from First to Ever, because as people become more comfortable in care and as they address some needs, other needs will be revealed. This is very typical, and an increase in needs from First to Ever is generally expected. The change from Ever to Last indicates the total number of needs addressed in care, and this change is generally larger for the Success group. As we see with the blue line in Figure 7.3.a, a typical pattern of success in this graph looks somewhat like a hockey stick if held over your head. In comparison, yellow line for the Not Success group looks much more like a bat.

Looking at the Total Strengths to Build over Time graph, you can see that the graph compares the number of strengths identified to build people in the Success and Not Success groups have at First, Ever, and at the Last assessment. In the example in Figure 7.3.a, you can see that people in the success group had 5 strengths to build indicated on their first assessment, on average compared to 6 strengths to build for people in the not success group. People in the success group had 6 strengths ever indicated to build, on average compared to 7 strengths to build for people in the not success group. People in the success group had 3 strengths to build indicated on their last assessment, on average compared to 6 strengths to build for people in the not success group. Therefore, people in the success group built 3 strengths on average compared to 1 built by the not success group.

The Top Items for Success table ranks the assessment items most strongly associated with success. While this is not a causal association, this does give justification for further investigation into these areas. It could be possible that the top items, when resolved, are drivers of success.

In the Items column, you will see a list of the names or labels for each question on your agency’s assessments. The names of Items use the abbreviations of the instrument, category and question/item, separated by underscores, as:  INSTRUMENT_CATEGORY_ITEM. You’ll see that most Items represent questions from assessments, but there are a few additional measures our analytics team determined were important to consider in the model, and these will be ranked in the list when significant. The additional items are listed in the following.

  • Additional Items in the Top Items for “Success” Table:
    • LOS_Days: Length of service in days
    • Age_First_Assessed: Age at first assessment
    • Total_Needs_Ever_Identified: The total number of needs ever identified during care
    • Number_of_Strengths_Identified_to_Build: The total number of strengths ever identified to build during care
    • Number_of_Strengths_Built: The total number of strengths built during care
    • Pcnt_Support_Needs_Resolved: The percent of support needs resolved during care
    • Total_Support_Needs_Ever_Identified: The total number of support needs ever identified during care

On the right is the Item’s Rank importance, based on how predictive it is for success. As seen in Figure 7.3.a, the top six ranked items are Living Situation, Anger Control, Social Functioning, Sleep, Intentional Misbehavior, and Strengths for Talents and Interests.

The Items of Success Decision Tree takes the top ranked items and identifies the detail of the association. The decision tree displays the probability that people will be in a success group based on items which were indicated resolved. In general, people with resolved items are on the right side of each branch of the tree, but use the key to read the tree. The tree identifies which items, if resolved, are most associated with success. Each branch of the tree splits into subgroups of people depending on whether or not the people have had an item indicated and resolved. The goal of each branch is to maximize the proportion of people who met success after each split. Therefore, the tree branches based on Item status which are most often associated with success. Each tree splits differently, and can be read using the following key.

  • Key to Reading the Branch Splits of the Tree:
    • PR = Presented and Resolved: For these people, this item was indicated on the very first assessment and then later resolved sometime during care.
    • DR = Discovered and Resolved: For these people, this item was not indicated on the very first assessment but was identified on a subsequent assessment and then later resolved sometime during care.
    • PU = Presented and Unresolved: For these people, this item was indicated on the very first assessment and then was still indicated on the last most recent assessment.
    • DU = Discovered and Unresolved: For these people, this item was not indicated on the very first assessment but was identified on a subsequent assessment and then then was still indicated on the last most recent assessment.
    • NI = Never Indicated: For these people, this item was never indicated during care.

Each time the tree branches, the tree will show a node, or a bubble, for the subgroup based on the split. The notes can three numbers, a top, middle and bottom number, which are described in the list below.

  • Key to Reading the Nodes (Bubbles) of the Tree:
    • Top Number:
      • 1 in the top row indicates that over 50% of people in this group achieved success
      • 0 indicates that less 50% of people in this group achieved success.
    • Middle Number: Represents the actual proportion of people in this group that met the definition of success. (If this is greater than 0.50, then the Top  Number will be a 1, and if this is less than 0.50, then the top number will be a 0.)
    • Bottom Number: The proportion of the total population that is represented by this node

Looking at the example in Figure 7.3.a, when Living Situation is DR (discovered and resolved) or PR (presented and resolved), you move to the right in the first branch of the tree to see that 100% of these people achieved success, and this represents 21% of the people in the tree. When Living Situation is DU (discovered and unresolved), or NI (not indicated), or PU (presented and unresolved), you move to the left in the first branch of the tree to see that 17% of these people achieved success. We can further break this group down by whether or not they resolved Sleep concerns. Those who did fall to the right and those who did not, or who did not have Sleep as a concern fall to the left. A total of 71% of people who resolved a concern for Sleep achieved overall success in their collaboration, which represents 13% of people in the tree. The last node, all the way to the left represents the remaining 65% of people, of which only 6% achieved success in our program. Here we can see that living situation and sleep are important aspects for success for over 30% of our population.

This is a lot of information. But what this is telling us is that these particular items may be important to the people we’re helping. These associations are not necessarily causal. It doesn’t necessarily mean that resolving these items will cause success, but it these items are associated with success more often, and this is a good first step into identifying what works for whom in a service population.

The Model Prediction Accuracy identifies the reliability of the identified patterns and priorities for success from the rankings and from the tree. This table updates each time the model is run to let you know how reliable the predictions are for success for the population you are evaluating. Accuracy is reported in two measures, model specificity and sensitivity, as described in the following. If one of the measures is missing, then the result of that measure is 0%. This could occur with really small samples.

  • Model Prediction Accuracy Measures:
    • Sensitivity: Shows us how good the model is at predicting success, providing a true positive. A high sensitivity means that people who resolved highly ranked items experienced success very often. A very low sensitivity means that people who resolved highly ranked items may not have experienced success more often.
    • Specificity: Shows us how good the model is at predicting not success, providing a true negative. A high specificity means that people who did not resolve highly ranked items experienced success much less often. A very low specificity means that people who did not resolve highly ranked items may have experienced success anyway.