FAQs: High Impact Questions & Other Frequently Asked Survey Questions

In this article: 

• High-Impact Questions (Key Drivers of Engagement)
• What is favorability?
• Why are the numbers of respondents different for different questions?
• Why is favorability the best way to look at the data?
• What does "insufficient response" mean?
• What is the minimum response required?
• What are Engagement Outcomes and High-Impact Questions?
• Why do we not focus on uncertainty and unfavorability?
• Is there ever a situation where it's useful to focus on the six-point average? 
• What do I do if I forget my password? 
• What is the difference between question-level and group-level favorability?
• What do the engagement profiles mean?
• Why doesn't Quantum Workplace calculate statistical significance?
• What is a Commitment Plan Owner vs a Plan Observer?
• What is the difference between Engagement and BPTW benchmarks? 

 

High-Impact Questions (Key Drivers of Engagement)

What is a High-Impact Question?

• A High-Impact question, also known as a key driver of engagement, is a survey question that has a larger impact on employee engagement compared to other survey questions.

Why are High-Impact questions valuable?

• An employee engagement survey measures perceptions around important workplace topics. The feedback from these surveys provides excellent insight into what employees find favorable or unfavorable about the company.
  
  High-Impact questions are valuable in that they help narrow the focus on what really matters to the employees. These can be relative strengths of the organization or relative opportunities. By improving perceptions on these questions, employee engagement is much more likely to increase and be sustained. 
  
  For example, an organization's least favorable question could be, "I am paid fairly." The organization may then decide to give everyone a pay raise and assume the problem is solved. However, if the question isn't a High-Impact question the pay raise is unlikely to make employees stronger advocates, show more discretionary effort or improve the chances of them staying with your organization. 
  
  While it is important that employees aren't paid below market standards and that they understand their compensation plan, solving your least favorable survey question is not likely to improve employee engagement in your organization. 

How are High-Impact questions determined? 

• Quantum Workplace's employee engagement surveys are created with two types of questions- Outcomes and High-Impact questions. 
  • Outcomes are at the core of how we measure employee engagement and help reveal the current state of employee engagement at an organization. Outcomes are not actionable because they are targets that organizations should strive to maintain or enhance. 
    
    Outcome questions can look like, "I recommend this organization as a great place to work." When outcome questions are consistently favorable, employee advocacy of the organization follows. 
  • High-Impact questions (sometimes known as Cultural Diagnostics) help organizations identify ways to influence engagement outcomes. 
    
    High-Impact questions often assess perceptions of career growth and development, communication, trust in leadership, future outlook, and individual needs. Identify the High-Impact questions to tailor your surveys with actionable questions based on feedback. 

• For those wanting a detailed explanation of the statistical analysis - High-Impact questions are calculated in a two-step process.
  1. Calculate the overall engagement of each respondent. 
     
     Overall Engagement = Sum of scores for all engagement outcome survey questions. This is typically nine outcomes, each rated on a 1-6 favorability scale. 
     
     For example- if a respondent rates all nine outcomes as "1" or least favorable, we add the nine scores of "1" for a sum of nine. If all nine outcomes are scored with "6," the sum is 54. Most overall engagement scores will fall in between. 
     
     Note- If your organization used more than nine outcome questions or prefers a different scale, the numbers in the calculation will change accordingly. 
  2. Determine which questions (non-engagement outcomes) are most related to employees' overall engagement ratings using a Pearson product-moment correlation, or "Pearson Correlation."
     
     Higher Pearson Correlation coefficient = survey question is more related to overall engagement. 
     
     High-Impact questions are the questions with the highest correlation coefficients. 

How many High-Impact questions are calculated?

• Quantum Workplace typically calculates the top ten High-Impact questions.
  
  However, if your organization prefers shorter employee engagement surveys, a shorter list such as only the top five can also be calculated. 

Does each team receive a unique set of High-Impact questions?

• No- Quantum Workplace typically calculates one set of High-Impact questions for an entire organization. This is for both practical and statistical reasons. 
  
  Group size can strongly influence the statistical analysis. Practically, a list of ten High-Impact questions allows for consistent effort internally and provides different teams with different areas to focus on as strengths (more favorable) and opportunities (less favorable) vary across teams. 

Will our High-Impact questions change year-to-year? 

• Quantum Workplace conducts a High-Impact analysis after each census engagement survey. While an organization's High-Impact questions may change, we tend to see more consistency than change year-to-year. 
  
  We believe it is important that we calculate High-Impact questions after each census survey to capture what is most important to your employees' engagement as your organization and workforce evolve over time. 

Are my organization's High-Impact questions typical?

• While your organization's list of High-Impact questions are unique to your organization, we do see common themes across different organizations. The following survey questions commonly appear in an organization's list of High-Impact questions:
  • I find my job interesting and challenging.
  • My job allows me to utilize my strengths.
  • This job is in alignment with my career goals.
  • My opinions seem to count at work.
  • I see professional growth and career development opportunities for myself in this organization.
  • I believe this organization will be successful in the future.
  • I trust our senior leaders to lead the company to future success.
  • If I contribute to the organization’s success, I know I will be recognized.
  • The senior leaders of the organization value people as their most important resource.
  • I know how I fit into the organization’s future plans.
  
  As you can see, there are strong themes of career growth and development, feeling valued, and future outlook common across different organizations. 

Which High-Impact question should we focus on?

• Rather than trying to focus on all ten High-Impact questions, Quantum Workplace encourages teams to select one or two High-Impact questions to focus on in the months after the engagement survey. Within your list of High-Impact questions, we recommend choosing:
  • High-Impact questions that have opportunity for improvement, to further strengthen the team's engagement. 
  • High-Impact questions that are in a team's control of improving, to set the team up for success.
  • High-Impact questions that a team is passionate about focusing on, to ensure buy-in and shared ownership within the team.

What is favorability?

Favorability is the combination of responses that are either "Agree" or "Strongly Agree" on our six-point rating scale. 

For example, if a survey question has a favorability rating of 80% across an entire organization, then 80% of all respondents rated that question with either a 5 (Agree) or 6 (Strongly Agree).

Why are the numbers of respondents different for different questions? 

The number of respondents next to a question tells you how many employees responded to that specific question.

The number can vary by question if survey takers chose not to answer a certain question, or if they closed their browser before submitting their survey. Our reporting captures responses for all answered questions, whether the survey was submitted or not. 

Why is favorability the best way to look at the data?

Favorability refers to the number of Agree and Strongly Agree responses on your survey out of the total number of responses. 

We focus on favorability for several reasons:

• Favorability is inherently goal-oriented. Ideally, all employees would respond to each question favorably, so an important goal for engagement survey programs is to focus on the metric that is tied to an end goal of enhancing favorable perceptions. 
• Differences between favorability ratings are easy to act on and interpret because we know the number the employees associated with that rating. As an example- if a 100-person team achieved 75% favorability on a question, then we know that 75 people responded with a 5 or 6. Therefore, we also know that we need to improve the perception of the remaining 25 team members. 
• Favorability ratings are easy to discuss. Whether it's communicating survey results to an organization or managers discussing results within their teams, favorability is an easily understood metric regardless of an employee's statistical or analytical background. Results that are easy to interpret and discuss help promote more meaningful discussions around the team, departmental and organizational improvement. 
• Focusing on one number reduces confusion. Comparing, analyzing, sharing, and discussing one metric is easier and more consistent than focusing on multiple metrics. 

What does "Insufficient Response" mean?

"Insufficient Response" is labeled on reports where the minimum response threshold was not met. This is to protect employee anonymity. For data to be revealed for any group, five or more people within that group must respond for the data to be visible. 

What is the minimum response required?

To see results within our reporting site- we require at least five unique employee responses. This helps protect employee confidentiality and prevent users from filtering or slicing results down to only one employee.

What are Engagement Outcomes and High-Impact Questions?

Our engagement survey is comprised of two types of questions: Outcomes and High-Impact questions. 

• Outcomes are at the core of how we measure employee engagement. They help to reveal the current state of employee engagement within organizations. Outcomes are not actionable because they are targets that organizations should strive to maintain or improve. There are nine Engagement Outcome questions, although you are not required to include all nine questions in your survey.
• High-Impact questions are actionable questions that help determine what can be done to maintain, improve or otherwise drive outcomes toward higher favorability. 

Why do we not focus on uncertainty and unfavorability? 

Consider a survey question with 50% favorability, 40% uncertainty, and 10% unfavorability. Another question has 50% favorability and 50% unfavorability. Despite favorability being the exact same in both questions, the first has more uncertainty. The second question is polarized between favorable and unfavorable perceptions. These differences suggest that focusing on the entire spectrum of ratings is more important than focusing on just one metric. 

One metric can never show the full complexity of results. However, it also overlooks three important aspects of focusing just on favorability:

• Simplicity and consistency are crucial for effective discussions around survey results. Focusing on one metric not only reduces complexity but also makes it easier to interpret and increases the chance that more employees will contribute to the discussion around survey results. Some teams may prefer more complexity, but these are decisions at the discretion of each leader. 
• Responses that are less than favorable are less than ideal. Regardless of uncertainty and unfavorability, these become irrelevant as the goal is to maximize favorability.
• It's easy to over-analyze and get stuck in "analysis paralysis." A lot of time and effort can be spent trying to figure out which questions take priority, why a question had high uncertainty, etc., all before talking with employees. This strategy is well-intentioned but is often a distraction from the most important activity- creating open and consistent dialogue about questions/topics with lower favorability. 

Is there ever a situation where it's useful to focus on the six-point average?

Yes- focusing on the six-point average can be helpful if multiple groups have identical favorability ratings. For example, if two teams each have an 80% favorability, looking at the six-point average could show if there is a difference in that rating. These cases are more likely when analyzing results for small organizations (e.g. 50 or fewer employees) or small teams/departments (e.g. 10 or fewer employees). 

What do I do if I forget my password?

Forgot your password? Go to https://login.quantumworkplace.com and click Forgot Password?, enter your username (often your email address) and click Submit. You will receive an email to reset your password shortly afterward. 

If you're still having trouble logging in, contact us at support@quantumworkplace.com. 

What is the difference between question-level favorability and group-level favorability? 

• Question-level favorability: if a survey question has a favorability rating of 80% across an entire organization, then 80% of all respondents rated that question with a 5 or 6. 
• Group-level favorability: if a team has overall favorability of 75%, then we can assume that each employee, on average, responded with a 5 or 6 to 75% of survey questions. 

The underlying calculations in both of those examples are technically the same- they include a combination of responses that are either "Agree" or "Strongly Agree" on our six-point agreement rating scale.

Question-level favorability focuses on a specific question across employees. It's representative of the proportion of employees who responded favorably to that specific question.

Group-level favorability focuses on a specific group of employees across questions. It's representative of the proportion of questions that were responded to favorably by a specific group of employees.

What do the engagement profiles mean? 

• Engagement Profile Calculations: every survey taker is categorized into one of four profiles based on their average response to the six-point agreement questions on a survey:
  • Highly Engaged: average response of 5 or greater
  • Moderately Engaged: average response between 4- 4.99
  • Barely Engaged: average response between 3- 3.99
  • Disengaged: average response is less than 3
• Engagement Profile Definitions: 
  • Highly Engaged: Employees are strongly connected to their places of work. They are often advocates of the organization who stick with the company through its ups and downs and make an extra effort to achieve great results.
  • Moderately Engaged: Employees are favorable towards the overall workplace experience, but something is holding them back from being highly engaged. 
  • Barely Engaged: Employees have an indifferent attitude toward their workplace. These employees generally have decreased motivation to perform optimally and are at risk for retention. 
  • Disengaged: Employees are mentally and emotionally disconnected from their places of work. These employees may be negative and even disruptive with their concerns. Their critiques typically go beyond healthy feedback and don't necessarily share their concerns with the intention of improving the workplace. While disengaged employees are relatively rare, they can hinder the productivity of others. 

Why doesn't Quantum Workplace calculate statistical significance?

Statistical significance, numerically represented as the p-value (e.g. p=0.03), is not reported for engagement surveys or pulse surveys for the sake of simplicity and statistical soundness. Here are a few of our top reasons for not including this element in our survey reporting:

• The average employee does not have a statistical background. As such, a large portion of users would not know how to accurately interpret the meaning of a p-value, making it a number that causes confusion and misinterpretation rather than clarity and guidance.
• Statistical significance is not the same as practical importance. Reporting p-values could give users a false sense of direction or security, with the misunderstanding that p-values under or above a certain threshold are less or more valuable to pursue positive organizational change.
• Organizations are populations. Statistical significance is part of the branch of statistics known as inferential statistics. These statistics focus on being able to infer results about a population (e.g., a country, an industry) from a smaller group of that population because it’s rarely possible to get data from an entire population. However, in the case of census engagement surveys, the organization is the population. Response rates for our engagement surveys tend to be quite high ( > 80%), so there is little room, and therefore little need, for statistical inference. 
• Statistical significance can be strongly impacted by group size. In particular, results are less likely to be “statistically significant” with smaller group sizes, and more likely to be “statistically significant” with larger group sizes. This relates directly to the second reason listed above; users could be misguided by an artifact of statistical significance (e.g., group size) rather than being guided by practically important differences.
• The p-value as a measurement has come under severe scrutiny within the broader scientific community, and its use is becoming increasingly controversial. 

Instead of relying on statistical significance to determine whether a difference or change is important, we recommend focusing on relative differences or changes within your data. 

For example, if most survey questions increased in favorability yet a few decreased since the previous survey, then relatively speaking, those questions that decreased in favorability are more practically important to focus on. And more specifically, those questions that decreased the most should receive the highest priority. Likewise, if one department has especially low overall favorability, then that department is most important to focus on. And if all departments have similar yet low favorability, then that suggests a strong organization-wide effort is required.

What is a Commitment Plan Owner vs a Plan Observer?

Plan Owners are individuals directly responsible for creating, assigning and completing commitments. They have access to reports associated with the group of employees they're creating plans for. 

If you are a plan owner, you can add and remove plan owners and observers, as well as assign commitments to any employee.

Plan Observers get visibility to commitment plans through regular email notifications on plan progress. They do not have access to the reporting site. 

Commitment Holders are assigned a specific commitment within a plan.

What is the difference between Engagement and Best Places to Work benchmarks?

Within your Engagement Survey reporting site- you will see two categories of benchmarks: Engagement and Best Places to Work (BPTW).

Engagement benchmarks represent data from organizations that partner with Quantum Workplace to conduct an annual Engagement Survey. 

We recommend focusing on these benchmarks rather than BPTW benchmarks. 

BPTW benchmarks represent data from companies that participate in Best Places to Work contests in cities across the United States. This data tends to skew higher than Engagement benchmarks, partly because companies participating in BPTW are self-selected to be higher scorers on Engagement surveys. These benchmarks are often aspirational.