How We Ensure The Integrity Of The Data We Report Upon?

Both strategic and tactical actions and decisions are likely to be taken by organisations to improve their levels of engagement based upon these reports. When this is considered, it becomes apparent how critical the integrity of this data is to ensure an organisation can precisely pinpoint the issues and where they lie. 

This data may also be used in the process of determining Best Companies Accreditation, in addition to rankings on one or more of the Best Companies to Work For lists. Without reliable underlying data, these accolades would become worthless. 

Providing that each respondent understands that what is required of them is their candid and honest personal opinion on each question, data issues should be extremely rare. 

Issues may however arise for a number of reasons: 

• Technical: A fault in the data collection or processing of survey responses 

• Erroneous: Where a respondent has misread or misunderstood a question, or has incorrectly recorded their opinion when selecting an option on the survey. A variety of factors such as allowing insufficient time to complete the survey and the respondent’s facility in the language used for the survey may contribute to such errors   

• Influenced: Where an additional psychological influence has been present 

The influences noted above may occur for a variety of reasons: 

• A respondent may have an extreme feeling about their place they work. That feeling may override their desire to answer each question independently. They may answer questions based purely on their overall feelings and may allow these to override their views in relation to the particular question under consideration 

• A misplaced feeling of loyalty may lead a respondent to do anything in their power to help their organisation achieve higher scores as they know that improvement in scores is a key objective of the organisation 

• A respondent may feel pressurised to submit a survey or just want to complete in order to increase their response rate and for some reason such as having limited time, they may answer without reading the questions 

• An overriding feeling of animosity, could lead a respondent to attempt to do anything in their power to reduce the overall score of their organisation 

• Respondents may have been coaxed, cajoled, pressured, induced or bullied by another party such as: 

• a work colleague with an ulterior motive or grievance 

• a manager in competition with other managers wanting to demonstrate how well they and the people they are responsible for are performing 

• the leadership of the organisation, who may want the organisation to be highlighted as a beacon of best practice  

It would be counterproductive to detail all of the tests and checks that are in place to ensure the integrity of the response data. However, the following information provides some insight into some of the measures we take to ensure that our calculations and reports are based on reliable data. 

Several statistical indexes are calculated to identify surveys with response patterns that are unusual or unlikely – either in the context of the rest of the data from the organisation or in the context of all the data we receive. These are used to flag surveys and groups of surveys that contain answer patterns that are particularly different from the norm. 

A variety of statistically improbable survey response patterns are also scanned for. 

The results of each of these procedures contribute to an overall health-check on the quality of data received. Thresholds for issues identified determine which data may be flagged for further analysis. However, whether flagged for further analysis or not, surveys containing certain patterns of answers will automatically be excluded. 

Surveys which are excluded as a result of these procedures do not contribute to an engagement score or to an organisation’s response rate. 

To find out more about how we keep responses anonymous, this video explains in more detail ...