Developing Occupational Questionnaires

Developing Occupational

Questionnaires

2 2

Agenda

• Background and Context

• Steps in Developing Occupational Questionnaires

• Implementation Issues

Background and Context

4 4

Impact of Automation

• Has drastically changed how agencies recruit,

receive applications, and assess and select

candidates

• Has made it easy for applicants to apply

• Challenge: Paring down the number of candidates

while making consistent, accurate and merit-

based distinctions among them

5 5

Current Hiring Context

• Need for high quality hires to carry out Federal

agency missions

• High applicant volume

• Timeliness goals

• Lack of assessment and measurement expertise

at many agencies

6 6

• Assessment method used

to screen, rate, and rank

job applicants

• Commonly delivered

through automated

staffing systems used for

Federal hiring

• Consists of self-ratings of

training and experience

(T&E)

What is an Occupational Questionnaire?

7 7

Benefits of Occupational Questionnaires

• Positions can be filled quickly

• Relatively inexpensive and efficient

• Low burden and high face validity for applicants

• Wide variety of KSAs/competencies can

be assessed

• Easy to automate

• Test security is not an issue

• Occupational Questionnaires are familiar to

agencies

8 8

Challenges of Occupational Questionnaires

• Response inflation/deflation

• Lower validity

• Less differentiation among candidates

• Less suitable for entry-level positions

• Requires adequate time to develop

• Subject matter expert (SME) involvement is

needed

9 9

Setting Up for Success

1. Quality job analysis information

2. Trained and experienced occupational

questionnaire developers

3. SME input and review

4. Infrastructure that promotes quality and

accountability

Steps in Developing

Occupational Questionnaires

11 11

Topics Covered

1. Developing an assessment plan

2. Assessing minimum qualifications

3. Writing items

4. Selecting rating scales

12 12

1. Developing an Assessment Plan

• Identify content areas

(KSAs/competencies &

tasks)

• Identify a target weight for

each content area

• Set a target for the total

number of items

• Map the total number of

items

1. Developing an Assessment Plan

Step 1: Identify content areas

• Review job analysis data to select competencies

(and supporting tasks) by reviewing scores for:

• Importance – Higher ratings indicate greater

importance

• Required at entry – Lower ratings indicate the

competency is needed at entry (not learned on

the job)

• Distinguishing value – Higher ratings indicate

the competency is valuable for distinguishing

superior from fully successful employees

1. Developing an Assessment Plan

Step 1: Identify content areas, continued

• Measurable - Not all competencies (e.g.,

Integrity/Honesty) are best measured with a self-

report instrument

• Unidimensional – Only measure one competency

at a time (e.g., measure Oral Communication and

Written Communication separately)

• Job domain coverage – Try to capture the full

range of general and technical job requirements

1. Developing an Assessment Plan

Step 2: Identify target weight for each content area

• Weights should equal 100%

• Recommend equal weights for each content area

• Use job analysis or subject matter expert input to

determine if any areas are more important and

should be given more weight

1. Developing an Assessment Plan

Step 3: Set a target for the total number of items to

assess

• Consider the following:

• Number of content areas that need to be

covered

• Complexity of the content areas

• Applicant burden

• General guidelines:

• No fewer than 10 items

• No more than 40 items

• 40 items take approximately 20 minutes to

complete

1. Developing an Assessment Plan

Step 4: Map the total number of items

• Determine the total number of items per content

area

• Issues to consider:

• Complexity of the content areas

• Target weight for each content area

1. Developing an Assessment Plan

Sample Completed Assessment Plan

HR Specialist GS-11

Content Area # of Items % of Items

Target

Weight

Content Area 1 3 20% 20%

Content Area 2 3 20% 20%

Content Area 3 3 20% 20%

Content Area 4 3 20% 20%

Content Area 5 3 20% 20%

Total 15 100%

Note: These numbers may change as you begin creating the questionnaire.

19 19

2. Assessing Minimum Qualifications

• Use Yes/No or multiple choice questions

• Use clear, unambiguous language that any

applicant can understand

• Include response options for both qualified and

unqualified applicants

• Describe experience requirements at each grade

level identified, such that distinctions between

grade levels are clear

• Describe education requirements, including

references as appropriate

• Provide a link to the Qualification Standard and/or

Individual Occupational Requirements (IOR) as

appropriate

20 20

3. Writing Items

• Goal is to write the item in terms of specific,

observable, and verifiable behaviors

• Crafting quality items requires time, effort, and

practice

• Pick a strong action verb

• Writes vs. Develops

• Assembles vs. Prepares

• Should be written so that a colleague or

supervisor could verify that the applicant

performed that behavior

3. Writing Items

Follow the Formula:

Perform

What?

(Action Verb)

To Whom or

What?

(Direct Object)

To Produce What?

OR Why? How?

(Qualifying

Statement)

+ +

Examples:

Sort + incoming mail + into functional groups

for distribution.

Proofread and edit + letters, memos, e-mail or

other written communication documents +

to address format or grammatical, spelling, or

typographical errors.

3. Writing Items

Item writing tips

• Write items in behavioral, observable, and

verifiable terms

• Write clearly and succinctly

• Develop items that meaningfully distinguish

among applicants

• Keep your competencies unidimensional

• Consider the level of job-specific experience

required for the position

• Emphasize quality over quantity

3. Writing Items: Pitfalls

Wordy:

Relay various types of written information and

communications, both in e-mail and memo format, to

individuals at all levels of the organization and on

multi and varied topics.

Concise:

Write correspondence (e.g., e-mails, memos) on

varied topics for distribution to individuals at all

organizational levels.

Unnecessary words add to the length of an item,

do not add meaning, and can actually make the task

seem more complicated that it is. You want your

items to be concise so they are easily understood.

3. Writing Items: Pitfalls

Double-

Barreled:

Assemble quarterly performance data from internal

office sources into a computer spreadsheet and

prepare a written report.

Two

Statements:

Assemble performance data (e.g., quarterly,

monthly) from internal office sources into a

computer spreadsheet.

Write a report summarizing trends in performance

data for internal decision making purposes.

Double-Barreled items include multiple tasks or

competencies, thus forcing applicants to choose

which part of the statement they will respond to,

giving you incomplete or ambiguous information as

to what experience the applicant has.

3. Writing Items: Pitfalls

Overly

Specific:

Add, subtract, multiply, divide, and compute

percentages to calculate pay, time, and benefits.

Appropriate

Level of

Generality:

Use addition, subtraction, multiplication, and

division, and compute percentages to perform basic

calculations.

Overly specialized items, or items based on job-

specific task statements, may unnecessarily omit

qualified applicants. Although there may be certain

specialized or technical expertise that is required,

experience in other job settings may be equally as

valuable, particularly when assessing general

competencies.

3. Writing Items: Pitfalls

Evaluative

Wording:

Perform all required quality control procedures

on time and attendance reports in an effective

and thorough manner.

Behavioral:

Verify the accuracy of data or information in a

report or database following established

procedures.

Vague, subjective, or evaluative wording can

cause the applicant to incorrectly interpret the tasks

or statements. Statements that are clear and

specific allow applicants to more accurately rate

themselves.

3. Writing Items: Pitfalls

Abbreviated: Prepare MOUs to secure new contracts.

Written Out:

Prepare proposals or Memorandums of

Understanding in order to secure new

contracts.

Abbreviations/Acronyms may not be understood

or hold the same meaning for all applicants. It’s

better to spell out all abbreviations and acronyms to

ensure the item is interpreted the same way by all

applicants.

4. Selecting a Rating Scale

• Types of Scales

• Generic vs. Customized

• Pros and Cons

• Various Formats

• Additional Tips

4. Selecting a Rating Scale

Generic Rating Scales

• Can be used across items regardless of the

content area being assessed or the position

being filled.

Customized Rating Scales

• Are specific to each occupation, grade level,

and/or item.

30 30

4. Selecting a Rating Scale

Generic Rating Scales

Pros

• Relatively inexpensive and efficient

• Measurement expertise is not required

• Positions can be filled quickly

• Can be used across items

(regardless of content area or position)

31 31

4. Selecting a Rating Scale

Generic Rating Scales

Cons

• Experience-based scales may result in less

discrimination among candidates for entry- and

high-level positions

• Experience-based scales may be less appropriate

for entry-level positions

• Transparency in terms of the ‘best’ answer

• May result in lower level of applicant engagement

in responding

32 32

4. Selecting a Rating Scale

Customized Scales

Pros

• Response alternatives are specific to each

occupation and grade level (increases face

validity for applicants)

• SME involvement leads to greater question

validity and upper management level buy-in

• Greater applicant differentiation

• Can be developed so that they are less

transparent to applicants

• Higher level of applicant engagement

Customized Scales

Cons

• Response alternatives are specific to occupation

and grade level

• Response alternatives are more difficult and time

consuming to develop

• Requires measurement experts to develop the

items

• Requires extensive SME involvement (dependent

upon availability, interest, and competence of

SME)

• Requires periodic review to ensure the questions

are still applicable to occupation

4. Selecting a Rating Scale

Various Formats

• Yes/No

• Multiple Choice

• Forced Choice (Ranking)

• Select All That Apply

35 35

Additional Tips:

• Goal: Help applicants be as accurate as possible

when making their self-report ratings

• Ensure that items and rating scales are focused

on clearly stated, observable and verifiable

behaviors

• Use specific and unambiguous language

• Use rating scales that are anchored by clear,

verifiable, and behaviorally based descriptions of

what each response option means

4. Selecting a Rating Scale

Implementation Issues

37 37

Design a Verification Process

• Manual verification of

supporting materials by

trained HR professionals

can provide a valuable

check on the accuracy of

self-report ratings

• Minimum qualifications

• Education

• Ensure a standardized

and consistent process

• Consider use of follow-up assessments (e.g.,

structured interview, work sample, reference

check) to verify competencies of top applicants

• Notify applicants in advance of agency verification

policy, as well as use of follow up assessments

as appropriate

• For example: ‘If, after reviewing your resume

and/or supporting documentation, a

determination is made that you have

overstated your qualifications and/or

experience, you will be removed from

consideration or your score will be lowered.’

Design a Verification Process

39 39

Sample Verification Statement

I certify that, to the best of my knowledge and belief, all of the

information included in this questionnaire is true, correct, and provided

in good faith. I understand that if I make an intentional false statement,

or commit deception or fraud in this application and its supporting

materials, or in any document or interview associated with the

examination process, I may be fined or imprisoned (18 U.S.C. 1001),

my eligibilities may be cancelled, I may be denied an appointment, or I

may be removed and debarred from Federal service (5 CFR part 731).

I understand that any information I give may be investigated. I

understand that responding “No” to this item will result in my not being

considered for this position.

A. Yes, I certify that the information provided in this questionnaire is

true, correct and provided in good faith, and I understand the

information provided above.

B. No, I do not certify the information provided in this questionnaire is

true, correct and provided in good faith.

C. No, I do not understand the information provided above.

40 40

Evaluating the Process

1. Were the items on the occupational

questionnaire clear and easily understood by all

applicants?

• Were applicants confused by any of the items,

rating scales, or response options? Can you

improve the questionnaire based on the

questions you received?

2. Did the occupational questionnaire items help to

make meaningful distinctions among applicants?

• Are the scores spread out or clumped

together? Are too many scores in the high or

low range?

41 41

Evaluating the Process, Continued

3. Did scores on the occupational questionnaire

help to identify the most highly qualified

applicants?

• Obtain feedback from the HR Specialist who

reviewed the applications and from the hiring

manager.

4. Is there a high return-on-investment associated

with use of the occupational questionnaire?

• Do the benefits of using the questionnaire

(e.g., better-quality applicants) outweigh the

cost of developing and implementing it?

42 42

Infrastructure

• Establish clear standards of quality

• Train HR professionals in good assessment and

item writing practices

• Arrange for SME involvement in the development

and review of every occupational questionnaire

• Create an environment that promotes quality

and accountability

• Centralize or designate certain individuals to

develop occupational questionnaires

• Evaluate results

43 43

Red Flags

• Low score variability

• Indicates applicants are answering the

question the same, negating the purpose of the

questionnaire

• Returned certifications

• No selection is made

• Hiring manager dissatisfaction

• High volume of applicant inquiries

44 44

One Final Thought

“The ultimate goal of staffing is to make good hires

as efficiently as possible. It is not to make bad

hires quickly.”

- Maximizing the Value of Your Investment

in Qualifications Screening Technology.

Rocket-Hire. Feb 2004.

Contact Information:

[email protected]