«

Q&A: Microsoft Improves Certification for IT Professionals with
Performance-Based Testing

An innovative new testing system for Microsoft Certified Professionals
uses simulation technology to more effectively evaluate the real-world
skills of IT professionals.

REDMOND, Wash., Feb. 1, 2005 -- The Microsoft Certified Professional
(MCP) program is one of the most successful certification programs in the
industry with over 1.7 million individuals credentialed all over the world.
The MCP credential is for professionals who have the skills to successfully
implement a Microsoft product or technology as part of a business solution
in an organization.

Microsoft wants to offer certification that helps assure MCPs are
experienced and qualified to help with their IT problems. But traditional
multiple choice tests don't always do a complete job of testing IT
professionals' ability to perform practical, real-world tasks end to end. So
the Microsoft Learning team has developed a new system of testing that
evaluates on-the-job skills of an IT pro.

Reinforcing its commitment to the MCP community, Microsoft is about to
incorporate performance-based testing in all core Windows 2003 exams.
Through simulated environments, IT professionals taking exams for
certification will now be required to demonstrate their knowledge of
products and processes they may encounter on the job. This dramatic change
in the testing protocol is designed to increase the value and credibility of
the MCP credential.

To learn about the new testing methodology, PressPass spoke with Al
Valvano, Lead Product Manager with Microsoft Learning, and representatives
of the two exam delivery partners participating in the program -
psychometrician Paul Jones, an expert on testing with Thomson Prometric, and
Randy Trask, VP of Market Development for Pearson VUE.

PressPass: What is performance-based testing?

Valvano: It's testing by doing. That's the definition used by the
Performance Testing Council, and it's a good one. It tests for an
individual's ability to perform a task as opposed to recalling information.
Think of it in terms of driving a car. If everybody who got their driver's
license only had to take a knowledge-based written test without the
behind-the-wheel road test component, they would not be proving their
ability to drive a car. The driving test adds validity to the written exam.

Training in IT today often employs lab-based exercises to reinforce
learning. It's evolved to the point where we use a lot of simulations. And
now we want to insure that when an individual takes an exam to be certified
in a particular technology, they're being tested by doing. It's a validation
of their ability to accomplish given tasks.

Jones: With only multiple-choice items in an exam you can't adequately
address certain constructs, which is the knowledge in the person's head that
you're trying to assess. You leave out components of the ability that you're
trying to measure. For that reason, simulation nicely complements the
multiple-choice exams.

In multiple-choice you have four or five items from which to choose.
But in a performance-based item, you have free play in an environment that
works like the real world. You solve a problem and you show how to get from
a starting point and navigate through a process. In addition, you must show
how to manage the whole problem-solving effort and demonstrate the results.
What performance-based testing adds is the components of self-management,
monitoring, problem solving and planning a solution - which gives the
testing program enhanced validity.

PressPass: Why is Microsoft incorporating performance-based testing
(simulation) into its certification program now?

Valvano: Because we've found that performance-based testing produces
more skilled candidates. And we're hearing from both Microsoft Certified
Professionals and IT hiring managers that there is greater benefit to them
in performance-based testing over traditional testing. We're always looking
for ways to add value for our customers and including simulations in our
exams increases the value of certification to the MCP community.

Testing with simulations raises the bar by requiring candidates to
actually perform tasks to accomplish a specific goal. Candidates have to
prepare for certification by really using the products. Memorization won't
cut it - you've got to develop the necessary skills. And that will help
differentiate you from other professionals and improve your career
opportunities. Hiring managers gain confidence that our certifications mean
that potential employees can do the jobs they need them to do.

Jones: The good thing about testing with simulations is that it
stimulates candidates to understand how to do things. And if that's your
focus as a student, your preparation is going to help you in the workplace.
Exam preparation is more appropriate to what you're trying to learn. In the
measurement literature, that's sometimes called "systemic validity." That
is, it's pushing the whole certification system in the right direction.
Candidates are focusing on the right kind of training and preparation.

PressPass: Is Microsoft taking any kind of unique approach to
performance-based testing?

Valvano: We're using a very robust simulation technology. Most other
programs that use simulations as an integral part of their testing
experience are for high-end certifications. They're typically live
applications. In other words, you come into a lab and you're given a server
and some hardware and specific problems to solve. You're probably evaluated
by an observer as well as with some automated scoring.

The problem with live application testing is that has limited scale.
It's very difficult to roll out on a global basis. So you're limiting your
audience. The Microsoft certification program has much greater reach, with
thousands of people taking exams every single day all around the world. Now
simulations will be available to everybody that can currently take our exams
and they're offered in eight languages: English, simplified Chinese, French,
Spanish, German, Korean, Japanese and Brazilian Portuguese. Everybody who
wants to attain a Microsoft certification will have the opportunity to have
the same innovative testing experience.

PressPass: Where do people go to take the certification exams?

Valvano: Customers can take the exams from either of two exam delivery
partners, Pearson VUE and Thomson Prometric. The simulation exams are going
to be available at more than 6,000 test centers around the world.
Implementing simulations into exams that are globally available has proven
to be a challenging endeavor for us. We've made the commitment to
performance-based testing and working with our partners, we've assembled an
incredible team to do it.

Jones: It's no cakewalk to do performance testing. It's more
complicated to develop and more expensive in every way. It sounds really
great to people who have never done it but any company that is implementing
performance testing technology wants to be sure that it's being targeted at
a place where there's going to be a good return on investment.

PressPass: How are you getting the word out to IT professionals about
the enhanced testing program?

Valvano: A page will be posted for our customers on the Microsoft
Learning Web site describing the simulations and the exam technology. In
March, they can also access a demo of the simulations and preparation
guidance. The information is also going to be on the secure MCP Web site
that is only available to those who are credentialed and in the monthly MCP
newsletter. And then there are the thousands of customers currently going to
Certified Partners for Learning Solutions (CPLS) for training and
certification. We've already communicated with the CPLS network about our
testing improvements and provided them with resources including videos to
talk about simulations with their customers.

PressPass: Which tests incorporate simulations?

Valvano: We're initially introducing simulations in the core exams for
both our Microsoft Certified Systems Engineer (MCSE) and Microsoft Certified
Systems Administrator (MCSA) certifications. These two exams - 70-290
Managing and Maintaining a Microsoft Windows Server 2003 Environment and
70-291 Implementing, Managing and Maintaining a Microsoft Windows Server
2003 Network Infrastructure - will be offered with simulations by March 31,
2005. Simulations will launch in many other exams throughout 2005. Look for
a release schedule on the Microsoft Certification Web page in March.

PressPass: Is there any incentive for existing MCPs and other
experienced IT professionals to take the exams with simulations?

Valvano: MCPs who have already taken a non-simulated version of 70-290
or 70-291 are not required to take the revised test but they may choose to
refresh their credentials with the performance-based criteria. Retaking the
exam with simulations can help an MCP refresh his skills and perhaps make
him a bit more marketable.

PressPass: If a person has already completed some of the exams leading
to certification, do they need to retake a version of the test with
simulations?

Valvano: If they have passed 70-290 or 70-291 by the launch date of
March 31, they don't need to retake the exams. But if they have not yet
taken these exams they must take the version with simulations. That's the
only form of the test that will be available after launch.

PressPass: What's the best way to prepare for a performance-based
exam?

Valvano: First, it's important to note that we will still have a range
of item types to measure the full spectrum of a subject: case studies,
multiple choice, hot area, drag and drop, and others that an MCP must know
thoroughly - so hitting the books is still something test takers will want
to do.

For the performance-based items, we suggest looking at the "Skills
Measured by this Exam" section of the exam prep guidelines. Make sure that
you can perform the tasks noted in the objectives such as "Monitor, manage,
and troubleshoot access to files and folders."

Simulations will generally be associated with objectives that require
the candidate to perform a specific task. Spend time in the environment
setting up situations that are called out in the exam objectives. Lots of
interaction in the environment will significantly contribute to a
candidate's ability to understand the environment and accomplish specified
tasks. So practice, practice, practice!

Trask: Certification exams have always been about demonstrating that
you have what it takes to perform a job. This really means that once
certified, you should be able to hit the ground running in a new job. If you
have the confidence you can get behind a computer and get the job done -
you're prepared, because that is exactly what Microsoft is going to ask you
to do!

PressPass: How will this performance-based testing initiative impact
IT organizations?

Valvano: IT professionals who achieve certification where
performance-based testing is part of the exam have demonstrated real world
competency and the ability to perform tasks required by a specific job role.
This makes IT professionals better qualified to resolve problems faster and
that translates into real cost savings and improved technical readiness.
Performance-based testing hones and validates skills. The IT organization
benefits because with highly skilled IT professionals it is better
positioned to provide a higher level of service to its internal and external
customers.

Trask: It's all about instilling confidence in IT managers that
Microsoft certification is a true and meaningful indicator of someone's
ability to perform a job. Whether used as a barometer for smart hiring, or
with existing staff to increase the likelihood of having successful IT
projects, like an Active Directory deployment, adding the simulations will
give IT managers the added assurance that Microsoft certification can be
used as a factor in making decisions.

PressPass: What's it like to take one of the exams with simulations?

Valvano: It's a "wow" experience. I've taken a lot of exams throughout
my life - from school, into college, and then in my professional career -and
this is a very different thing. Simulations incorporate many domains of
skills in a single question. You're in a simulated environment and the
simulation records all the steps you take to solve the problem. Your score
is based on the path you take and the outcome you achieve. It's directly
connected to what you do on a day-to-day basis in IT. What's really being
tested is: Do I know how to do this?

Trask: Microsoft is to be applauded for including real-world
simulations within their certification exams. Demonstrating that you can
complete tasks offers a level of assurance we believe IT managers will
welcome with open arms.