9 Ideas for Reviewing Vendor Work Samples

If you’re thinking of hiring a company or a contractor to develop custom curriculum for you, it’s common to take a look at other work they’ve done. Here are a few thoughts on how to do that effectively. To a large degree, they apply to both online and classroom programs (though they’re weighted a little more to online).

1. Define your motives. Why are you viewing work samples? What’s important to you and your program? What would rule a vendor in or out based on the samples you’re going to view? If you can’t define your motives, you’re just “window shopping” and likely won’t advance your selection decision in a meaningful way.

2. Ask the company, “What was your role in developing this program?” Did they do it all or just a piece of it? Once you know that, you can focus on the work they did and not spend time evaluating work someone else did on the project.

3. Evaluate the appearance in context. The program needs to look pleasing and professional to students, but it also needs to be instructionally effective. Don’t let the look—good or bad—keep you from evaluating the instructional design. Also keep in mind that many times the appearance is a function of the client’s branding or their previous learning programs rather than the creativity of the vendor. When you’re finding out what the vendor was responsible for (see the previous item), make sure to include appearance.

4. Probe the instructional design. You are an experienced student as well as a learning professional. Review the program from both perspectives. As a student, is the material clear? Does it respect your time? Are you learning from it? As a learning professional, is the instructional design intentional? Does it incorporate appropriate and effective instructional techniques?

5. Consider the learner engagement. This is a tricky one for online programs. It would be great if every program had the budget to use all the tools and techniques available to engage and interact with the student. But that’s not always the case. Evaluate the engagement for what it is. If it’s very simple Level 1 interactivity (based on the U.S. Department of Defense handbook “Development of Interactive Multimedia Instruction”), assume that’s what the client could afford and that the decision to restrict interactivity is appropriate for the content and audience. Then evaluate whether it was done well, is appropriate, and supports learning, retention and application. Similarly, if the engagement techniques are higher level and obviously reflective of a big-budget project, was the investment used wisely? Does it achieve engagement? Is it more entertaining than educational?

6. Understand the development tools. This item depends on whether you have the expertise to make the call. Assuming you do, by finding out what tools were used and how they were used you learn a lot about the expertise of the developer. Did they use the wrong tools (like a pair of pliers to drive a nail)? Or do they understand and have access to the right tools for the job. Evaluating this will give you the opportunity to learn whether they are “tool neutral”—that is, they have access to virtually any development tool and select one based on the instructional design or requirements of the client—or whether they only use one program and make every program conform to their tool and limited expertise.

7. Read the descriptive materials. Is the sample accompanied by a paragraph description or case study that details the background of the project as well as learning tools and techniques used? If so, analyze the descriptive materials to help you gain additional context of the sample. If there isn’t, ask the company if they can provide background material.

8. Understand that this isn’t your program. Good custom learning programs are unique. If you see one sample and determine that’s not what you want your program to be like, you’re missing the point. First, evaluate more than one program from a vendor so you can develop a sense for whether they have a style (which is good sometimes but not always) or whether they’re a chameleon and can make the program your style rather than theirs (most always a positive, but not 100% of the time). But unless you see multiple programs and they all look the same, don’t assume that’s how your program would look if you hired this vendor.

9. Understand that communication, responsiveness, positive attitude, expert advice, project management, overall partnership skills, and even price are also critically important. Work samples are valuable, but they’re only one factor. They’re usually more helpful as a qualifier than the deciding factor. Use them to help you determine which firms have the expertise to do your job. But then make your final selection based on which vendor you want to live with through the course of the project.

One final thought. Understand that custom curriculum is proprietary. It’s typically owned by the client, not the developer, and sometimes contains confidential and/or sensitive information. As such, custom curriculum developers should receive client permission to show you work samples. (Many don’t.) Ask if they regularly seek permission to show a client’s program. If they act like they don’t know what you’re talking about (or are evasive), chances are they’ll be cavalier about allowing others to see your program. If you will benefit from the exposure, great. If it would be harmful to you for a competitor, regulator, client or shareholder to see it, not so good. By the same token, appreciate that a quality vendor has some restrictions and probably can’t show you everything they’ve done, may have to restrict how and for how long you can see a work sample, and so forth.

What have we missed relative to reviewing work samples? What experiences can you share where work samples were helpful or not helpful in hiring a vendor for custom curriculum development?