White Papers
Interested in learning more about the scientific theories and proven facts that drive Knowledge Factor’s Assessment and Learning products? Nerd.
Naked Truth About Learning Analytics
Case Studies
Whether you’re a Cliff’s Notes™ kind-of-person (no judgment here) or a 3-time reader of War and Peace (wow), our case studies are the perfect, pithy snapshots of client successes.
In the News
We’re not black and white, but we’re read all over.
Indiana Housing & Community Development Authority Adopts CBL
Knowledge Factor Announces Partnership with Hearst Newspaper
Professional Postgraduate Services Launches CBL-based Program
Events
Knowledge Factor is kind of like a ruby red cocktail dress at a black tie event - tradition-bucking, eye-catching and vibrantly hued.
We like to keep our finger on the pulse of the training industry, and that means attending tradeshows and networking within associations throughout the year. Make sure you look for us at the next tradeshow you attend. Stop by and introduce yourself. No dress code required.
Our Next Events Are:
AORN Congress in Denver, CO March 15-16
CompTIA Colloquium in Chicago, IL April 7-8
FAQs
If you’ve made it this far, we’re sure you have some lingering questions.
While we’re happy to get on the phone with you and discuss everything from the merits of our multiple choice answer format to ease-of-use within our authoring tool, we know that some people are visual learners and need to see the answers to their questions in print. Fine by us.
-
Shades of Knowledge Explained
You say Knowledge Factor shows the ’shades of knowledge’. How does it do this?
Great question! Instead of forcing a learner to answer either correctly or incorrectly, which shows you only that they are right or wrong, we give them the opportunity to identify their level of confidence in their answer. This gives us more insight into their ’shades of knowledge’ because their answers fall into 4 categories instead of 2. The four categories are: Mastery, Misinformation, Doubt, or Uniformed.
How can you say true and false is false?
We can for the moment, because true and false answers don’t give you the shades of knowledge. We said that a little tongue and cheek. What we really mean is that you shouldn’t limit yourself to black and white information or one-dimensional metrics. There is a place for one-dimensional metrics, they fit into level 1 and 2 evaluations … but to really begin measuring levels 3 and up we strongly feel that you need to have a more robust metric ground to stand on. And, wouldn’t it be nice if these multi-dimensional metrics were empirical reports versus subjective (and potentially biased) checklists and self-evaluations? We’ll let you decide!
If you’re going to build a house, it’s great to have the floor plans as a starting point. But you wouldn’t have a company build the house based on that, would you? No, you’d want blueprints and even a 3D representation of how the finished home would look. The floor plan is a great place to start, and is exactly what current assessments give you. Our tool takes that floor plan and brings it to life.
-
Guesswork
You say Knowledge Factor gets rid of guess work. How can it do this?
Knowledge Factor puts quite a few things in place to reduce guesswork. The first is our patented answer key. Since it gives learners the option to say, "I don’t know," they are more likely to be honest in their answers.
The second line of defense is our hidden scoring options and product algorithms. They allow the author to pick the number of questions to be pulled from the pool of total questions for each iteration and set definitions around ’mastery’ status for each question (# of times in a row you needs to answer a question correctly for it to be considered mastered).
Our complex algorithm randomizes the answers each time a question appears, and determines which question will be shown to the learner based on his or her previous answers and whether or not a question has been seen.
Expand Your Knowledge:
Why is getting rid of guesswork important?
Just because you guess right, doesn’t mean anything other than you got lucky! Traditional assessments and their results don’t take guessing into account. In fact, we’re taught to make ’educated guesses’ when we aren’t 100% sure about an answer. My question to you is: how often are you 100% about your answer? Of these times, how often are you wrong? You may say ’rarely’, but that means you’re still wrong some of the time even though you’re sure you were right. Then, what about the answers you aren’t so sure about? You guess! There isn’t much that is black and white about this process, except for the results you’re used to seeing. Knowledge Factor takes these nuances into account and the result is the shades of your knowledge.
-
Reports Showing Shades of Knowledge
What kind of data can be reported on in Knowledge Factor?
Just about anything, actually. You can report on the learner attributes you decide to use as part of the registration, you can report on the overall success and progress of your entire learner population, or you can focus on the individuals. You can also associate three layers of data to each question and report on these as well. We’ve found that our clients like the versatility of our reporting … whether you’re using it for competencies, coaching, or reporting on your departments’ progression towards set forth business goals, we are confident our reports will give you the information you need.
What are the 3D metrics Knowledge Factor talks about?
Yes, 3D reports and no special glasses required! Most assessment and test reports show you flat 1-dimensional metrics – pass/fail rates, correct/incorrect response rates, knowledge gaps identified based on the previous results listed. All of these results, in our opinion, are very black and white.
We give you much more – our metrics are 2-and 3-dimensional. Sure, we can show you who completed a module, but we also show you:
- Knowledge and Confidence – details the knowledge and confidence quality of your learners and your organization. (2-Dimensional)
- Learning Progress – shows you how your learners are progressing (or progressed) through each step of the learning loop – yes, you actually get a visual that shows how they are, in fact, learning. Imagine that! (3-Dimensional)
- Mastery – shows how your organization achieved mastery and, if you like, how they have retained that information over time. (3-Dimensional)
Why do the reports show that everyone got 100%?
We blow up the bell curve, honestly! We realize that not only are the results of traditional assessment black and white, but also most companies want to put their employees into a bell curve. Funny to think about, isn’t it? You only want some of your employees to be top performers? Well, we encourage clients to set the passing score for learning to 100%. You have the option to set it lower. But in all honesty, if you’ve developed training, is there anything in that training you don’t want your entire audience to know and take action on?
-
Learning Process
Why does Knowledge Factor start with a question instead of starting with the content?
First of all, this learning style is older than dirt. Ok, maybe not, but it’s been used for well over 2,000 years and is still a key teaching method in top Law Schools around the world. Why? Asking a learner a question creates an opportunity to engage the learner in critical thinking.
While our method isn’t 100% Socratic (that is a virtual impossibility online) we know that curiosity is heightened when people are posed with challenging questions. We leverage their curiosity. And when they are wrong? We are able to take advantage of their cognitive dissonance and turn it into a teachable moment they are more likely to remember!
Expand Your Knowledge:
http://en.wikipedia.org/wiki/Socratic_method
http://www.criticalthinking.org/articles/the-role-socratic-questioning-ttl.cfm
Why do people see questions more than once?
If you’re seeing questions more than once, it helps you store the information into your long term-memory. And, if you see those questions randomly presented, this further helps increase your long-term knowledge retention. This is based on the ’spacing effect’. With the repetition and random design of our methodology, learners are more apt to retain the information and apply it on the job. Their behaviors start to change because of this.
It also helps us decrease your ability to guess – because if you answer a question correctly two times in a row, it’s less likely that you guessed the correct answer!
Expand Your Knowledge:
Kerfoot BP. Learning benefits of online spaced education persist for two years. Journal of Urology, 2009 Jun;181(6):2671-3.
Matzie KA, Kerfoot BP, Hafler JP, Breen EM. Spaced education improves the feedback that surgical residents give to medical students: a randomized trial. Amer J Surg 2009: 197(2), 252-257.
-
Question Format
Why does Knowledge Factor only use multiple choice questions?
In all of life, there are tradeoffs. What we’re trading off here may be giving up the diversity of different question formats but what you’re gaining is so much more. You’re getting shades of knowledge and depth of reporting. The only way we can do that right now, terribly well, is to have this current methodology.
Right now, this is simply how our patented answer key was designed. Nothing more or less. We know that the eLearning training gurus have written countless papers on and given numerous webinars about creating online assessments that encourage interaction and various question types in order to assess all 6 cognitive levels of Blooms taxonomy. And we don’t disagree with the theories.
But even given this, we don’t think our system is ’broken’. Quite the contrary. There are several advantages of using only multiple choice questions. For example:
- You’ll become an expert MC question writer and your overall development time will decrease. (That’s right, instead of spending time trying to make your assessments ’diverse’, you’ll be able to focus that energy on making them solid).
- They give you the ability to evaluate everything from recalling information to making inferences associated with evaluation.
- They can be written as fill in the blank statements, or can contain answers that are either correct or the best option given a scenario.
Expand Your Knowledge:
Why does Knowledge Factor limit me to three answer choices?
Because we said so! No, really, it’s true … it’s simply the way our patented answer key was created and it’s the world we’ve been operating in since our inception. Also, we’re trying to take it easy on our product engineers, adding more answers into the mix might cause them to throw themselves overboard.
As you well know, writing plausible incorrect answers is key! Typically, when you have more than two incorrect answers, the others are ’throw aways’, and the learners don’t typically consider them as possible correct answers anyway. So, instead of spending time writing throw away answers, we help you focus on writing good answers.
Expand Your Knowledge:
http://testing.byu.edu/info/handbooks/betteritems.pdf
Paper: Bruno-Optimal Alternatives in Multi Choice
-
New Learning Process
How long does it take a learner to go through an Assessment or Knowledge Factor?
The number of questions in a module vary – typically we recommend about 30 or so questions per module to keep the average seat time to around an hour. Of course, the amount of correct information the learner knows will help him or her get through the learning more quickly. Also, the more honest the learner is when answering the questions will certainly help move them through the process more quickly.
Expand Your Knowledge:
No research on this -- est
-
Product Differences
What the difference between the Assessment and Learning & Mastery products?
The Assessment allows you to get a baseline of the shades of knowledge of the audience – individuals and the group. From this, you can determine where to focus training and coaching efforts.
Learning & Mastery ties the learning in – the learners iterate through the questions and explanations until they have mastered the information to fill their knowledge and confidence gaps.
-
Blending
How does the Assessment blend with other efforts our company is currently implementing?
The Assessment is a great tool that can be used to evaluate the progress your audience is making through existing training efforts. It’s also a terrific coaching tool because you can target misinformation and areas where the learner is lacking confidence to help them be more successful. You can also use the reports to help you determine how successful your current training efforts are and where you might need to add more information.
How does Knowledge Factor blend with other training initiatives our company is implementing?
Knowledge Factor is great to blend into areas where you need to ensure your audience has foundational information mastered so they can begin putting that knowledge into action. It’s also a terrific level-setting tool. By this, we mean that you can ensure that an audience with diverse knowledge and experience come into a case study or simulated role playing activity with equal knowledge and confidence. In these circumstances, the newer members participation increases and experts don’t perpetuate misinformation.
-
Development
How hard is it to take existing questions and put them into the Knowledge Factor format?
It’s actually quite easy! We have a series of quick trick papers for turning true/false, essay, fill in the blank, matching, and multiple choice questions with more than three answers into the Knowledge Factor format.
How easy is it to author in Knowledge Factor?
It’s so simple! Our authoring tool is so intuitive, even a novice can walk through it. We have editorial functionality that makes the authoring experience similar to using software products that most people are very familiar with.
What media does Knowledge Factor support?
We support audio, video, images, flash, document files … and even packages exported and published from Lectora, Articulate and Adobe.
What is the typical development time for an Assessment and/or Knowledge Factor?
So, the industry standard for ILT development is approximately 40 hours per 1 hour of seat time. And for e-Learning, the numbers typically go well above 60 hours per 1 hour of seat time. We have found that the typical development time is approximately 30 hours of authoring for 1 hour of learning.
Just like development for other modalities or in other tools, the development time will increase or decrease based on the complexity of the project you’re developing, the quality of existing content, the number of media assets, etc.
We don’t have the internal resources to get our content into Knowledge Factor. Can you help us?
We certainly can! We offer a variety of professional services and design and development of Assessments and Knowledge Factor Programs is one of them.
Can I author online? How about offline?
You can author online or offline. We have a wonderful template for offline authoring that uploads into our system. It even captures the formatting of text – Yup, you read that one correctly. We’re here to make this as simple as possible!
Our content changes annually. How easy is it to make changes to uploaded content?
Maintaining content is a cinch in our tool! Our media library allows you to organize your media assets into a file folder structure so you don’t need to sort through hundreds of images to find the right one. You can search and sort on a variety of fields to help you locate the exact question you want to edit … and then, poof! You can release a new version of the program, course or module to your employees.
-
Integration
We have an LMS. Can you pass results to our LMS?
Yes, we can! Knowledge Factor is AICC compliant and can be integrated with your LMS. We actually have an in house group of experts that have completed several integrations and they’d be more than happy to talk with you about your specific LMS and needs.