Assessment Management System
NAEP conducts assessments every year on millions of students across the US. Test administrators manage student participation and data, and monitor student progress and laptop battery health using an Assessment Management tool designed and developed by Fulcrum.
Overview / Challenges
The stakes for this system are very high. NAEP considers the loss of data for even a single student to be a failure. There are hundreds of test administrators using this system. Many are not tech savvy, and require a system that is easy to learn over the course of a 2-day training session.
End users were unavailable to us during the design of this system, so our work relied heavily on heuristic and task analyses. Unlike the other case studies in this portfolio, the information below is not presented in sequential order, but is a sampling of the information and interaction design tasks required for this project.
End users were unavailable to us during the design of this system, so our work relied heavily on heuristic and task analyses. Unlike the other case studies in this portfolio, the information below is not presented in sequential order, but is a sampling of the information and interaction design tasks required for this project.
Basic Logic Architecture
Test administrators have to document many events that happen during an assessment, and there is a set of complex rules that the system must use to evaluate the documentation entered into the system and guide the user to correct errors when needed.
Key factors considered were:
1. Which rules warrant a warning; which ones require a specified input.
2. How to provide warnings in a minimally disruptive fashion.
3. How to inform the user of a specific required input in a way that is easily understood
4. How to allow the user to escape the current documentation event to get the required information, without losing potentially-useful, but incomplete entries
1. Which rules warrant a warning; which ones require a specified input.
2. How to provide warnings in a minimally disruptive fashion.
3. How to inform the user of a specific required input in a way that is easily understood
4. How to allow the user to escape the current documentation event to get the required information, without losing potentially-useful, but incomplete entries
Documentation Entry Validation Logic
Task Analysis
Task analyses can contain only user actions, or they can add in automatic processes, and even mental calculations of the user. The bulk of the task elements in these task analyses are specific system behaviors that guide the users in completion of their task.
Mockup: Rule 1 (Warning)
Task Analysis: Rule 1 (Warning)
Heuristic Analysis
Other components of the management system were less complex, and relied on heuristic analysis to make usability recommendations.
Something as simple as changing a password can be a source of frustration if the system doesn’t provide sufficient guidance for the user.
I generally consider this type of work “low hanging fruit”. It’s fun and easy, but impresses people and has a big impact on the final product
Password Reset Email Sent - Before
Password Reset Email Sent - Recommendation
Visual and Interaction Design
Though the basic information organization system is a grid, there are multiple grids, which we decided to use tabs to access.
The grids generally look similar but have very different functions. Adhering to usability 101 principles, the tabs are arranged from left to right in the order they are used by the administrators.
The grids generally look similar but have very different functions. Adhering to usability 101 principles, the tabs are arranged from left to right in the order they are used by the administrators.
We also use redundant coding for information that needs to be processed quickly, such as the status indicators in the right-most column. Finally, we used a lower-contract color scheme that will reduce eye-strain on the administrators who use this system continuously throughout the testing day.
Fourteen mockups showing all grids, cell states, and popup windows were created, along with design specs for the developers to use in creating the final product.
Fourteen mockups showing all grids, cell states, and popup windows were created, along with design specs for the developers to use in creating the final product.
Final Design Mockups
