Simplifying AI Model Optimization
0-1 UX research and design leadership to transform a highly technical trial and error process into a simple visual cloud-based application, saving developers thousands of hours of time testing and optimizing AI models for deployment onto Edge hardware.
The Problem & Initial State Of Experience
To understand how AI developers performance test and optimize AI models, dozens of AI developers working for vertical market companies developing edge device AI solutions with existing solutions were interviewed
This initial research discovered the following challenges with current workflows:
- Measuring and tuning the performance of AI models has a steep learning curve and is perceived as a black art.
- Reaching optimal performance with trade-offs is experimental and time consuming.
- Interpreting performance of AI models on target hardware with existing command-line based tools is cumbersome.
- Comparing AI model performance on different hardware configurations is highly desired but difficult to accomplish and requires custom testing methods.
Outcomes & Next Steps
Define user journeys (existing versus desired).
Communicate generative research insights with product team and collaborate cross-functionally to establish initial design requirements.
Test design assumptions and understand technical and business constraints.
Core Features Desired From Initial Feedback
- Establish a web-based tool that is able to import AI models and sample datasets and rapidly identify inference performance of the model on different hardware configurations.
- Visualize AI model topology to aid in model analysis
- Automate metric analysis for optimal performance configuration (streams, batches, latency)
- Enable experimentation with int8 calibration for optimal tuning
- Provide accuracy information to identify trade-offs when tuning for performance
- Provide direct access to open source Model Zoo in addition to custom model imports.
UX Definition
Feedback from initial discovery research coupled with internal stakeholder discussions provided the context needed to define the product solution and kick off a design process with a cross-functional agile team. The design process I followed roughly looked like the following:
The UX design and research process was iterative with incremental levels of fidelity with user studies to inform the process in between phases.
Concept Testing
Early wireframing allowed us to test design assumptions and help the team align on product requirements. Preliminary user testing with small groups of internal and external participants provided design insights early in the process.
Prototyping
A full clickable prototype was developed using Axure. Iterative changes were subsequently made to the design based on usability assessment criteria and task-based feedback. UX requirements were documented in Jira for the frontend development team to implement in code.
Functional Application
Once the application was developed based on agreed upon UX requirements, summative user testing was applied to validate UX quality against technical trade-offs made and to identify UX defects prior to release.
Prototype Testing - Early Feedback
Initial wireframes were tested with AI developers to elicit design requirements and test workflow and data comprehension.
Detailed User Research Reports
Usability test reports contained detailed feedback from participants which were applied to prototype revisions and requirements documented in JIRA for the development team to begin building.
MVP Developer Experience Assessment
Once an MVP prototype was developed by the team and validated, a more thorough developer experience assessment was conducted to holistically measure the end-to-end UX readiness.
Formative usability studies were conducted with iterative prototypes and results were summarized via a UX MVP score that was calculated with the following metrics:
1. Task completion rate and time on task
2. Subjective assessment data from users performing each task
3. Developer ratings from system usability scale (SUS) performed after all tasks were completed.
2. Expert assessment rating from SUS
Task Evaluated
1.Import a model from the Model Zoo
2.Import a custom model that needs to be converted
3.Import validation dataset corresponding with model type
4.Execute baseline inference targeting the CPU
5.Run an additional experiment with parallel streams
6.Quantize the model to Int-8 precision
7.Compare performance between FP16 and Int-8 models
8.Check for Accuracy
9.Deploy a package.
UX Design Audit Prior To Launch
Heuristic Evaluation
Expert reviews based on UX best practices and heuristics were delivered to the product team prior to product launch once a fully functional beta quality MVP was developed. Issues and recommendations were documented in JIRA as bugs or development requests with different levels of granularity depending on the audience (product executives, development team, UX design team).
UX Research & Design Process Summarized
An iterative UX research and design approach was applied to guide the software team from product definition to MVP launch. The UX research process employed is summarized as follows:
Foundational research consisting of contextual inquiry, stakeholder & subject-matter expert interviews, task analysis and comparative benchmarking.
Recruit AI developers for qualitative research on AI developer tasks, tools and workflows. Define user journeys (existing versus desired).
Communicate generative research insights with product team and collaborate to establish intiial design requirements.
Test design assumptions and constraints with user stakeholder interviews.
Based on an understanding of underlying needs. Design is informed by expert walkthroughs, personas, feature prioritization, business objectives and an understanding of technical constraints.
- Create conceptual prototype to establish initial design stimuli for user testing.
- Recruit, moderate and evaluate design mockups with AI developers for user feedback.
- Ratify design requirements as plan of record and negotiate trade offs with engineering team development.
- Develop high fidelity prototype with realistic design guidelines and ‘clickable’ end-to-end simulated functionality for usability testing.
- Usability test design prototype and establish validated requirements for MVP product development.
- Usability test functional beta build to pinpoint areas of UX deficiency due to technical workarounds or developer deviations from design specification.
- Perform detailed heuristic evaluations and communicate strategic UX design objectives to prioritize future feature innovations and iteratively improve UX quality. Informed by UX assessments, summative usability results, heuristic evaluations and longitudinal beta testing.
- Manage UX defects identified in JIRA and advocate for prioritization and iterative improvements.
- Define UX roadmap.