Situation - what were our goals?

The DIA Design Guild has been developing a rolling research program to find out how to better support our apprentices. This initiative offers action items to the program directors on what could be improved by collecting feedback and generating insights from our apprentices’ experiences. An advantage of a rolling research project like this one is that we can compare one year’s research to last year’s to validate our findings and any changes made.

Based on our stakeholder needs and questions, we focused our research objectives in the following areas:

• The application process
• The interview process
• The orientation events

Tasks - what did we have to do?

To find areas of improvement for the apprentice program, we needed to know:

• The apprentices’ goals and expectations coming into the program.
• The strengths and weaknesses of our application, interview, and orientation processes.
• Our pain points and other aspects of our process that did not support our apprentices.

Actions - what did we do?

Our Team

We put together a team of UX Researchers and a Product Designer:

• Caroline Craner and Alex Forseth (UX Research Leads)
• May Ng and Oghogho Ekhator (UX Researchers)
• Jenny Ear (Product Designer)

We also had several mentors available that gave us feedback on our work and guided us along the way: Alesha Arp, Amy Silvers, Adam Shreve, Julia DeBari, Grace Lau, and Marc Ravelo.

Exploratory Research

We developed a survey designed to give us a high-level understanding of each apprentice’s experience of the program so far focusing on:

• Their onboarding experience with applying to DIA, interviewing with the program director(s), attending apprentice orientation, etc. 
• Their experience learning from and working with their mentors. 
• Their perspectives on the importance and relevance of the program’s subject matter.
• Their use of the tools and resources provided by the program.
• Their general program experience.

We had a 73% response rate to this survey. Our results were summarized and shared with stakeholders and used to develop the research plan along with the discussion guides for our interviews in the next phase of our research.

Interviews

Our interview protocol focused on the application and onboarding experience of our apprentices. However, our findings from our exploratory survey showed how the following areas were also important to our apprentices and may offer further insights into their experience along with our original focus areas:

• Mentorship
• Projects
• Apprentice expectations and goals
• The tools and resources available to the apprentices in the program

To make this accessible to our students who would be unable to attend interviews or did not feel comfortable sharing their opinions with an interviewer, we developed open surveys for them to share their thoughts anonymously and asynchronously.

We spoke with 11 apprentices in total. We reached out to each of the apprentices to explain the purpose of the interviews. For 64% of the apprentices, we were able to have in depth remote moderated interviews. For the other 36%, we were able to get responses to a survey with open ended questions around the areas we needed clarity around. 

Data Analysis

We analyzed the interviews, pulling quotes and notes into a spreadsheet. We removed Personal Identifying Information (PII) and marked the timestamps, developing general theme categories and distilled ideas to organize our data. Then, we transferred our findings into sticky notes in Miro for affinity mapping which allowed us to arrange our data and generate research insights.

Each sticky note was coded with the initials of the participant and a number to anonymize participant data and to reference the finding’s original context when needed.

We also realized this would be a great opportunity to learn how to use artificial intelligence (AI) to assist in data analysis. With new AI tools permeating society in different areas, we wanted to practice using it for UX Research in a responsible manner. We chose ChatGPT because it’s easy to access and is free.

After organizing our affinity notes on our own first and generating our own themes, we ran our affinity notes through ChatGPT three times. Between prompts we checked the results against our own and reorganized our affinity notes and themes. This proved useful in validating our created themes and gaining different perspectives for the same data.

We also used it to suggest strategic next steps for stakeholders to improve the apprentice experience within our main focus areas as we developed our own considerations.

Our experience with using AI for data analysis showed us that it works best as a tool in a tool belt. We found that it’s best used to validate affinity mapping done by a human, clear up confusion with data, and explore our data in another way.

Results - what were our insights?

Key Values

We separated the key points by our apprentices into seven main values:

Community

The job process can be lonely. Developing connections is valuable for the mental health of our apprentices, and ensures they will get the encouragement they need along the way.

Value-Alignment

DIA is committed to accessibility, diversity, and inclusion. We want to practice what we preach, and help our apprentices make powerful and lasting change in their communities and professional lives.

Professional Excellence

We want our apprentices to leave this program with a depth of understanding and practical experience in their field. We work closely with our apprentices to make sure they leave our program set up for career success.

Self-direction

Everyone’s professional path is unique. There is no one-size-fits all. That’s why we empower our students to create an apprentice experience that fits their unique needs and vision for their future.

Networking

The working world is a large web of possibility and connection. Helping our apprentices meet other professionals can spark exciting new conversations, open up new opportunities, and inspiration for next steps in their careers.

Guidance

Our apprentice program offers one-on-one mentorship opportunities for our students to develop supportive and ongoing relationships where they can get insights, resources, and advice from professionals in their field of interest.

Flexibility

Life is complicated, and many people are juggling the demands of jobs, school, taking care of family members, and even vastly different time zones. We create options and resources for our apprentices to allow them to complete their program on their schedule.

Archetypes

From our key values, we created seven apprentice archetypes that were designed to illustrate the apprentice experience for our stakeholders that would make it easier to design the program for their needs.

Structure Driven Shea

• Needs guidance and personalized support from professionals, organized resources, and structured project opportunities.
• Dislikes disorganized and overwhelming information, ambiguous priorities, and feeling confused.

Overcommitted Omar

• Needs bite-sized activities to do on their own, flexible learning options, and clear checklists on the next steps.
• Dislikes mandatory events without asynchronous options, feeling overwhelmed without support, and disorganized content and resources.

Job Driven Jade

• Needs experiences to sharpen relevant skills, job placement support, and learning aligned with career goals.
• Dislikes busy work, tasks unrelated to career growth, and too much theory without practical application.

Community Driven Caia

• Needs interaction with peers, collaborative opportunities, and authentic connection with others.
• Dislikes isolation in learning, project work without teamwork opportunities, and activities with no opportunity for engagement.

Network Driven Nora

• Needs building meaningful skill sets with peers, evaluation on their professional direction, and ways to learn from and connect with industry experts.
• Dislikes lack of opportunities to connect with others, limited exposure to peers, and lack of guidance and insight from those with industry experience.

Values Driven Vera

• Needs projects aligned with a mission they resonate with, activities with both an ethical and practical focus, and matched with a mentor aligned on values and a meaningful career.
• Dislikes organizations that do not practice what they preach, projects or learning without connection to their values, and lack of exposure to peers with similar values.

Adventure Driven Andre

• Needs projects that fit with specific career and life interests, inspiration to think of career possibilities, and a mentor match that allows them to create a career on their terms.
• Dislikes busy work with ambiguous rationale, mandatory events that do not interest them, and a lack of opportunities to tailor their education.

We used behavioral archetypes to avoid some of the pitfalls with using traditional personas. Traditional personas tend to rely too heavily on creating fictional demographic information which can lead to harmful side effects like stereotyping. Instead, we created archetypes based on traits, like needs, beliefs, values, lifestyle, activities, and interests, and our key values. Our inspiration came from a Games User Researcher, Steve Bromley.

Journey Map

Once we had the key values and archetypes, we illustrated each archetype’s experience in the onboarding phase of the apprentice program using a journey map.

For each phase of the beginning of the program, the journey map describes the goals of both the archetype and DIA, archetype actions and thoughts, and opportunities for DIA to tackle archetype pain points.

Our journey map plots each archetype along the path of the application process and onboarding process for the apprentice program. It moves through the different phases starting with discovering and researching the program, applying and waiting for a response, interviewing, attending orientation, and then moving forward into the rest of the program. Our journey map illustrates the positive, negative, or neutral experiences for each archetype based on their needs and pain points.

Presentation

To conclude our project, we created a presentation in Canva with our main findings, considerations, and deliverables. We met with key stakeholders including the Director of the Apprenticeship Program and one of the founders of DIA Design Guild. We also presented to members of the DIA community and mentors.

Our findings were well received and seemed helpful for our audience. They also gave us great feedback and follow up questions both during the presentation and later when we did a project retrospective to prepare for the 2024 research project. 

Next Steps

We gave the following considerations to DIA based on our findings:

• Address application waiting periods by maintaining a communication strategy meant to share resources and prepare the applicants for the program.
• Have apprentices define their goals and expectations around projects and mentors at the beginning of the program.
• Reduce an overload of initial program information by hosting multiple orientation events, each one focused on a specific part of the program.
• Find other ways for apprentices to connect at the beginning of the program for those who are unable to attend orientation.

The insights of this research are currently being used to design our 2024 apprentice research project.

Impact

Based on our research, the program director made some changes to their process. First, the interview process underwent some changes according to our findings and insights. Second, the program switched to a rolling acceptance process, rather than send out acceptances all at once.

The research project also affected us as researchers and setting up our 2024 research. We are taking what we learned from our 2023 research process and how to run our next research phase. Additionally, the findings from our 2024 research will help focus our research questions and goals moving forward.

Our Learnings

What Went Well

• Dedicated coworking hours helped us stay accountable and productive.
• Our deliverables felt impactful.
• Using Coda to organize our research materials and track progress went well.
• We had multiple mentors associated with DIA review our work and give us feedback helping us improve.
• We learned more about using ChatGPT responsibly and effectively for UX Research data analysis.

What We Need to Improve

Difficulty Narrowing Scope and Objectives

• There was so much to explore in our program, and as this was our first time diving in, we struggled to know where to focus our efforts first.
• We plan to improve our project scope and focus next time by developing clear objectives from our stakeholders, and then move through the list, one objective at a time.

Lack of Previous Infrastructure

• We did not have a clear blueprint of how to go about holding a research repository, having a clear information architecture, nor clear guidelines. The benefit was the freedom and depth of knowledge we were able to gain, but the drawback was that it was difficult to know where to start.
• With the support of our mentors and stakeholders, we are developing a rolling research plan that is tailored to the program's needs. We are currently building the information architecture and internal structure of our research repository, which will let us have a clearer starting place in future research plans.

Lack of Clear Project Management

• While we were skilled at research, there were meta skills that we were developing along the way - project management being one of the top ones. We struggled with keeping track of tasks to complete and timelines. For future projects, we now know to create clear goals for each role, and document our tasks and timelines in a central place like Coda.
• We are looking into structuring our upcoming research better and analyzing what systems work best for us while we adopt them. We will also empower apprentices to participate in the research process, from research planning to execution of methods and presenting results. This will free up the team leads so they can focus on managing the workflow, provide more direction, and set clear timelines.

Tool Stack

• Canva - Creating our presentation
• ChatGPT - Data analysis
• Coda - Planning and tracking our research
• Discord - Communicating with team members, participants, and stakeholders
• Figma - Creating wireframes and prototypes for our archetypes and journey maps
• Google Docs - Writing research documents and outlines
• Google Meet - Conducting interviews and meetings
• Google Sheets – Data analysis
• Loom - Storing and transcribing interview recordings
• Miro - Affinity mapping
• Tally - Creating and distributing surveys
• Zoom - Conducting interviews and meetings

Thank you for reading!

DIA Design Guild is committed to supporting and uplifting our apprentices through our apprenticeship program. We are grateful for the opportunity to do this research work to continue that. We are also grateful for our peers and mentors who support us along the way.