Mathematical proficiency is critical for high school students, and metacognitive knowledge plays an important role in supporting this development (Desoete and De Craene, 2019). Metacognitive knowledge, which refers to an individual’s awareness of personal learning processes, enables students to monitor their progress, adjust learning strategies, and strengthen their problem-solving skills (Pintrich, 2002; Radmehr and Drake, 2020). However, in classrooms with diverse learning needs, providing timely and personalized support that responds to individual learners remains a challenge.
Advancements in generative artificial intelligence (GAI) offer new opportunities to address this need. This proposal introduces MathPal, a user-centered conversational AI agent created to support math learners through metacognitive prompts and conceptual guidance. As GAI tools receive increasing attention in education, it is essential to ensure their usability, trustworthiness, and alignment with actual classroom practices before they are widely implemented (Bahroun et al., 2023). This proposal, therefore, presents the user-centered development process of MathPal and evaluates its features through usability testing with both educators and students. Insights gathered from these users will guide ongoing refinements and inform future iterations of the tool.
MathPal serves as a browser extension that is compatible with various digital math platforms used in the participating schools. MathPal's core functionality in supporting math problem-solving is structured in relation to growth mindset and metacognition knowledge (Pintrich, 2002). It provides detailed explanations for specific mathematical concepts through a conversational interface (see Figure 1), offering practice questions and problem-solving strategies without prematurely revealing solutions. By delivering step-by-step guidance and alternative methodologies, the tool facilitates personalized learning tips tailored to individual student needs. Furthermore, MathPal is designed to redirect off-topic discussions by strategically linking student interests back to relevant mathematical principles, thereby enhancing engagement while maintaining instructional focus (see Figure 2).
Figure 1
Interface of MathPal
Figure 2
The Redirected Conversation from Music to Math
To evaluate MathPal’s effectiveness, two rounds of usability testing were conducted in Spring and Fall 2024. The first usability testing included one 9th-grade math teacher and a district supervisor of Professional Development and Technology Integration. They tested both the conceptual and metacognitive support of MathPal. The second usability testing included three 9th-grade math teachers and 78 high school students who utilized MathPal in class for a month. Testing followed a task-based protocol (Lu et al., 2022), with participants exploring MathPal’s key functions. After each session, semi-structured interviews gathered feedback on usability, functionality, and potential improvements.
Participants in the first testing interacted with MathPal for two hours, and they reported that it effectively provides precise and reliable explanations of mathematical concepts, ensuring clarity in problem-solving. They also found that MathPal generated accurate performance reports, allowing both students and educators to track progress and identify areas for improvement. Additionally, the problem-solving strategies suggested by MathPal were considered reasonable and useful, helping students reinforce their understanding and approach problems more effectively.
Participants also suggested three key improvements for MathPal: a stronger user management system to ensure secure and retrievable logins for teachers, enhanced data security measures to protect large volumes of student-generated conversations, and better off-topic handling to keep discussions focused on math by effectively redirecting unrelated queries.
In response to their feedback, several improvements were made to strengthen MathPal’s functionality and security. A Firebase-based user management system was implemented to ensure secure login and controlled access, allowing teachers to manage student accounts effectively. To address data security concerns, cloud storage solutions were integrated, ensuring that student-generated conversational data is securely stored and protected. Additionally, prompt engineering refinements were applied to enhance off-topic handling, ensuring that MathPal redirects students back to relevant math discussions whenever conversations stray from the subject.
In this round of usability testing, ninth-grade Algebra I students interacted with MathPal for a month. They used MathPal to practice math problems after receiving instructions, while teachers circulated around the classroom to address any additional questions students had.
Teachers reported significant improvements in system reliability and security, reinforcing their confidence in using MathPal as a classroom tool. Encouraged by these refinements, participants expressed readiness to implement MathPal in their math classrooms and proposed further enhancements to better align the AI tool with real-world educational needs. These ongoing iterations underscore the importance of continuous improvement in AI-driven educational solutions.
Teachers recommended Teacher-Controlled Access for managing student use during assessments, Customization Capabilities for uploading personalized worksheets, and Teacher-Informed Analytics to categorize student interactions for easier progress tracking. These suggestions emphasize the need for AI tools that support educators while maintaining oversight and enhancing, rather than replacing, instruction.
Students found MathPal particularly helpful in guiding them through complex math problems by breaking down solutions into step-by-step suggestions. They appreciated its support when teachers were unavailable and found that interacting with MathPal introduced them to new problem-solving strategies, such as organizing their steps methodically and creating structured lists. Many students noted that MathPal simplified explanations of concepts like linear functions and equations, making learning more engaging and accessible. Through continuous interactions, students enhanced their AI prompting skills, an essential component of AI literacy, by learning to formulate more precise and detailed questions, such as incorporating numerical data or graph descriptions, to receive better responses. Additionally, they suggested a "stuck detection" feature, where MathPal could recognize struggles and proactively offer hints, presenting an exciting opportunity for future development.
MathPal represents a promising step toward integrating AI into high school mathematics education. Grounded in a user-centered design approach, the tool was shaped through direct and repeated input from teachers and students to ensure that its features aligned with authentic classroom needs, instructional goals, and learner behaviors. Rather than being designed solely around technical capabilities, MathPal evolved through iterative usability testing and feedback, allowing the interface, functionality, and support features to better reflect real user needs. By fostering metacognitive awareness, personalizing learning experiences, and providing clear mathematical guidance, MathPal supports students in developing stronger and more confident problem-solving skills.
This study highlights several implications for AI-driven learning support. Metacognitive knowledge should guide AI design to enhance student engagement and self-monitoring. Customization features should allow teachers to tailor activities to curricular goals. Teacher-in-the-loop mechanisms are essential for enabling educators to oversee student interactions, maintain instructional alignment, and intervene when needed. Data security also requires careful attention, and collaboration with school districts is necessary to ensure responsible and ethical use of student data.
Looking ahead, the proposed improvements will continue strengthening MathPal’s usefulness in real classrooms. These efforts will support broader implementation and contribute to ongoing exploration of AI’s role in K–12 education through a lens that prioritizes learner experience, educator needs, and ethical design.
EdTech Archives