Assessing the Quality of Arguments in Students’ Mathematical Problem Solving

Background/purpose – Argumentation plays an essential role in higher-order activities and in the communication of mathematical knowledge. Although the purposes of argumentation have piqued the interest of many researchers, few have simultaneously investigated the quality of argument in students’ mathematical problem solving. Materials/methods – In this case study, 41 middle school students in Indonesia solved an argumentative task. The students’ responses were then analyzed for the quality of their arguments. In line with the study’s goal of assessing the quality of the students’ arguments in mathematical problem solving, their responses were coded according to the Polya method and the CER model. Results – The results revealed that more than half of the students misunderstood the given mathematical problems, which were not appropriately evidenced or reasoned in their response. Conclusion – These results indicate that further handling is needed to improve the quality of students' arguments and to develop the importance of activities that support students in explaining, justifying, and correcting their reasoning during mathematical argumentation.


INTRODUCTION
In recent decades, argumentation has been emphasized by many researchers as one of the most important activities for mathematics students, and which has become an interesting field of academic research (Kartika et al., 2021;Knipping & Reid, 2015).Argumentation scientifically involves certain higher-order activities such as communication between peers, critical thinking, knowledge evaluation, reasoning skills, social behavior, information gathering skills, and decision-making from multiple perspectives (Nussbaum, 2011;Rapanta, 2019).The ability to conduct a convincing argument is deemed critical for the establishment, development, and communication of mathematical knowledge (Stylianides, 2019).
Furthermore, argumentation skills have become broadly a part of current mathematics curriculum standards in many countries, and the development of these abilities is considered a primary goal in secondary school education (Shi, 2020).For example, in the United States, the Common Core State Standards for Mathematical Practice (CCSSMP) include standards for argumentation (Kartika et al., 2021), requiring students to "construct viable arguments and critique the reasoning of others" (National Governors Association Center for Best Practices, 2010).Although the concept of argumentation is not explicitly stated in the Indonesian Mathematics Curriculum Standards, some key elements in the standards have defined the issue.For example, knowledge competence and mathematical skills at the lower secondary education level should emphasize students' conceptual and procedural understanding of knowledge and present it in the concrete realm (e.g., explaining and justification) and abstract domains (e.g., written responses) related to the material studied on the topic of problem solving (Ministry of Education and Culture of the Republic of Indonesia, 2018).As a result, researchers and educational policymakers have paid close attention to the nature of students' mathematical argumentation and strategies for assisting them in developing argumentation skills.

LITERATURE REVIEW
Mathematical argumentation is known as a sequence of statements and reasons, which aims to show that a claim (conclusion) is either true or false (Cardetti & LeMay, 2018).Argumentation involves constructing claims, providing evidence to support those claims, and evaluating evidence in order to judge the validity of the claims, as well as critique by examining the reasoning that connects the evidence to the claim (Osborne et al., 2016;Schwarz et al., 2010).This process involves various activities such as guesswork, example testing, experimental thought, representation of mathematical ideas, taking into account others' points of view, and then analyzing and revising the findings so as to reach a conclusion (Staples & Newton, 2016).In addition, argumentation is also found to arise through various means, such as questioning, persuasion, negotiation, or disagreement (agree or disagree).Argumentation is the means by which we rationally resolve questions, issues, and disputes and to solve problems (Jonassen, 2011).
Problem solving is a pattern of individual (student) decisions that aims to solve problems by simply eliminating or overcoming symptoms, through diagnosis and changes in the underlying causes (Mohaghegh & Grobler, 2020).Problem solving refers to cognitive processes directed towards achieving goals when the problem solver is not initially aware of the solution method (Mayer, 2013).Problem solving is an individual's capacity to utilize cognitive processes in dealing with and solving real situations (Organisation for Economic Cooperation and Development, 2003).

Kartika and Budiarto | 30
Üniversitepark Bülten | Bulletin | 2022 | 11(2): 28-40.Polya (1973) stated that four cognitive processes need to be conducted in order to investigate problems, namely: 1) understanding the problem; 2) devising a plan; 3) performing the plan; and, 4) looking back.This indicates that students need to be equipped to undertake certain steps in order to solve problems.When learning about principles, students can employ one of two approaches: a) studying an example problem in which each step is worked out and provided with the solution (referred to as a worked example), or b) attempting to solve a problem from beginning to end without assistance (referred to as problem solving) (Foster et al., 2018).
In conventional learning and textbooks, worked examples are often presented to students, but are the most common case type presented as examples of problem solving (Jonassen, 2011).A worked example consists of a problem formulation, its solution, and additional explanations, and thus provides students with a high level of assistance (Grobe, 2018;Isotani et al., 2011).In contrast to worked examples, an erroneous example is a stepby-step guide to solving a problem in which one or more steps are erroneous (Chen et al., 2019).Studying erroneous examples, according to Adams et al. (2014), promotes deeper cognitive processing aimed at organizing learning material and relating it to students' prior knowledge.Furthermore, explaining why an erroneous answer is indeed erroneous requires students to first consider the correct solution and its range of application (Heemsoth & Heinze, 2016).On the basis these explanations, we identified that similarity exists between worked and erroneous examples in their presentation of the problem as an example with a troubleshooting guide.In the current study, although it has a purpose and concept similar to an erroneous example, we presented the case as a problem without support (referred to as erroneous answers).
Erroneous answers are the result of completing mathematical tasks that contain several erroneous solving steps, and thereby require students to find and fix the errors.In particular, erroneous answers are a problem representation that can be used as an attempt to trigger students to argument their way through the controversy (agree or disagree).Argumentation for erroneous answers is a cognitive process that requires students to follow the problemsolving stages of providing supporting reason and evidence for claims.This approach aligns with Toulmin's (2003) modeling arguments into three main components, namely data, warrant, and claim.Claims are conclusions or statements that are discussed, while data provide evidence to back up these claims.Warrants are defined as specific or categorical statements that are relevant to the conclusions and explanations of the data (Freeman, 2011).Therefore, warrants connect the data and the conclusions (claims).Warrants also serve to show that the conclusion is valid and to explain how and why the data supports the conclusion.Toulmin (2003) stated that warrants can be in the form of formulae, definitions, axioms, or theorems, or may consist of inductive aspects such as pictures, diagrams, or graphs.
The influence of Polya and Toulmin's model was considered crucial for the pioneers of mathematical argumentation (Carrascal, 2015).Moreover, McNeill and Krajcik (2008) developed a Claim-Evidence-Reasoning (CER) model derived from Toulmin's more complex argument model, but was adapted to provide a simplified model for the science education context.The CER model has been implemented in mathematical argumentation designed for elementary school students in Australia by Fielding-Wells (2016) and for high school students in the United States by Graham and Lesseig (2018).A claim is considered a student's statement on solving the problem, and evidence is information collected and used to support the truth of a claim.While the reason is defined as that presented by students to reduce uncertainty and to clarify/state the overall problem solving of all problems that may occur (Zambak & Magiera, 2020).From this perspective, we designed a framework by questioning the components of the argument using Polya's problem-solving steps and the CER model of argumentation.The first step involves the understanding of problems by making a claim.Then, steps two through four involve planning, plan performance, and confirmation of the answers, as a means of providing reason and evidence to support the claim.This concept is presented as shown in Figure 1.2012) suggested that errors should be fictitious examples of other students' errors so that the student reviewing an error is not unnecessarily embarrassed or potentially demotivated by having their mistakes exposed.Therefore, we positioned the students as error correctors, while their peers with pseudonyms pretended to be the one who made the error.Adam et al. ( 2014) stated that students understand and learn mathematics more deeply and better when assessing errors made by their peers.Furthermore, pointing out the errors of others may help prevent students from feeling embarrassed and losing motivation to face their errors (Tsovaltzi et al., 2010).
From an international perspective, several researchers have attempted to develop students' mathematical argumentation skills.For example, Dogruer and Akyuz (2020) designed an inquiry-based learning environment using Geogebra in order that junior-high school students (aged 14 years old) in Turkey can engage in argumentation.Youkap et al. (2019) designed a geometry assignment on the parallelogram problem to examine the argumentation skills of students aged 14-16 years in Yaoundé, Cameroon.Yopp (2018) designed instructional learning to examine the argumentation skills of eighth-grade students in the United States using rational numbers, while Liua et al. (2016) designed an interview instrument to quiz eighth-grade students in the midwestern United States to explain their reasoning in evaluating arguments that justify conjectures on several problems involving numbers, geometry, probability, and algebra.However, few studies have looked at the quality of arguments in students' mathematical problem solving.As previously mentioned, the current study aims to assess the quality of arguments in students' mathematical problem solving.This case study aims to provide some indication of the quality of arguments of seventh-grade students in Indonesia during mathematical problem solving.

METHODOLOGY
A quantitative case study was conducted in the current study.Participants were selected according to convenience sampling, whereby 41 students from a public middle school in Indonesia were recruited.The average age of the participant students was approached 13 years old.The sample consisted of 22 (53%) male and 19 (47%) female students.
A diagnostic test adapted from the work of Rushton (2018) was used in the current study.The validity of the test instrument was first assessed by two experts prior to being used, and recommended changes applied in response to their feedback.This test consisted of two open-ended questions on the topic of algebraic equations.The problems provided are shown in Table 1.The students were tasked with analyzing two mathematical problems and asked to provide responses in the form of their agreeing, disagreeing, or being unsure of the solutions provided by their peers.The response given was perceived to be the participant's claim.Subsequently, the students were asked to provide evidence and reasoning for their responses or claims.
This research was conducted during the second semester of the 2020/2021 academic year, and the test administered online by a teacher outside of their normal teaching hours.The students were explained that their responses would only be used for research purposes, that they were to take part in the study voluntarily, and that their responses would remain confidential with their anonymity assured.Furthermore, the participants were given a specific timeframe, which allowed them approximately 15 minutes to complete the test.The researchers collected the participants' responses but withheld them from the students' teacher since they may have influenced decisions about the students' achievement assessments.
The students' responses to the administered Google Forms test document were then exported to Microsoft Excel in order to be analyzed for the quality of their arguments.First, the responses were coded according to the Polya method and the CER model (see Figure 2).Furthermore, adjustments and additions were made to the words "because" and "since," in the flow of the argument (Nordin & Boistrup, 2018).

Figure 2. CER model
Students with similar claims were grouped into three categories in order to determine the frequency and proportion of each response, namely i) disagree; ii) agree; and iii) unsure.Subsequently, a response was evaluated as inappropriate if no reasons or evidence were provided for the claim(s), or none of the reasons or evidence given were deemed relevant to/support the claim(s).Therefore, "appropriate but incomplete" was coded if reasons were provided for the claim(s) but without appropriate evidence."Qualified" was coded if the reasons and evidence were provided for the claim(s) or the reasons and evidence were deemed relevant to/support the claim(s).Finally, the structure of the students' arguments was analyzed according to descriptive statistics using absolute and relative frequencies.

RESULTS
Based on the results of the data obtained, the frequencies for each response to problem 1 and problem 2 are illustrated in Table 2.The results presented for problem 1 in Table 2 reveal that more students provided claims of agreement, compared to those who disagreed.However, it was found that they were presented with the wrong problems to solve.Additionally, 19.5% (f = 8) of the students failed to provide a clear (true or false) response to the problem presented.This finding indicated that most of the participant students had misunderstood the problem, having failed to fully understand the problems with which they were provided.In addition, the students did not adequately concentrate and misread that there were errors in some of the problem-solving steps.Although 7.3% (f = 3) answered "disagree," their response was not backed with valid reasoning or evidence.For example, the students who answered disagree only provided an explanation because the answer was incorrect.However, they failed to provide a follow-up explanation of the correct completion steps (see Table 3). .Multiply both sides by -1, so that

Appropriate but incomplete
Reasons provided for claim(s), but no evidence.

7.Disagree because the answer is wrong [Student 12];
Disagree because the answer is wrong [Student 3].

Inappropriate
No reasons/evidence provided for claim(s) or none of the reasons or evidence were deemed relevant to/support the claim(s).Furthermore, similar results also occurred in problem 2. Based on Table 2, 68.3%, 14.6%, and 17.1% of the students agreed, disagreed, and were unsure, respectively.Most of them were observed not to have concentrated on analyzing the problem-solving errors provided.Furthermore, the students were not used to providing responses, evidence, and reasons when presented with problems related to argumentation (see Table 4).

DISCUSSION
Errors analysis is an activity that is usually carried out by teachers in the assessment of students' learning achievement or in the diagnosis of students' mathematical abilities.This, however, is not an activity that is usually performed by students or their peers.When arguing through erroneous answers, students must be able to relate previously acquired knowledge with the problems they are trying to resolve.This statement is in line with Sitzman et al. (2015), who stated that prior knowledge has a very important role in the correction of errors.It also affects argumentation skills (Klopp & Stark, 2020).Accordingly, students will likely be unable to analyze errors if they lack the appropriate prior knowledge.When analyzing errors, students tend to ignore possible calculation errors, and the application of mathematical concepts, procedures, and operations during the problem-solving process.In addition, students' low quality argumentation during problem solving may depend upon factors such as their lack of prior knowledge or a misunderstanding of the question being asked.
According to the results of the current study, even though some of the students made correct claims, they were not backed by relevant evidence or reasoning.This was likely due to the students failing to realize that they had been provided with some invalid steps when attempting to solve the given problem.Students are generally unaccustomed to engaging in the activity of argument, or expected to provide evidence and reasoning that involve definitions, theorems, or mathematical properties.Also, the students did not analyze the errors or understand the problems that were provided to them.Furthermore, we identified that some of the students misunderstood certain mathematical concepts based on the evidence they presented to support their claims.In order for a particular set of argumentation activities to be deemed effective, conceptual mathematical aspects should first be identified for specific students, and certain patterns of error analysis should be employed in discerning these critical aspects.

CONCLUSION AND SUGGESTIONS
This study showed that most students in this region were unable to produce appropriate or valid argumentation in mathematical problem solving in the context of algebraic argumentative tasks.A minority of students understood that there were erroneous answers, but were unable to provide adequate justification or reasoning, nor were they able to correct the answers.Therefore, this study can be said to have revealed that further handling is needed in order to improve the quality of students' arguments in mathematical problem solving.This finding is a matter of significant concern since argumentation is considered essential to the communication of mathematical knowledge and higher-order activities.Therefore, we recognize the importance of activities that help students to explain, and correct their reasoning during the process of mathematical argumentation.

DECLARATIONS
Author Contributions: The first author contributed to this manuscript with the literature review, data collection, and data analysis.The second author contributed to this manuscript with reviewing and supervision.Both authors have read and approved the published final version of the article.
Conflicts of Interest: The authors declare no conflict of interest.
Ethical Approval: All procedures performed in studies involving human participants were conducted following the ethical standards of the institutional and/or national research committee, and also with the 1964 Helsinki Declaration and its subsequent amendments or comparable ethical standards.Funding: None.

Figure 1 .
Figure 1.Argumentation framework in mathematical problem solvingIn order errors are valuable for learning and are therefore helpful to students.McLaren et al. (2012) suggested that errors should be fictitious examples of other students' errors so that the student reviewing an error is not unnecessarily embarrassed or potentially demotivated by having their mistakes exposed.Therefore, we positioned the students as error correctors, while their peers with pseudonyms pretended to be the one who made the error.Adam et al. (2014) stated that students understand and learn mathematics more deeply and better when assessing errors made by their peers.Furthermore, pointing out the errors of others may help prevent students from feeling embarrassed and losing motivation to face their errors(Tsovaltzi et al., 2010).From an international perspective, several researchers have attempted to develop students' mathematical argumentation skills.For example, Dogruer and Akyuz (2020) designed an inquiry-based learning environment using Geogebra in order that junior-high school students (aged 14 years old) in Turkey can engage in argumentation.Youkap et al. (2019) designed a geometry assignment on the parallelogram problem to examine the argumentation skills of students aged 14-16 years in Yaoundé, Cameroon.Yopp (2018) designed instructional learning to examine the argumentation skills of eighth-grade students in the United States using rational numbers, whileLiua et al. (2016) designed an interview instrument to quiz eighth-grade students in the midwestern United States to explain their reasoning in evaluating arguments that justify conjectures on several problems involving numbers, geometry, probability, and algebra.However, few studies have looked at the quality of arguments in students' mathematical problem solving.As previously mentioned, the current study aims to assess the quality of arguments in students' mathematical problem solving.This case study aims to provide some indication of the quality of arguments of seventh-grade students in Indonesia during mathematical problem solving.
you agree, disagree, or are unsure about Ani's answer?Students include evidence to support their claim.b) Explain why you agree, disagree, or are unsure about solving the given problems.

38 92. 7
Unsure as do not completely understand [Student 25]; Unsure because the answer is not sure [Student 39]; Agree it is known that 64 / 8 = 8 and what you are looking for is the x-axis, then 64 / 8 = 8 moves left to -8x / 8 then -8x moves to the right to +8 = x [Student 26]; Agree the result of 64, as 8 x 8 = 64 is the result of x + x = + not -so the result is x = 8 [Student 35].Total 41 100

Table 1 .
Description of the Topic and Problems

Table 2 .
Student Ability to Envisage a Claim with Different Problems

Table 3 .
Student Ability to Envisage Evidence and Reason in Problem 1

Table 4 .
Student Ability to Envisage Evidence and Reason in Problem 2