Educational Design Research Practice illustration Iteration type I of holistic DBR – transferability of design principles

The iterative nature of design-based research (DBR) is a well-known and valued aspect of research in education. Nevertheless, iterations may take different formats, especially if a complex design object is at hand. Holistic DBR outlined by Reinmann (2020) provides an opportunity to display development of complex research objects. Still, there are not many examples to illustrate holistic DBR iterations. The current paper illustrates iteration type I of the holistic DBR and also displays how transferability of design principles can be tested. The iteration is a part of a wider DBR (Cycle 10) outlined in the article: Design principles for developing online ethics resources – the outcome of holistic DBR process . The current study utilised document analysis and a short questionnaire to evaluate whether the design principles would support creating ethics resources in various disciplines. The results of Cycle 10 contributed to elaboration of design principles, and elicited recommendations on facilitation of ethics training. holistic DBR, design principles, ethics resource, iteration type The current paper is a practice illustration accompanying the article: Design principles for developing online ethics resources – the outcome of holistic DBR process (Tammeleht, 2022). The paper provides an illustration for Cycle 10 of the holistic DBR outlined in the abovemen-tioned article (named ‘the main article’ from now on) as an illustration of iteration type I. The motivation, theoretical grounding and information about the holistic DBR are given in the main article and are not repeated here.


Introduction
The current paper is a practice illustration accompanying the article: Design principles for developing online ethics resources -the outcome of holistic DBR process (Tammeleht, 2022). The paper provides an illustration for Cycle 10 of the holistic DBR outlined in the abovementioned article (named 'the main article' from now on) as an illustration of iteration type I. The motivation, theoretical grounding and information about the holistic DBR are given in the main article and are not repeated here.
Based on the cycles preceding the current one, design principles were compiled. As the quality of DBR can be evaluated based on the transferability of the results (i.e. design principles), another cycle was required to verify their applicability (see Figure 4 and Table 2 in the main article presenting the entire holistic DBR process) (Tammeleht, 2022).
A research question was proposed: Do the design principles support the creation of ethics resources in various domains?

Methods used during Cycle 10
Various methods can be used as part of DBR, often mixing qualitative and quantitative data and analysis methods. For the current cycle, document analysis and a questionnaire were selected as it was possible to collect this data by implicating as little inconvenience on participants as possible. Document analysis is a process of evaluating documents of various formats (both physical and digital) (Bowen, 2009). According to Bowen (2009), document analysis is often used to triangulate findings but can also be used as a method of its own, especially to verify or support findings. The main functions of the documents in the current study were to provide evidence of the resource development process and to verify previous findings (i.e. design principles) (Bowen, 2009). Document analysis may include deductive or inductive qualitative approaches, either content or thematic analysis (Bowen, 2009). The current study will utilise deductive thematic analysis, where the codes are the design principles, and evidence of their manifestation was sought in the documents.

2.0
Cycle 10 process and participants Based on the conducted cycles of the holistic DBR, initial design principles were outlined. They pertained to conceptualising, authoring and facilitating ethics training (see Figure 6 in the main article) (Tammeleht, 2022).
The piloting of the design principles was conducted in 2019-2020 by the author and one practicing teacher in Estonia by designing a Cyber Ethics online resource for upper secondary students. The teacher continues to use the resource as part of her Cyber Security course. No data was collected from the learners (other than general feedback), but the teacher feedback fed into improving the design principles.
For Cycle 10 volunteers were sought at a university in Estonia in January 2021. A proposal was made for an interdisciplinary course LIFE (https://elu.tlu.ee/en) where students can create their own content. 24 bachelor and master level students volunteered and worked in 4 groups for 5 months. All groups were given a table (see Table 1) with design principles and clarification of the principles. This table scaffolded the entire design process and kept the goal orientation. Four groups created different online ethics resources for various domains. Based on document analysis of team-work, resource evaluation and a short questionnaire conducted among resource creators, design principles were elaborated and their transferability was evaluated.

Research ethics
The research followed the European Code of Conduct for Research Integrity (ALLEA, 2017), the Estonian National Code of Conduct (Hea Teadustava, 2017). No ethics review was required since the study did not involve an intervention in the physical integrity of research participants or deviate from the principle of informed consent. Participation was voluntary, and the participants were asked for their informed consent with all their rights clarified to them. The document analysis and resource evaluation was group-based, questionnaires were individual and anonymous. Even though the LIFE course was part of the students' study programme, the analysis only took place after the course had ended, so the participation or non-participation had no effect on their passing the course.

Cycle 10 data collection
To evaluate the effectiveness and applicability of the design principles, data was collected from groups. Most data was qualitative in nature and came in the form of documentation of the group work process compiled by groups in their Google Drive folder (folders of 4 groups in total). The Google Drive folders contained (including visuals, slides and text documents): Group 1 -31 items
Three major documents were selected as these were consistent in all groups: 2. Mid-term and final reports of the group work process; 3. Final version of the online ethics resource.
In addition, a short questionnaire was compiled to be answered anonymously by individuals asking about the usefulness of the design principles and collecting ideas for improvements.

Cycle 10 data analysis
Qualitative data (in the form of documents and the final ethics resource) was analysed utilising deductive thematic analysis. The codes were derived from the design principles (DP) 1-10. Design principle 11 (reflection) was not made compulsory for the teams due to time constraints set for the course. Instead, the groups had two extra principles -improving the resource and dissemination, which were set by the course requirements. The extra principles were not included in the current analysis as they did not pertain to the design principles of the main article.
The three sets of documents were thoroughly examined and manifestations of design principles were sought there. Two tables were compiled (see Tables 2 and 3) to see how the manifestation of design principles was displayed.
The questionnaire collected both quantitative and qualitative data.
The first question inquired about the perceived usefulness of the design principles and the answers could be selected on the 1-5 Likert scale (1 -the principles were not useful, 5 -the principles were very useful). The second question was an open-ended and optional question asking about improvement ideas. About 60 % of respondents provided a response.

Cycle 10 results and discussion
First, design principles were identified in various documents. To illustrate them, Table 2 was compiled. Each source lists the design principles that could be identified in them. It seems that the majority of design principles were manifested in all documents selected for analysis.

Teacher ethics
Group 4 -

Local political ethics
DP To get an overview of the manifestation and effectiveness of the design principles, an overview table was compiled. Table 3 includes manifestations of design principles across all documents included in the analysis. The first column displays the topic sought (based on the design principles created). Columns 2-5 have manifestations of those design principles in all the groups.
All in all, as displayed in Table 3, most design principles were targeted at some point in the design process. Background search was quite thorough by all groups, various sources were consulted and this helped the groups' orientation in the selected field. In addition to background, all groups had at least one expert. Even though some groups did not provide much proof of the role of the expert, the teams that collaborated closely with the expert provided excellent support material (e.g.

Preventing bullying, Cyber ethics and Teacher ethics).
It was also visible how one design principle contributed to the outcomes of others. This was especially prominent in the creation of the framework and cases. Groups would often start collecting cases and based on their topics, then identified the more general topics and created the framework. The epistemic object proved to be a challenge for the groups.
The epistemic object is necessary to provide structural scaffolding to the learners, display their knowledge building as well as keep the focus (Scardamalia & Bereiter, 2006, Hakkarainen, 2009). Groups struggled with choosing the best option; the originally chosen epistemic object was occasionally replaced due to technical problems when they tried to include the chosen object in the online environment. Free platforms occasionally limit the options to be included.
The selected target group was also considered while compiling the ethical analysis steps. For example, the Preventing bullying resource was targeted at students aged 10-15 and the ethical analysis steps were adapted to their level. Cyber ethics training was targeted at adolescents and adults and various ethical approaches were included (which would be challenging for younger learners). Support material was provided by all groups, but occasionally it preceded the tasks (e.g. for the Local political ethics resource). The groups that provided support material in the form of 'expert opinion' were able to compile the most logical learning trajectory. In knowledge building the additional information (often provided by the textbook or teacher) is evaluated by the group, often also compared to their own responses and new knowledge is constructed (Scardamalia & Bereiter 2006). To do that, it would be more logical for learners to construct their own answers and only then make the comparison with the expert opinion. At the same time, it is important to provide possible solutions to ethical analysis steps to avoid misconceptions. Almost all teams provided extra resources for further reading (included as links to the online resource).
The usability of the online platform seemed to depend on the skills of the team members -with more skilled members, a more professional layout was created, while with more limited skills, user-friendliness suffered. It is advisable to find a more skilful web designer to enhance user experience. Piloting was conducted by 2 groups and it proved to be beneficial. Feedback from users was very positive and improvements were made to the online resource based on the user feedback.
Questionnaire results showed that about 70 % of the participants thought the design principles were useful or very useful. About 20 % of respondents were neutral and 7 % considered them not so useful. 60 % of respondents also provided their feedback -most responses said that the design principles were logical and supportive. One respondent claimed that it was difficult to cover all the principles in 5 months and one comment was about the epistemic object -noting, that it may change throughout the process.

Conclusion
Based on the results it can be concluded that the design principles indeed provide sufficient guidance to create new ethics resources and thus also answer the research question of the study. Based on the lessons learned the design principles were elaborated.
Principle 3, building a framework in the conceptualisation phase was elaborated. The study revealed that building the framework and collection of cases may take place hand in hand. Occasionally, collecting various cases in one document may show the common topics and contribute to finalising the framework.
Principle 6 (guiding questions) was also in the focus during the study. The target groups should be considered as the ethical analysis steps and/or additional questions may need to be modified according to the age of the potential users. Also, questions should have some consistency to provide structural scaffolding. Support material in Principle 7 and its content was also present in Cycle 10. The study showed that the support material -perhaps in the format of 'expert opinion' or 'possible solutions' -should follow the case discussion and ethical analysis, as the users see the need for 'advice' only after having provided solutions of their own. Comparing their own answers and the expert's opinion is an important learning opportunity.
Principle 8 (website design) was also elaborated after Cycle 10, where data indicated that well-considered layout and collecting user experience feedback may help improve the online environment. All groups who piloted the resource collected valuable feedback and were able to improve their resource.
The choice of the epistemic object may have an impact on the group work outcome. Indeed, the technical solutions may be limited, but it is advisable to choose a simple option for the training, e.g. a Google Drive document, a form to fill or Jamboard. All of these can be shared online; documents can also be printed out for face-to-face collaboration. It is acceptable to change the epistemic object throughout the design process, especially if the user experience feedback suggests it.
All in all, the design principles outlined in the main article do support creating ethics resources in various domains according to the outcomes of the current study. The design principles guide developers to create evidence-based ethics resources and help learners gradually 3.0 advance their ethics competencies in their field. Future studies should focus on monitoring the advancement of ethics competencies with the help of the new ethics resources. Anu Tammeleht is a research ethics advisor at Tallinn University, and a PhD student at the University of Helsinki. She has worked in education for more than 20 years, from elementary school to the university level. Her previous research has involved curriculum development and educational management. Her recent research has focused on research ethics and integrity, and ethics education. She is now designing strategies for developing research ethics and integrity leadership competencies to support early-career researchers.

4.0
Author Profile