Qualitative Analysis of Written Reflections during a Teaching Certificate Program
a University of Arkansas for Medical Sciences College of Pharmacy, Little Rock, Arkansas
b University of Arkansas at Little Rock
c University of Arkansas for Medical Sciences Office of Educational Development, Little Rock, Arkansas
Corresponding Author: Amy M. Franks, Department of Pharmacy Practice, University of Arkansas for Medical Sciences College of Pharmacy, 4301 West Markham St., Slot #522-4, Little Rock, AR 72205-7199. Tel: 501-296-1296. Fax: 501-296-1168. E-mail: ude.smau@sknarFA
Author information ►Article notes ►Copyright and License information ►
Received 2014 Dec 15; Accepted 2015 Mar 5.
Copyright © 2016 American Association of Colleges of Pharmacy
This article has been cited by other articles in PMC.
Objective. To evaluate the success of a teaching certificate program by qualitatively evaluating the content and extent of participants’ reflections.
Methods. Two investigators independently identified themes within midpoint and final reflection essays across six program years. Each essay was evaluated to determine the extent of reflection in prompted teaching-related topic areas (strengths, weaknesses, assessment, feedback).
Results. Twenty-eight themes were identified within 132 essays. Common themes encompassed content delivery, student assessment, personal successes, and challenges encountered. Deep reflection was exhibited, with 48% of essays achieving the highest level of critical reflection. Extent of reflection trended higher from midpoint to final essays, with significant increases in the strengths and feedback areas.
Conclusion. The teaching certificate program fostered critical reflection and self-reported positive behavior change in teaching, thus providing a high-quality professional development opportunity. Such programs should strongly consider emphasizing critical reflection through required reflective exercises at multiple points within program curricula.
Keywords: teaching certificate program, reflection, qualitative, teaching development
Reflection is an intentional, dynamic process that allows improvement in one’s actions, abilities, and knowledge by learning from past experiences.1-4 While this process can be useful in almost all aspects of life, reflection in the workplace can be particularly beneficial. Successful professionals must be able to reflect on their experiences in order to find solutions to complex problems encountered on a daily basis.1-2 Such reflection is not only necessary for pharmacists and other health care professionals to improve their practice, but also to further hone their expertise as educators.5 Reflection should be used by pharmacists committed to professional growth as lifelong learners.6
The practice of reflection during residency programs offers a valuable opportunity to observe and guide residents in this process at the beginning of their careers. Teaching certificate programs within pharmacy residency programs were founded on the idea that having specialized pharmacy knowledge does not necessarily equate with being an excellent teacher.7 Teaching certificate programs give participants general pedagogical knowledge to combine with their existing content knowledge. Participants complete didactic and experiential teaching activities to develop such teaching skills. A critical component of the teaching certificate program at the University of Arkansas for Medical Sciences (UAMS) College of Pharmacy is the extensive reflection required throughout the year-long program. Because reflection is such an integral part of professional development, evaluation of the content and extent of participants’ reflections is imperative.
Research on the topic of reflection is extensive, but investigation of reflection on teaching by pharmacy faculty members or faculty members in training is not described in the literature. Additionally, methods to assess teaching certificate programs focus on surveys but lack the details offered by more in-depth analysis. Qualitative analysis of reflective essays could provide better understanding of program benefits and participant growth. Our mixed-methods evaluation is the first to examine thematic composition of reflections as well as the extent of reflection evidenced in the written essays of potential future pharmacy faculty members as they participate in teaching certificate programs.
This study was designed as a mixed methods thematic analysis of teaching certificate program participants’ reflective essays. A qualitative approach was chosen because this method allows deep analysis of the text not obtainable from survey-based research. A modified constant comparison method of analysis was employed, and categories and themes were constructed from open and axially coded data.8 The coding scheme arose from the data as researchers explored them. Resulting themes were evaluated in comparison to the typology derived from teacher reflection theoretical framework. Qualitative methods were quantified to provide comparison of themes and level of reflection. The data evaluation provides a descriptive evaluation of the effectiveness of this teaching certificate program activity and its impact on teaching development.
The school’s teaching certificate program was originally developed in 2005 to enhance the teaching skills of pharmacy residents but quickly expanded to include preceptors because of an increased demand for preceptor development in teaching. The program, described in a previous manuscript,9 facilitated development as an educator through the following experiences: formulating personal goals for development in teaching, tailoring teaching approaches to learning setting and audience, practicing effective assessment and feedback skills, receiving ongoing feedback from program faculty members, reflecting upon individual teaching experiences, developing a personal teaching philosophy, and documenting experiences through the development of a comprehensive teaching portfolio.
Over the course of the program (July to May), participants attended formal teaching seminars, self-selected teaching activities, and met with a faculty teaching mentor, who monitored their progression in the program. Participants also were required to write two reflective essays describing their teaching development. The midpoint reflection was submitted in December, and the final reflection was submitted in May of the program year. In these global reflections, participants were asked to discuss their teaching development and specifically include commentary on each of four topic areas: teaching-related strengths, teaching-related weaknesses, ability to effectively assess learners, and ability to provide effective learner feedback. Pharmacy residents and preceptors from across the state participated in all aspects of the program. All retrievable essays from participants who submitted written essays in December and May were analyzed. Participants were excluded if they did not complete the teaching certificate program by submitting December and May essays or if the data were not available.
De-identified electronic copies of reflection essays were used to extract themes and the extent of reflection using the four prompted topic areas of strengths, weaknesses, assessment, and feedback. The 4-category coding scheme described by Kember et al10 was adapted to determine the extent of reflection for each of the four prompted topic areas as well as the highest level obtained overall (Table 1). A fifth level of “0” was added to be assigned when the participant did not discuss a topic area. Because the instructions for the reflective assignment served only as prompts for reflection, participants were not required to write about each area, and therefore could receive a score of “0.”
Five-Category Scheme for Assessing Extent of Reflection in Written Essays10
Two investigators in the team independently identified the themes discussed in the essays along with the extent of reflection of each essay in the four topic areas using NVivo, v10 (QSR International Pty Ltd., Doncaster, Victoria, Australia). The primary coder used a sample of 10 essays to create initial categories of themes and shared this initial set of themes with the second coder. Coders independently analyzed essays for themes discussed and the extent of reflection obtained in each topic area. The coders met weekly to discuss the extent of reflection assigned to each essay and resolved any differences.
By using a constant comparative approach, coders agreed by consensus on every essay to achieve 100% inter-rater reliability. Additional reflective themes also were added based on coder agreement when a new topic was discussed that did not fit into one of the predetermined themes. A third investigator periodically reviewed the coding process and identified themes for verification to enhance agreement between coders. An electronic coding platform made it possible to efficiently code, analyze, and organize this large amount of data. Each essay, depending on length, took about 10-20 minutes to analyze.
Further patterns emerged from the data during qualitative analysis and were explored with additional detail using triangulation methods. Coder agreement on coding of themes was assessed using Cohen’s kappa. Data were analyzed to identify differences in the extent of reflection in the midpoint and final essays using Wilcoxon signed-rank test. These data were further analyzed to identify whether extent of reflection varied according to participant gender and experience (preceptor vs resident) using the Wilcoxon rank sum test with Stata/SE, v12.0 (StatCorp LP, College Station, TX). The thematic analysis was granted exempt status by the UAMS Institutional Review Board.
All available pairs of essays were evaluated from participants completing the teaching certificate program between 2006 and 2012. One hundred thirty-two essays were analyzed from 66 participants. Of the 66 participants, 53 (80%) were female, and 10 (15%) were preceptors.
The coders identified more than 11 000 references to 28 themes discussed within the 132 essays. Themes covered a broad range of professional development topic areas, including delivering educational content, interacting with students, evaluating success, and encountering challenges. A full list of identified themes is available from the authors. Agreement between the coders on themes discussed in each essay was high (kappa=0.74). Data saturation of the identified themes occurred early in the coding process, indicating that the list of themes was representative of the context of the essays.
Themes were tracked by the number of essays in which the theme was discussed (“mention”) as well as the total number of references to the theme in all essays (“weight”), as themes could be discussed multiple times within a single essay. Table 2 provides the 10 most common discussed themes and weight placed on theme according to total number of references to that theme. The same 10 themes ranked highly according to both mentions and weight, indicating the emphasis placed on these ideas by the participants was consistent between and within essays. (A detailed description of themes and results of analyses are available from the corresponding author.)
Ten Most Common Themes by Mention and Weight
High levels of reflection were exhibited in the participant essays, with 48% of essays achieving the level of critical reflection (level 4). Quotations from teaching certificate program participant essays specific to the area of assessment were extracted to illustrate the extent of reflection. An example of nonreflection (level 1) was: “Assessment questions are aimed at evaluating comprehension of stated objectives.” A statement that was consistent with understanding (level 2) was: “There are various characteristics that students display that allow assessment of their abilities.” Reflection was evident in the level 3 statement: “It is easier to see two students evaluate similar patients than it is to see students present two different cases and discern which is more proficient.” Critical reflection (level 4) was evidenced by the statement: “But then I realized that I didn’t know if I was being an effective teacher without an assessment of some type. So I have started incorporating pre- and post-lecture questions to see what the learner gained from the presentation.”
Figure 1 shows the levels of reflection obtained overall and in the four prompted topic areas of strengths, weaknesses, assessment, and feedback for midpoint and final essays. Mean levels of reflection achieved in combined midpoint and final essays in each topic area were 2.4, 2.9, 2.7, and 2.9, respectively, indicating the greatest extents of reflection in the weakness and feedback topic areas. Each topic area showed increases in the mean level of reflection between midpoint and final essays [strengths, 2.2 (1.1) vs 2.5 (1.1); weaknesses, 2.8 (1.0) vs 2.9 (1.0); assessment, 2.6 (1.0) vs 2.8 (0.9); and feedback, 2.4 (1.0) vs 3.0 (1.0)].
Level of Reflection10 Achieved in Midpoint and Final Reflection Essays. Number of essays according to the highest level of reflection (Level 0=Absent; Level 1=Non-reflection; Level 2=Understanding; Level 3=Reflection; Level 4=Critical Reflection) attained...
The extent of reflection between midpoint and final essays significantly increased in the prompted topic areas of strengths (p=0.03) and feedback (p=0.0002). In the strengths topic area, 29 (43.9%) participants did not show change in the extent of reflection from midpoint to final reflections, but 25 (37.8%) showed deeper reflection, compared with only 12 (18.2%), who had decreased reflection scores. In the feedback topic area, 28 (42.4%) participants did not show change in the extent of reflection from midpoint to final reflections, but 30 (45.5%) showed deeper reflection, compared with only 8 (12.1%) who had decreased reflection scores. The highest level of reflection achieved seemed to be higher in the final essays [mean value 3.53 (0.53)] when compared with the midpoint essays [mean value 3.36 (0.57)], but this change did not reach significance (p=0.055) despite increases in each individual topic area. There were no differences observed when comparing change in the extent of reflection at midpoint and final according to gender or preceptor/resident status.
Further patterns emerged from the data during qualitative analysis and were explored with additional detail using triangulation methods. Program participants often discussed others’ evaluation of their teaching activities, but focused primarily on receiving evaluations from students and faculty members (65% and 49% of essays, respectively). Just 24% of essays mentioned gaining evaluation by peers (despite a teaching certificate program requirement for obtaining a peer teaching evaluation), and only 4% commented on gaining feedback from patients on their teaching. In contrast to their discussion on receiving evaluations, only five (7.6%) participants discussed the process of evaluating others’ performance in teaching. Four of these regarded providing peer evaluations, and one participant commented on giving feedback to faculty members.
A frequent conclusion of the participants was that they did not have the opportunity to complete a specific type of teaching development activity. This theme was reported in 31 (23.5%) essays. As expected, just over 70% of these occurrences were found in midpoint essays, when participants had completed only the first half of the teaching certificate program. This theme often disappeared from the final essays. However, the perceived lack of opportunities to participate in formally grading students and writing test questions (n=4), serving as an experiential education preceptor for students (n=3), giving a formal presentation (n=1), and providing student feedback (n=1) remained in the nine final reflective essays.
Reflection is vital in the life of an educator.11 Even more, reflection is the key to learning, which occurs when we create meaning from a past event and use this to shape future experiences.4,12 While other disciplines such as education have been using reflective practices for some time, the health sciences professions have more recently adopted this concept in the training of future health care professionals. Medicine, nursing, and pharmacy are among the disciplines that are adapting these types of reflective processes in curricula to aid in learning as well as improve patient care.13-21
The extent of medical residents' reflection has been explored 13-15 and showed that physicians' decision-making skills were improved in complex clinical cases if they were able to critically reflect on those experiences.15 An evaluation of nursing students concluded that interview sessions (individual, paired, and group) on reflective practice were viewed as beneficial for the participating students and encouraged them to practice reflective thinking on their own.17 Like many other professions over the past few decades, accreditation standards for pharmacy education require student reflection and subsequent assessment of these skills.18-20
Additionally, the American Society of Health-System Pharmacists Required Competency Areas, Goals, and Objectives, for Postgraduate Year One Pharmacy Residencies address the need for residents to reflect on their personal performance and professional development.21 Published works on reflection in pharmacy center on pharmacy students, but the literature suggests reflection is necessary in the day-to-day practice of a pharmacist as well.22,23 However, little is known about the reflective abilities of pharmacy educators and how these skills affect teaching-related self-development.
Assignments in this teaching certificate program served as tools to facilitate and encourage reflective thinking in order to promote growth in teaching abilities, and participants achieved high levels of reflection when completing these assignments. Almost 50% of essays evidenced critical reflection by describing how change occurred in their practice of teaching (behavior change). This high level is rarely achieved by so many.10 This could be because participants in this program were highly achievement-oriented as supported by their admission to and completion of pharmacy school as well as pursuit of residency training.
Additionally, preceptors in the program were seeking teaching development and voluntarily participated. The teaching certificate program also emphasized reflection throughout the program. In addition to the reflections reviewed in this evaluation, participants were asked to reflect immediately after each teaching activity; this practice of reflective thinking and writing may have facilitated deeper reflection in the more summative midpoint and final essays. Another factor that could have contributed to the high levels of reflection was the mentorship of program faculty members, who reviewed reflection drafts and further prompted participants to consider their experiences, personal characteristics, and teaching development goals.
When analyzing themes discussed by participants, it was evident that they considered the reflection prompts when writing their essays because strengths, weaknesses, assessment, and feedback all appeared often. A closer look at the references to these categories revealed that participants wrote most frequently about the areas of assessment and feedback. These areas stood out because they were the areas in which participants lacked confidence early in the teaching certificate program, but they were also the areas in which participants recognized the most growth at the conclusion of the program. Collectively, the extent of reflection significantly increased from the midpoint essays to the final essays in the feedback category.
There was no intervention on reflective writing between the midpoint and final essays, so the change evidenced by our evaluation seems to have occurred naturally. Participants gained more practice with these skills over the course of the program year, and they subsequently reflected more and explained their growth in this area. This finding of deeper reflection in the area of feedback validated the design of the UAMS teaching certificate program and its focus on self-reflection as a method of teaching skills development.
The area discussed by all participants in the final essays was their confidence and comfort in teaching. This linked nicely with the significant increase in the extent of reflection seen in the strengths category. Participants seemed to be more deeply aware of their strengths as the program progressed, and they discussed areas of perceived confidence in these essays. This is consistent with our previous findings that demonstrated increased self-perceived teaching abilities during the program.9
Because the teaching certificate program provides the opportunity for gaining practical experience in teaching, confidence in these abilities is expected to grow. Gaining confidence can allow a deeper awareness of personal strengths. Growth was evidenced as experiences led to increased confidence, and prompted reflection on these experiences led to gains in self-perceived teaching abilities and strengths. Increased confidence through experiences was the aim of the teaching certificate program and explains why such programs are beneficial to participants, future employers, and academic institutions.
Participants most often solicited feedback from students and faculty members. Feedback from peers and patients were reported much less frequently. Because it is a teaching certificate program requirement for participants to receive feedback from faculty members, peers, and students, it is not surprising that these perspectives were discussed in the reflections. However, patient teaching also is encouraged by the program, and the perspective and feedback of patients should be valued. Although the number of participants soliciting feedback from patients was not collected in our evaluation, few discussed these perspectives in their reflections.
Twenty participants (30%) listed patient counseling as a teaching activity, but only three participants discussed receiving teaching evaluations from patients. This could be because this information was not solicited from participants and, therefore, they do not reflect on it specifically or because the participants perceived patient evaluations to be lower quality or less importance than those from faculty members or peers. Additionally, participants might have viewed the patient as a different type of learner than students and not seen the need for reflection on their evaluations. Curricular emphasis of patients as learners increased in later program years to encourage variation of feedback from learners. To foster the view of patients as learners, programs should consider requiring patient teaching activities along with assessment of patient learning and completion of teaching evaluations by patients.
It is also noteworthy that participants wrote more frequently about receiving feedback from students and faculty members and less frequently about feedback received from peers. This corresponds to the small number of participants who reported giving feedback to faculty members (n=1) and peers (n=4). Giving and receiving feedback from peers may be an area of discomfort for participants despite the teaching certificate program requirement to give and receive this type of feedback. Additionally, only one participant reflected on the program requirement to provide feedback to at least two faculty members after observing their teaching. Overall, reflection on giving feedback to peers and faculty members was lacking, and participants did not reflect on receiving feedback from all sources.
The extent to which feedback was sought is not known, so additional research is needed to clarify this finding. Many factors could contribute to this pattern, including the perceived need to obtain feedback from certain groups, the feasibility and convenience of sampling, the perceived relative importance of feedback from more experienced or more educated groups, or the comfort level of the participants to give feedback to and receive it from certain groups. Teaching certificate programs should consider increased emphasis on 360-degree evaluations of teaching and both giving and receiving feedback to help participants understand the necessity and value of feedback from others. If participants gain comfort in this activity, they could provide higher quality feedback to others and potentially could benefit more from the feedback they receive.
Participants were not prompted to write about opportunities they did not get to experience; therefore, it can be assumed that participants discussing this theme must have been expecting to participate in such activities. By identifying these patterns, program directors can get an idea of skills that some participants might want to gain from the program. As expected, more than 70% of the themes coded for no opportunity were expressed in the midpoint essays. Of the remaining nine items discussed in the final essays, grading students and writing test questions accounted for the most themes discussed. Our program has taken these comments into consideration, incorporating additional opportunities for examination item review sessions to give participants insight on how to develop and evaluate examination items.
Qualitative evaluation of participant reflections can provide feedback on teaching certificate program effectiveness. These reflections provide a detailed view of how participants develop teaching skills throughout the program and impart realizations of the effectiveness of the program. By using reflective essays as quality indicators, program directors can shape program content to better develop teachers. At the same time, the reflection itself can aid participants in the development of a well-informed, highly individualized written statement of teaching philosophy, another potential quality measure of the teaching certificate program. This research did not evaluate participants’ teaching philosophy statements. However, qualitative evaluations of these documents also could provide a proxy of the teaching certificate program’s effectiveness in increasing participants’ awareness of their own teaching style and ideals.
Although our qualitative evaluation was rigorous and followed the guidelines presented by Anderson,24 care must be taken when generalizing the results to other programs. Our data represent the products of a single teaching certificate program and are specific to the participants and experiences of this program. The characteristics of the residents and preceptors completing the program are varied, but they may reflect these groups at other institutions.
The use of quantitative and qualitative measures to evaluate this teaching certificate program is a strength of the study, and additional analyses are needed to determine if participants’ reflective abilities remain constant over time and predict future teaching performance. Finally, because of the extensive nature of our evaluation and the volume of the qualitative data, the potential exists for coding errors and inconsistencies between coders. Several mechanisms were in place to limit these inaccuracies and preserve the integrity of the data.
Participation in the teaching certificate program appeared to increase confidence and enhance awareness of strengths through participant reflection. Pharmacy residents and preceptors frequently achieved the highest level of reflection (critical reflection) in global self-assessments of teaching experiences. Such deep reflection is indicative of professional development because teaching certificate program participants evidenced change not only in teaching attitudes, but also teaching behaviors, as discussed in written essays. Just as reflective exercises are emphasized for pharmacy students and residents, findings from this analysis suggest that teaching certificate programs should strongly consider emphasizing purposeful critical reflection through required reflective exercises at multiple points within the program curricula. Qualitative evaluation of participant reflections can provide quality indicators to assist program directors in shaping program content.
1. Kember D, Jones A, Loke A, et al. Determining the level of reflective thinking from students’ written journals using a coding scheme based on the work of Mezirow. Int J Life Educ. 1999;18(1):18–30.
2. Schön DA. The Reflective Practitioner: How Professionals Think in Action. New York: Basic Books, Inc.; 1983.
3. Dewey J. How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process. Boston: D C Heath and Co.; 1933.
4. Boud D, Keogh R, Walker D. Reflection: Turning Experience into Learning. London: Kogan Page; 1985.
5. Regehr G, Hodges B, Tiberius R, Lofchy J. Measuring self-assessment skills: an innovative relative ranking model. Acad Med. 1996;71(10):S52–S54.[PubMed]
6. Fjortoft N. Self-assessment in pharmacy education. Am J Pharm Educ. 2006;70(3):Article 64.[PMC free article][PubMed]
7. Romanelli F, Smith KM, Brandt BF. Certificate program in teaching for pharmacy residents. Am J Health Syst Pharm. 2001;58:896–898.[PubMed]
8. Corbin J, Strauss A. Grounded theory research: procedures, canons, and evaluative criteria. Qual Sociol. 1990;13(1):3–21.
9. Castleberry A, Payakachat N, Neill KK, Franks AM. A 5-year evaluation of a postgraduate teaching certificate program’s effect on self-perceived teaching abilities. Curr Pharm Teach and Learn. 2014;6(3):401–405.
10. Kember D, McKay J, Sinclair K, Wong FKY. A four-category scheme for coding and assessing the level of reflection in written work. Assess Eval High Educ. 2008;33(4):1–10.
11. Van Manen M. On the epistemology of reflective practice. Teachers and Teaching: Theory and Practice. 1995;1(1):33–50.
12. Daudelin MW. Learning from experience through reflection. Organ Dyn. 1996;24(3):36–48.
13. Hildebrand C, Trowbridge E, Roach MA, Sullivan AG, Broman AT, Vogelman B. Resident self-assessment and self-reflection: University of Wisconsin-Madison’s five-year study. J Gen Intern Med. 2009;24(3):361–365.[PMC free article][PubMed]
14. McNeill H, Brown JM, Shaw NJ. First year specialist trainees’ engagement with reflective practice in the e-portfolio. Adv Health Sci Educ Theory Pract. 2010;15(4):547–558.[PubMed]
15. Mamede S, Schmidt HG, Penaforte JC. Effects of reflective practice on the accuracy of medical diagnoses. Med Educ. 2008;42(5):468–475.[PubMed]
16. Wong FKY, Kember D, Chung LYF, Yan L. Assessing the level of student reflection from reflective journals. J Adv Nurs. 1995;22(1):48–57.[PubMed]
17. Getliffe KA. An examination of the use of reflection in the assessment of practice for undergraduate nursing students. Int J Nurs Stud. 1996;33(4):361–374.[PubMed]
18. Rodgers C. Defining reflection: another look at John Dewey and reflective thinking. Teach Coll Rec. 2002;104(4):842–866.
19. Accreditation Council for Pharmacy Education. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree, 2011. https://www.acpe-accredit.org/standards/. Accessed December 15, 2014.
20. Accreditation Council for Pharmacy Education. Accreditation Standards and Key Elements for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree, 2016. https://www.acpe-accredit.org/pdf/Standards2016FINAL.pdf. Accessed February 24, 2015.
21. American Society of Health-System Pharmacists. Required Competency Areas, Goals, and Objectives, for Postgraduate Year One Pharmacy Residencies. http://www.ashp.org/DocLibrary/Accreditation/Newly-approved-Competency-Areas-Goals-and-Objectives-September-2014.pdf. Accessed February 19, 2015.
22. Wallman A, Lindblad AK, Hall S, Lundmark A, Ring L. A categorization scheme for assessing pharmacy students’ level of reflection during internships. Am J Pharm Educ. 2008;72(1):1–10.[PMC free article][PubMed]
23. Black PE, Plowright D. A multi-dimensional model of reflective learning for professional development. Reflective Practice. 2010;11(2):245–258.
24. Anderson C. Presenting and evaluating qualitative research. Am J Pharm Educ. 2010;74(8):1–7.[PMC free article][PubMed]
Articles from American Journal of Pharmaceutical Education are provided here courtesy of American Association of Colleges of Pharmacy
In an earlier paper,1 we presented an introduction to using qualitative research methods in pharmacy practice. In this article, we review some principles of the collection, analysis, and management of qualitative data to help pharmacists interested in doing research in their practice to continue their learning in this area. Qualitative research can help researchers to access the thoughts and feelings of research participants, which can enable development of an understanding of the meaning that people ascribe to their experiences. Whereas quantitative research methods can be used to determine how many people undertake particular behaviours, qualitative methods can help researchers to understand how and why such behaviours take place. Within the context of pharmacy practice research, qualitative approaches have been used to examine a diverse array of topics, including the perceptions of key stakeholders regarding prescribing by pharmacists and the postgraduation employment experiences of young pharmacists (see “Further Reading” section at the end of this article).
In the previous paper,1 we outlined 3 commonly used methodologies: ethnography2, grounded theory3, and phenomenology.4 Briefly, ethnography involves researchers using direct observation to study participants in their “real life” environment, sometimes over extended periods. Grounded theory and its later modified versions (e.g., Strauss and Corbin5) use face-to-face interviews and interactions such as focus groups to explore a particular research phenomenon and may help in clarifying a less-well-understood problem, situation, or context. Phenomenology shares some features with grounded theory (such as an exploration of participants’ behaviour) and uses similar techniques to collect data, but it focuses on understanding how human beings experience their world. It gives researchers the opportunity to put themselves in another person’s shoes and to understand the subjective experiences of participants.6 Some researchers use qualitative methodologies but adopt a different standpoint, and an example of this appears in the work of Thurston and others,7 discussed later in this paper.
Qualitative work requires reflection on the part of researchers, both before and during the research process, as a way of providing context and understanding for readers. When being reflexive, researchers should not try to simply ignore or avoid their own biases (as this would likely be impossible); instead, reflexivity requires researchers to reflect upon and clearly articulate their position and subjectivities (world view, perspectives, biases), so that readers can better understand the filters through which questions were asked, data were gathered and analyzed, and findings were reported. From this perspective, bias and subjectivity are not inherently negative but they are unavoidable; as a result, it is best that they be articulated up-front in a manner that is clear and coherent for readers.
THE PARTICIPANT’S VIEWPOINT
What qualitative study seeks to convey is why people have thoughts and feelings that might affect the way they behave. Such study may occur in any number of contexts, but here, we focus on pharmacy practice and the way people behave with regard to medicines use (e.g., to understand patients’ reasons for nonadherence with medication therapy or to explore physicians’ resistance to pharmacists’ clinical suggestions). As we suggested in our earlier article,1 an important point about qualitative research is that there is no attempt to generalize the findings to a wider population. Qualitative research is used to gain insights into people’s feelings and thoughts, which may provide the basis for a future stand-alone qualitative study or may help researchers to map out survey instruments for use in a quantitative study. It is also possible to use different types of research in the same study, an approach known as “mixed methods” research, and further reading on this topic may be found at the end of this paper.
The role of the researcher in qualitative research is to attempt to access the thoughts and feelings of study participants. This is not an easy task, as it involves asking people to talk about things that may be very personal to them. Sometimes the experiences being explored are fresh in the participant’s mind, whereas on other occasions reliving past experiences may be difficult. However the data are being collected, a primary responsibility of the researcher is to safeguard participants and their data. Mechanisms for such safeguarding must be clearly articulated to participants and must be approved by a relevant research ethics review board before the research begins. Researchers and practitioners new to qualitative research should seek advice from an experienced qualitative researcher before embarking on their project.
Whatever philosophical standpoint the researcher is taking and whatever the data collection method (e.g., focus group, one-to-one interviews), the process will involve the generation of large amounts of data. In addition to the variety of study methodologies available, there are also different ways of making a record of what is said and done during an interview or focus group, such as taking handwritten notes or video-recording. If the researcher is audio- or video-recording data collection, then the recordings must be transcribed verbatim before data analysis can begin. As a rough guide, it can take an experienced researcher/transcriber 8 hours to transcribe one 45-minute audio-recorded interview, a process than will generate 20–30 pages of written dialogue.
Many researchers will also maintain a folder of “field notes” to complement audio-taped interviews. Field notes allow the researcher to maintain and comment upon impressions, environmental contexts, behaviours, and nonverbal cues that may not be adequately captured through the audio-recording; they are typically handwritten in a small notebook at the same time the interview takes place. Field notes can provide important context to the interpretation of audio-taped data and can help remind the researcher of situational factors that may be important during data analysis. Such notes need not be formal, but they should be maintained and secured in a similar manner to audio tapes and transcripts, as they contain sensitive information and are relevant to the research. For more information about collecting qualitative data, please see the “Further Reading” section at the end of this paper.
DATA ANALYSIS AND MANAGEMENT
If, as suggested earlier, doing qualitative research is about putting oneself in another person’s shoes and seeing the world from that person’s perspective, the most important part of data analysis and management is to be true to the participants. It is their voices that the researcher is trying to hear, so that they can be interpreted and reported on for others to read and learn from. To illustrate this point, consider the anonymized transcript excerpt presented in Appendix 1, which is taken from a research interview conducted by one of the authors (J.S.). We refer to this excerpt throughout the remainder of this paper to illustrate how data can be managed, analyzed, and presented.
Interpretation of Data
Interpretation of the data will depend on the theoretical standpoint taken by researchers. For example, the title of the research report by Thurston and others,7 “Discordant indigenous and provider frames explain challenges in improving access to arthritis care: a qualitative study using constructivist grounded theory,” indicates at least 2 theoretical standpoints. The first is the culture of the indigenous population of Canada and the place of this population in society, and the second is the social constructivist theory used in the constructivist grounded theory method. With regard to the first standpoint, it can be surmised that, to have decided to conduct the research, the researchers must have felt that there was anecdotal evidence of differences in access to arthritis care for patients from indigenous and non-indigenous backgrounds. With regard to the second standpoint, it can be surmised that the researchers used social constructivist theory because it assumes that behaviour is socially constructed; in other words, people do things because of the expectations of those in their personal world or in the wider society in which they live. (Please see the “Further Reading” section for resources providing more information about social constructivist theory and reflexivity.) Thus, these 2 standpoints (and there may have been others relevant to the research of Thurston and others7) will have affected the way in which these researchers interpreted the experiences of the indigenous population participants and those providing their care. Another standpoint is feminist standpoint theory which, among other things, focuses on marginalized groups in society. Such theories are helpful to researchers, as they enable us to think about things from a different perspective. Being aware of the standpoints you are taking in your own research is one of the foundations of qualitative work. Without such awareness, it is easy to slip into interpreting other people’s narratives from your own viewpoint, rather than that of the participants.
To analyze the example in Appendix 1, we will adopt a phenomenological approach because we want to understand how the participant experienced the illness and we want to try to see the experience from that person’s perspective. It is important for the researcher to reflect upon and articulate his or her starting point for such analysis; for example, in the example, the coder could reflect upon her own experience as a female of a majority ethnocultural group who has lived within middle class and upper middle class settings. This personal history therefore forms the filter through which the data will be examined. This filter does not diminish the quality or significance of the analysis, since every researcher has his or her own filters; however, by explicitly stating and acknowledging what these filters are, the researcher makes it easer for readers to contextualize the work.
Transcribing and Checking
For the purposes of this paper it is assumed that interviews or focus groups have been audio-recorded. As mentioned above, transcribing is an arduous process, even for the most experienced transcribers, but it must be done to convert the spoken word to the written word to facilitate analysis. For anyone new to conducting qualitative research, it is beneficial to transcribe at least one interview and one focus group. It is only by doing this that researchers realize how difficult the task is, and this realization affects their expectations when asking others to transcribe. If the research project has sufficient funding, then a professional transcriber can be hired to do the work. If this is the case, then it is a good idea to sit down with the transcriber, if possible, and talk through the research and what the participants were talking about. This background knowledge for the transcriber is especially important in research in which people are using jargon or medical terms (as in pharmacy practice). Involving your transcriber in this way makes the work both easier and more rewarding, as he or she will feel part of the team. Transcription editing software is also available, but it is expensive. For example, ELAN (more formally known as EUDICO Linguistic Annotator, developed at the Technical University of Berlin)8 is a tool that can help keep data organized by linking media and data files (particularly valuable if, for example, video-taping of interviews is complemented by transcriptions). It can also be helpful in searching complex data sets. Products such as ELAN do not actually automatically transcribe interviews or complete analyses, and they do require some time and effort to learn; nonetheless, for some research applications, it may be a valuable to consider such software tools.
All audio recordings should be transcribed verbatim, regardless of how intelligible the transcript may be when it is read back. Lines of text should be numbered. Once the transcription is complete, the researcher should read it while listening to the recording and do the following: correct any spelling or other errors; anonymize the transcript so that the participant cannot be identified from anything that is said (e.g., names, places, significant events); insert notations for pauses, laughter, looks of discomfort; insert any punctuation, such as commas and full stops (periods) (see Appendix 1 for examples of inserted punctuation), and include any other contextual information that might have affected the participant (e.g., temperature or comfort of the room).
Dealing with the transcription of a focus group is slightly more difficult, as multiple voices are involved. One way of transcribing such data is to “tag” each voice (e.g., Voice A, Voice B). In addition, the focus group will usually have 2 facilitators, whose respective roles will help in making sense of the data. While one facilitator guides participants through the topic, the other can make notes about context and group dynamics. More information about group dynamics and focus groups can be found in resources listed in the “Further Reading” section.
Reading between the Lines
During the process outlined above, the researcher can begin to get a feel for the participant’s experience of the phenomenon in question and can start to think about things that could be pursued in subsequent interviews or focus groups (if appropriate). In this way, one participant’s narrative informs the next, and the researcher can continue to interview until nothing new is being heard or, as it says in the text books, “saturation is reached”. While continuing with the processes of coding and theming (described in the next 2 sections), it is important to consider not just what the person is saying but also what they are not saying. For example, is a lengthy pause an indication that the participant is finding the subject difficult, or is the person simply deciding what to say? The aim of the whole process from data collection to presentation is to tell the participants’ stories using exemplars from their own narratives, thus grounding the research findings in the participants’ lived experiences.
Smith9 suggested a qualitative research method known as interpretative phenomenological analysis, which has 2 basic tenets: first, that it is rooted in phenomenology, attempting to understand the meaning that individuals ascribe to their lived experiences, and second, that the researcher must attempt to interpret this meaning in the context of the research. That the researcher has some knowledge and expertise in the subject of the research means that he or she can have considerable scope in interpreting the participant’s experiences. Larkin and others10 discussed the importance of not just providing a description of what participants say. Rather, interpretative phenomenological analysis is about getting underneath what a person is saying to try to truly understand the world from his or her perspective.
Once all of the research interviews have been transcribed and checked, it is time to begin coding. Field notes compiled during an interview can be a useful complementary source of information to facilitate this process, as the gap in time between an interview, transcribing, and coding can result in memory bias regarding nonverbal or environmental context issues that may affect interpretation of data.
Coding refers to the identification of topics, issues, similarities, and differences that are revealed through the participants’ narratives and interpreted by the researcher. This process enables the researcher to begin to understand the world from each participant’s perspective. Coding can be done by hand on a hard copy of the transcript, by making notes in the margin or by highlighting and naming sections of text. More commonly, researchers use qualitative research software (e.g., NVivo, QSR International Pty Ltd; www.qsrinternational.com/products_nvivo.aspx) to help manage their transcriptions. It is advised that researchers undertake a formal course in the use of such software or seek supervision from a researcher experienced in these tools.
Returning to Appendix 1 and reading from lines 8–11, a code for this section might be “diagnosis of mental health condition”, but this would just be a description of what the participant is talking about at that point. If we read a little more deeply, we can ask ourselves how the participant might have come to feel that the doctor assumed he or she was aware of the diagnosis or indeed that they had only just been told the diagnosis. There are a number of pauses in the narrative that might suggest the participant is finding it difficult to recall that experience. Later in the text, the participant says “nobody asked me any questions about my life” (line 19). This could be coded simply as “health care professionals’ consultation skills”, but that would not reflect how the participant must have felt never to be asked anything about his or her personal life, about the participant as a human being. At the end of this excerpt, the participant just trails off, recalling that no-one showed any interest, which makes for very moving reading. For practitioners in pharmacy, it might also be pertinent to explore the participant’s experience of akathisia and why this was left untreated for 20 years.
One of the questions that arises about qualitative research relates to the reliability of the interpretation and representation of the participants’ narratives. There are no statistical tests that can be used to check reliability and validity as there are in quantitative research. However, work by Lincoln and Guba11 suggests that there are other ways to “establish confidence in the ‘truth’ of the findings” (p. 218). They call this confidence “trustworthiness” and suggest that there are 4 criteria of trustworthiness: credibility (confidence in the “truth” of the findings), transferability (showing that the findings have applicability in other contexts), dependability (showing that the findings are consistent and could be repeated), and confirmability (the extent to which the findings of a study are shaped by the respondents and not researcher bias, motivation, or interest).
One way of establishing the “credibility” of the coding is to ask another researcher to code the same transcript and then to discuss any similarities and differences in the 2 resulting sets of codes. This simple act can result in revisions to the codes and can help to clarify and confirm the research findings.
Theming refers to the drawing together of codes from one or more transcripts to present the findings of qualitative research in a coherent and meaningful way. For example, there may be examples across participants’ narratives of the way in which they were treated in hospital, such as “not being listened to” or “lack of interest in personal experiences” (see Appendix 1). These may be drawn together as a theme running through the narratives that could be named “the patient’s experience of hospital care”. The importance of going through this process is that at its conclusion, it will be possible to present the data from the interviews using quotations from the individual transcripts to illustrate the source of the researchers’ interpretations. Thus, when the findings are organized for presentation, each theme can become the heading of a section in the report or presentation. Underneath each theme will be the codes, examples from the transcripts, and the researcher’s own interpretation of what the themes mean. Implications for real life (e.g., the treatment of people with chronic mental health problems) should also be given.
In this final section of this paper, we describe some ways of drawing together or “synthesizing” research findings to represent, as faithfully as possible, the meaning that participants ascribe to their life experiences. This synthesis is the aim of the final stage of qualitative research. For most readers, the synthesis of data presented by the researcher is of crucial significance—this is usually where “the story” of the participants can be distilled, summarized, and told in a manner that is both respectful to those participants and meaningful to readers. There are a number of ways in which researchers can synthesize and present their findings, but any conclusions drawn by the researchers must be supported by direct quotations from the participants. In this way, it is made clear to the reader that the themes under discussion have emerged from the participants’ interviews and not the mind of the researcher. The work of Latif and others12 gives an example of how qualitative research findings might be presented.
Planning and Writing the Report
As has been suggested above, if researchers code and theme their material appropriately, they will naturally find the headings for sections of their report. Qualitative researchers tend to report “findings” rather than “results”, as the latter term typically implies that the data have come from a quantitative source. The final presentation of the research will usually be in the form of a report or a paper and so should follow accepted academic guidelines. In particular, the article should begin with an introduction, including a literature review and rationale for the research. There should be a section on the chosen methodology and a brief discussion about why qualitative methodology was most appropriate for the study question and why one particular methodology (e.g., interpretative phenomenological analysis rather than grounded theory) was selected to guide the research. The method itself should then be described, including ethics approval, choice of participants, mode of recruitment, and method of data collection (e.g., semistructured interviews or focus groups), followed by the research findings, which will be the main body of the report or paper. The findings should be written as if a story is being told; as such, it is not necessary to have a lengthy discussion section at the end. This is because much of the discussion will take place around the participants’ quotes, such that all that is needed to close the report or paper is a summary, limitations of the research, and the implications that the research has for practice. As stated earlier, it is not the intention of qualitative research to allow the findings to be generalized, and therefore this is not, in itself, a limitation.
Planning out the way that findings are to be presented is helpful. It is useful to insert the headings of the sections (the themes) and then make a note of the codes that exemplify the thoughts and feelings of your participants. It is generally advisable to put in the quotations that you want to use for each theme, using each quotation only once. After all this is done, the telling of the story can begin as you give your voice to the experiences of the participants, writing around their quotations. Do not be afraid to draw assumptions from the participants’ narratives, as this is necessary to give an in-depth account of the phenomena in question. Discuss these assumptions, drawing on your participants’ words to support you as you move from one code to another and from one theme to the next. Finally, as appropriate, it is possible to include examples from literature or policy documents that add support for your findings. As an exercise, you may wish to code and theme the sample excerpt in Appendix 1 and tell the participant’s story in your own way. Further reading about “doing” qualitative research can be found at the end of this paper.
Qualitative research can help researchers to access the thoughts and feelings of research participants, which can enable development of an understanding of the meaning that people ascribe to their experiences. It can be used in pharmacy practice research to explore how patients feel about their health and their treatment. Qualitative research has been used by pharmacists to explore a variety of questions and problems (see the “Further Reading” section for examples). An understanding of these issues can help pharmacists and other health care professionals to tailor health care to match the individual needs of patients and to develop a concordant relationship. Doing qualitative research is not easy and may require a complete rethink of how research is conducted, particularly for researchers who are more familiar with quantitative approaches. There are many ways of conducting qualitative research, and this paper has covered some of the practical issues regarding data collection, analysis, and management. Further reading around the subject will be essential to truly understand this method of accessing peoples’ thoughts and feelings to enable researchers to tell participants’ stories.
Appendix 1. Excerpt from a sample transcript
The participant (age late 50s) had suffered from a chronic mental health illness for 30 years. The participant had become a “revolving door patient,” someone who is frequently in and out of hospital. As the participant talked about past experiences, the researcher asked:
What was treatment like 30 years ago?
Umm—well it was pretty much they could do what they wanted with you because I was put into the er, the er kind of system er, I was just on
endless section threes.
But what I didn’t realize until later was that if you haven’t actually posed a threat to someone or yourself they can’t really do that but I didn’t know
that. So wh-when I first went into hospital they put me on the forensic ward ’cause they said, “We don’t think you’ll stay here we think you’ll just
run-run away.” So they put me then onto the acute admissions ward and – er – I can remember one of the first things I recall when I got onto that
ward was sitting down with a er a Dr XXX. He had a book this thick [gestures] and on each page it was like three questions and he went through
all these questions and I answered all these questions. So we’re there for I don’t maybe two hours doing all that and he asked me he said “well
when did somebody tell you then that you have schizophrenia” I said “well nobody’s told me that” so he seemed very surprised but nobody had
actually [pause] whe-when I first went up there under police escort erm the senior kind of consultants people I’d been to where I was staying and
ermm so er [pause] I . . . the, I can remember the very first night that I was there and given this injection in this muscle here [gestures] and just
having dreadful side effects the next day I woke up [pause]
. . . and I suffered that akathesia I swear to you, every minute of every day for about 20 years.
Oh how awful.
And that side of it just makes life impossible so the care on the wards [pause] umm I don’t know it’s kind of, it’s kind of hard to put into words
[pause]. Because I’m not saying they were sort of like not friendly or interested but then nobody ever seemed to want to talk about your life [pause]
nobody asked me any questions about my life. The only questions that came into was they asked me if I’d be a volunteer for these student exams
and things and I said “yeah” so all the questions were like “oh what jobs have you done,” er about your relationships and things and er but
nobody actually sat down and had a talk and showed some interest in you as a person you were just there basically [pause] um labelled and you
know there was there was [pause] but umm [pause] yeah . . .
This article is the 10th in the CJHP Research Primer Series, an initiative of the CJHP Editorial Board and the CSHP Research Committee. The planned 2-year series is intended to appeal to relatively inexperienced researchers, with the goal of building research capacity among practising pharmacists. The articles, presenting simple but rigorous guidance to encourage and support novice researchers, are being solicited from authors with appropriate expertise.
Previous articles in this series:
Bond CM. The research jigsaw: how to get started. Can J Hosp Pharm. 2014;67(1):28–30.
Tully MP. Research: articulating questions, generating hypotheses, and choosing study designs. Can J Hosp Pharm. 2014;67(1):31–4.
Loewen P. Ethical issues in pharmacy practice research: an introductory guide. Can J Hosp Pharm. 2014;67(2):133–7.
Tsuyuki RT. Designing pharmacy practice research trials. Can J Hosp Pharm. 2014;67(3):226–9.
Bresee LC. An introduction to developing surveys for pharmacy practice research. Can J Hosp Pharm. 2014;67(4):286–91.
Gamble JM. An introduction to the fundamentals of cohort and case–control studies. Can J Hosp Pharm. 2014;67(5):366–72.
Austin Z, Sutton J. Qualitative research: getting started. Can J Hosp Pharm. 2014;67(6):436–40.
Houle S. An introduction to the fundamentals of randomized controlled trials in pharmacy research. Can J Hosp Pharm. 2014; 68(1):28–32.
Charrois TL. Systematic reviews: What do you need to know to get started? Can J Hosp Pharm. 2014;68(2):144–8.
Competing interests: None declared.
1. Austin ZA, Sutton J. Qualitative research: getting started. Can J Hosp Pharm. 2014;67(6):436–40.[PMC free article][PubMed]
2. Hammersley M, Atkinson P. Ethnography: principles in practice. London (UK): Taylor and Francis; 2007.
3. What is grounded theory? Mill Valley (CA): Grounded Theory Institute; 2008. [cited 2015 May 2]. Available from: www.groundedtheory.com/what-is-gt.aspx.
4. Brewer J. Naturalism. In: Miller RL, Brewer JD, editors. The A–Z of social research. London (UK): Sage Publications; 2003. pp. 147–59.
5. Strauss AL, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. Thousand Oaks (CA): Sage Publications; 1998.
6. Smith JA, Jarman M, Osborn M. Doing interpretative phenomenological analysis. In: Murray M, Chamberlain K, editors. Qualitative health psychology: theories and methods. London (UK): Sage Publications; 1999. pp. 218–40.
7. Thurston WE, Coupal S, Jones CA, Crowshoe LF, Marshall DA, Homik J, et al. Discordant indigenous and provider frames explain challenges in improving access to arthritis care: a qualitative study using constructivist grounded theory. Int J Equity Health. 2014;13:46. doi: 10.1186/1475-9276-13-46.[PMC free article][PubMed][Cross Ref]
8. Rosenfelder R. A short introduction to transcribing with ELAN. Philadelphia (PA): University of Pennsylvania Linguistics Lab; 2011. [cited 2015 Jun 4]. Available from: http://fave.ling.upenn.edu/downloads/ELAN_Introduction.pdfTitle.
9. Smith JA. Beyond the divide between cognition and discourse: using interpretative phenomenological analysis in health psychology. Psychol Health. 1996;11(2):261–71. doi: 10.1080/08870449608400256.[Cross Ref]
10. Larkin M, Watts S, Clifton E. Giving voice and making sense in interpretative phenomenological analysis. Qual Res Psychol. 2006;3(2):102–20. doi: 10.1191/1478088706qp062oa.[Cross Ref]
11. Lincoln YS, Guba EG. Naturalistic inquiry. Thousand Oaks (CA): Sage Publications; 1985.
12. Latif A, Boardman HF, Pollock K. A qualitative study exploring the impact and consequence of the medicines use review service on pharmacy support-staff. Pharm Pract. 2013;11(2):118–24.[PMC free article][PubMed]
Examples of Qualitative Research in Pharmacy Practice
- Farrell B, Pottie K, Woodend K, Yao V, Dolovich L, Kennie N, et al. Shifts in expectations: evaluating physicians’ perceptions as pharmacists integrated into family practice. J Interprof Care. 2010;24(1):80–9.[PubMed]
- Gregory P, Austin Z. Postgraduation employment experiences of new pharmacists in Ontario in 2012–2013. Can Pharm J. 2014;147(5):290–9.[PMC free article][PubMed]
- Marks PZ, Jennnings B, Farrell B, Kennie-Kaulbach N, Jorgenson D, Pearson-Sharpe J, et al. “I gained a skill and a change in attitude”: a case study describing how an online continuing professional education course for pharmacists supported achievement of its transfer to practice outcomes. Can J Univ Contin Educ. 2014;40(2):1–18.
- Nair KM, Dolovich L, Brazil K, Raina P. It’s all about relationships: a qualitative study of health researchers’ perspectives on interdisciplinary research. BMC Health Serv Res. 2008;8:110.[PMC free article][PubMed]
- Pojskic N, MacKeigan L, Boon H, Austin Z. Initial perceptions of key stakeholders in Ontario regarding independent prescriptive authority for pharmacists. Res Soc Adm Pharm. 2014;10(2):341–54.[PubMed]
Qualitative Research in General
- Breakwell GM, Hammond S, Fife-Schaw C. Research methods in psychology. Thousand Oaks (CA): Sage Publications; 1995.
- Given LM. 100 questions (and answers) about qualitative research. Thousand Oaks (CA): Sage Publications; 2015.
- Miles B, Huberman AM. Qualitative data analysis. Thousand Oaks (CA): Sage Publications; 2009.
- Patton M. Qualitative research and evaluation methods. Thousand Oaks (CA): Sage Publications; 2002.
- Willig C. Introducing qualitative research in psychology. Buckingham (UK): Open University Press; 2001.
Group Dynamics in Focus Groups
- Farnsworth J, Boon B. Analysing group dynamics within the focus group. Qual Res. 2010;10(5):605–24.
- Creswell J. Research design: qualitative, quantitative, and mixed methods approaches. Thousand Oaks (CA): Sage Publications; 2009.
Collecting Qualitative Data
- Arksey H, Knight P. Interviewing for social scientists: an introductory resource with examples. Thousand Oaks (CA): Sage Publications; 1999.
- Guest G, Namey EE, Mitchel ML. Collecting qualitative data: a field manual for applied research. Thousand Oaks (CA): Sage Publications; 2013.
Constructivist Grounded Theory
- Charmaz K. Grounded theory: objectivist and constructivist methods. In: Denzin N, Lincoln Y, editors. Handbook of qualitative research. 2nd ed. Thousand Oaks (CA): Sage Publications; 2000. pp. 509–35.