ABSTRACT
Public health nurses from 13 local public health agencies and nurse educators from five schools of nursing developed population- based public health nursing competencies for new graduates and novice public health nurses. Educators in one nursing program used a portfolio assignment to measure achievement of the competencies by traditional and RN to BSN students in a community health nursing course. Data were collected from surveys and focus groups to determine students’ responses to the portfolio and their use of population-based public health nursing concepts. The assignment enhanced students’ critical thinking skills; however, concerns about the structure and evaluation of the portfolio decreased student satisfaction. Recommendations are made for improving the portfolio format, increasing students’ valuing of the portfolio, managing the tension between assessment and learning, and orienting clinical agency staff and nursing instructors.
Keywords Portfolio – Competencies – Evaluation – Population- Based Public Health Nursing – Baccalaureate Nursing Students – Critical Thinking
IN TODAY’S CHALLENGING HEALTH CARE ENVIRONMENT, where more must be done with fewer dollars, nurses must be able to clearly articulate what they do. The development of clearly defined competencies, and the identification of strategies to measure those competencies, is critical to the education of a competent nursing workforce. * THIS STUDY EVALUATED THE EFFECTIVENESS OF USING PORTFOLIOS TO ASSESS THE ACHIEVEMENT OF POPULATION-BASED PUBLIC HEALTH NURSING COMPETENCIES BY BACCALAUREATE NURSING STUDENTS. The competencies were developed for new graduates and novice public health nurses by the Henry Street Consortium, a partnership of five colleges/schools of nursing and 13 local public health agencies (1). The Henry Street Consortium is one of five Local Linking Projects supported by the Public Health Nursing Practice and Education to Promote Population Health Grant, awarded to the Minnesota Department of Health by the Division of Nursing, Health Services and Resources Administration. The grant also provided for training and support of public health nurse preceptors in local agencies (2). Members of the consortium reviewed public health and public health nursing competencies found in the literature and spoke with public health nurses from local agencies before identifying 11 competencies. (The competencies are itemized in Table 4.) The competencies are used in conjunction with the Public Health Intervention Wheel, which specifies three levels of practice (community, systems, and individual/family) and 17 public health interventions (3). (See Figure 1.)
Review of Literature The increasing emphasis on population health requires reframing and updating the knowledge and skills of public health professionals. Today’s public health nurse (PHN) needs knowledge about the measurement of health status, data analysis strategies, organizational change, health economics, and the politics of health. Because PHNs collaborate with other disciplines and communities to solve problems, develop, evaluate, and implement health policy, measure quality of care and health outcomes, and analyze how social determinants affect health (4), they require skills in working with interdisciplinary teams, conducting program evaluation, and building coalitions (5).
Competency-based models provide a structure for identifying and measuring student learning outcomes that are connected to the skills needed for contemporary nursing practice (6,7). Kaiser and Rudolph (8), who evaluated the use of American Nurses Association (ANA) standards for community and public health nursing (9,10) as the basis for a clinical evaluation tool, pointed to the need for additional constructs for the identification of competencies not reflected in ANA standards. Although the standards were found to be useful for organizing the performance evaluation tool, measurement strategies and evaluation tools must accurately reflect expected competencies for practice (11). Planning processes for populations add complexity to public health nursing practice. Thus, the competencies required must reflect the socialization and learning experiences needed for populationbased care.
Figure I
Public Health Intervention Wheel
The literature identifies many benefits derived from the use of portfolios in teaching and in the evaluation of competency achievement. Portfolios have the potential to enhance the achievement of learning outcomes and competencies while providing a structure for assessing the achievement of competencies (12) and documenting the achievement of skills and learning experiences. The portfolio is defined as a “focused purposeful collection of student work that documents evidence of traditional and nontraditional sources of student learning, progress, and achievements over time” (13, p. 209). It results from a collaborative process between students and faculty, with the faculty role that of coach and mentor rather than didactic teacher (13-15).
In an integrative literature review and meta-analysis of the use of portfolios and the assessment of competence, McMullan and colleagues (16) determined that the use of portfolios is based on theories of adult and experiential learning. Portfolios can be an effective strategy to promote lifelong learning and professional goal achievement for nurses at the novice, experienced, and expert levels (17,18).
Self-reflection, whereby one analyzes and synthesizes one’s thoughts and actions (13), is a critical component of educational portfolios. Sorrell and colleagues (19) analyzed nursing students’ reflective essays for critical thinking outcomes. They identified two themes: 1) critical thinking was expressed by “seeing” things related to nursing practice in new ways, and 2) creative thinking was present, along with analysis and synthesis. McMullan and colleagues pointed out that the reflective component of the portfolio encourages integration of ethics and values and helps frame practice in theory (16).
Portfolios in which students reflect on their progress in competency development have been found to promote active student engagement in the learning process (20). Karlowicz (21) suggested that the use of portfolios increases student responsibility for learning, promotes self-confidence as students witness their own development, and enhances faculty-student interactions. For educators, the use of portfolios to evaluate student learning can encourage the examination of the relevance of teaching strategies and assignments to course objectives and program outcomes (21).
Limitations of the portfolio approach must be considered. First, there is a lack of evidence to support the validity of the portfolio score as a measure of the acquisition of knowledge and skills that enable students to provide competent nursing care. second, the reliability of the scoring rubric between faculty members must be tested. Other concerns include the time commitment needed for adequate portfolio development and storage issues (21).
Method A portfolio assignment to measure achievement of the population-based public health nursing competencies was developed and piloted with two groups of baccalaureate students: 43 generic baccalaureate students working in six clinical groups in local public health agencies, and 45 RN to BSN students assigned to preceptors in a variety of community settings. Seven faculty members who taught community health nursing participated.
Students were instructed to complete portfolios with four components: 1) a notebook with sections for each competency; 2) a table of contents; 3) artifacts, which could be assignments, evaluations, teaching plans or outlines, photos, pamphlets on resources or programs, and meeting agendas; and 4) reflections for each competency. Students could use the same artifact for more than one competency and could include several artifacts for one competency if they desired.
Table I. Portfolio Evaluation: Student Survey Responses
An online discussion forum over the duration of the course and a portfolio “help lab” offered several weeks before the portfolio was due provided support to students as they worked on the assignment. Faculty members developed an evaluation rubric consisting of four criteria:
* Artifacts demonstrate the accomplishment of competencies.
* Reflections make connections to population-based public health nursing theory, knowledge, and practice.
* Personal and professional growth through self-examination is demonstrated.
* The portfolio is clearly organized, scholarly, and creative. During the grading process, faculty members met to compare scoring strategies.
Evaluation methods included a survey, focus groups with students, and an analysis of portfolio content. Generic BSN students (n = 35) completed an electronic version of the survey; RN to BSN students (n = 18) completed a paper-and-pencil version. The survey used a four- point Likert scale (4 = strongly agree, 1 = strongly disagree) and open-ended questions to assess students’ overall impressions of the portfolio assignment, how portfolios contributed to the integration of theory intopractice, and their views of the portfolio process.
A graduate student and faculty member who were not involved with students conducted two focus groups with generic students (n = 12) and four focus groups with RN to BSN students (n = 37). The researchers determined mean scores for agreement/disagreement on survey items and used content analysis to identify themes in the focus group responses. In addition, a random sample of nine portfolios submitted by RN to BSN students was analyzed to identify examples of critical thinking that illustrated comprehension of population-based public health nursing. Portfolios for the generic students were returned prior to graduation and were not available for analysis.
Table 2. Focus Group Themes for Traditional Students
The study was approved by the university Institutional Review Board. All students who agreed to participate in the research signed consent forms.
Table 3. Focus Group Themes for RN to BSN Students
Results Table 1 summarizes mean scores on survey items for traditional and RN to BSN students. Most students agreed that portfolios helped explain the role of the PHN and facilitated their use of population-based public health nursing ideas and language. Also, they agreed that clinical experiences provided an adequate number of artifacts or activities to demonstrate the competencies. Students expressed less satisfaction with the portfolio process and indicated that the assignment did not influence them to think about employment as a PHN. RN to BSN students generally disagreed that the portfolio facilitated the learning of new skills.
Responses to open-ended survey questions revealed that the portfolio assignment helped students recognize the number of public health nursing interventions they had accomplished during their clinical experience. Students repeatedly indicated that they had to think critically when recognizing and applying public health nursing interventions. Students also commented that they learned about themselves and the kinds of nursing they would like to practice in the future. Certain concerns were also revealed: 1) the portfolio was not useful if the student was not interested in public health nursing; 2) students felt they had to make up reflections based on what the instructor wanted to hear; 3) it was difficult to understand how to connect the artifact and the competency; and 4) it was difficult to understand how to complete the reflections.
Key themes from the focus group analysis varied for the two groups of students. Four themes emerged for traditional students: 1) confusion, 2) devaluation of the portfolio, 3) critical thinking, and 4) increased understanding of the Public Health Intervention Wheel (Table 2). Students frequently indicated that they were confused about the directions for the project, portfolio components, and expectations for grading. They did not seem to appreciate the reflection process and questioned the worth of the portfolio assignment. However, their comments indicated that they recognized that the portfolio promoted the development of critical thinking skills. Some students commented they had developed a more complete and comprehensive understanding of the Public Health Intervention Wheel.
Eight themes emerged for the RN to BSN students: 1) difficulty finding experiences for the competencies, 2) lack of preceptor understanding of the competencies, 3) confusion, 4) the time- consuming nature of the portfolio, 5) lack of benefit for practicing nurses, 6) expectations for reflective writing, 7) seeing the whole picture, and 8) appreciation for the scope of public health nursing practice (Table 3). Confusion about how to do the portfolio was a theme shared by both groups. RN to BSN students emphasized the intense time commitment involved in completing the portfolio. Some indicated that they had a limited pool of opportunities to choose from, which made it necessary for them to go outside their clinical site to complete the competencies. Students indicated that their preceptors lacked an adequate understanding of the competencies and the Public Health Intervention Wheel.
Table 4.
Examples of Critical Thinking for Population-Based Public Health Nursing Competencies in RN to BSN Student Portfolios (n = 9)
Table 4.
Examples of Critical Thinking for Population-Based Public Health Nursing Competencies in RN to BSN Student Portfolios (n = 9)
Although the RN to BSN students commented that the portfolio assignment was not beneficial for RNs who already had many practice experiences, several indicated that the portfolio helped them see the whole picture of public health nursing and the extent of its influence on the community. Table 4 provides examples from RN to BSN student portfolios of incidents of critical thinking for each public health competency.
Discussion The differences in focus group themes between the traditional and RN to BSN students are likely attributable to the practice experiences and work and family responsibilities generally characteristic of degree-completion (22). Their comments reflected the desire to have their previous practice experiences recognized, which in this study may have contributed to the perception that the portfolio was less beneficial for them.
Data on the lack of understanding of the competencies by preceptors was an important finding. Nursing faculty members for the course realized that RN to BSN students experienced a very different clinical environment – placement with preceptors in schools, clinics, and occupational settings versus placement in official public health agencies with an instructor present. Preceptors for the RN to BSN students generally had little familiarity with the competencies or the Public Health Intervention Wheel.
The response that some students felt they had to “make things up” or write what they thought their instructor wanted to hear is a concern. This response conflicts with the goal of reflective writing, which focuses on the unique experiences and thinking of the writer rather than on an ideal or expected response. Mistakes, negative feelings, and difficulties, when examined, have great potential to promote growth. McMullan and colleagues (16) suggest that the inclination to write what the assessor wants to read may be exacerbated when portfolios are used for summative assessment. Summative assessment of the portfolio may contribute to a dislike of the portfolio, a devaluation of its worth, and a decrease in its efficacy for enhancing learning by stifling honest and examined reflective writing. The challenge is to promote student autonomy in the learning process while using the portfolio as an assessment measure.
Confusion about the portfolio format, connection of the artifact to the competency, and the process of reflective writing can be attributed to two factors. First, the portfolio was piloted for the first time in only one clinical course and was new to both faculty members and students. No examples of previously completed portfolios were available. second, not all community health nursing faculty members had participated in the development of the competencies. The instructors varied in their understanding of the competencies and the assignment.
Finally, data demonstrated that the portfolio assignment succeeds in promoting critical thinking for both traditional and RN to BSN students. Selections from the student portfolios featured in Table 4 show both the variety of experiences and quality of students’ analyses of their actions, interactions, and comprehension of population-based public health nursing. In other words, they “got it.” The portfolio increased student understanding of the role of the PHN and contributed to their accurate application of the Public Health Intervention Wheel to practice. Students gained the conceptual understanding and language needed for population-based nursing practice.
Implications for Portfolio Use in Nursing Education Based on study findings, several changes have been instituted in the implementation of the portfolio assignment. The structure of the assignment has been refined; there is increased emphasis on explaining the value and purpose of the assignment; and clinical agency staff receive an orientation to the assignment and the competencies.
Faculty members have added a page for each competency to the course manual. They encourage students to take notes on the activities and artifacts associated with the competencies to serve as stimuli for in-depth reflective writing.
Timing is seen as crucial to the students’ understanding and acceptance of the assignment. Faculty members introduce the portfolio assignment after the first week of the semester when students are generally less overwhelmed with the many details of course requirements. Portfolio examples are placed on reserve in the library. To encourage consistency and clarity of instructions and grading criteria among faculty, team members are required to participate in a workshop conducted by a portfolio expert. In addition, instructors are intentional about helping RN to BSN students link their clinical experiences with competencies.
Staff at the participating clinical agencies are expected to have knowledge of the competencies, artifacts, instructions, and grading rubric for the portfolio as well as the Public Health Intervention Wheel. Those who have a deficit in these areas are offered opportunities to help increase their knowledge.
Several strategies have been implemented to help students understand the value of the portfolio assignment. The portfolio now accounts for a greater percentage of the course grade to reflect the time investment. Instructors emphasize the use of portfolios for professional development, writing one’s resume, or preparing for a job interview. In addition, a celebration is held to share learning among peers and instructors in a collegial manner, and the nursing department is planning to use portfol\ios to document student learning for the entire curriculum. Instructors attempt to clarify the dual purposes of the portfolio – learning and assessment. They incorporate strategies to encourage and reinforce reflective writing that help students examine their mistakes and negative feelings. An environment of acceptance is enhanced by having students submit their portfolios twice for ungraded feedback before the entire portfolio is due. It is hoped that these strategies will promote student honesty in writing about their learning and growth as public health nurses. (See Figure 2 for the revised portfolio assignment.)
Figure 2. Portfolio: Population-Based Public Health Nursing Competencies
Conclusion The portfolio assignment enhanced students’ critical thinking ability as demonstrated in their application of population- based public health nursing concepts. A comment on the course evaluation form by an RN to BSN student illustrates that the purpose of the portfolio was realized: “The clinical experience and relating the portfolio to the intervention wheel was very helpful for organizing my learning and doing self-evaluation of my learning.” While portfolios have the potential to promote the development of critical thinking, nurse educators also need to consider how using the portfolio for assessment will affect students’ reflections about their learning.
The public health nursing competencies achieved by the students will serve as a strong foundation for graduates who choose public health nursing lor their practice arena. The portfolio approach promotes self-evaluation and reflective practice. Nurses who develop competence through reflecting about their actions will be better able to anticipate and respond to the challenges of an ever- changing health care environment.
Data demonstrated that the portfolio assignment SUCCEEDS in promoting critical thinking for both traditional and RN to BSN students. Selections from the student portfolios show both the variety of experiences and quality of students’ analyses of their actions, interactions, and comprehension of population-based public health nursing. In other words, they “got it.”
References
1. Center for Public Health Nursing, Minnesota Department of Health. (2003). Entry level population-based public health nursing competencies for the new graduate or novice public health nurse. [On- line]. Available: http://www.health.state.mn.us/divs/chs/phn/ henrystreet16.pdf.
2. Center for Public Health Nursing, Minnesota Department of Health. (2003). Linking public health nursing practice and education to promote population heath: Preparing public health nurses for population-based practice. [On-line]. Available: http:// www.health.state.mn.us/divs/chs/phn/partnerships.html.
3. Keller, L. O., Strohschein, S., Lia-Hoagberg, B., & Schaffer, M. A. (2004). Population-based public health interventions: Practice- based and evidence-supported (Part I). Public Health Nursing, 21(5), 469-487.
4. Berkowitz, B. (2002). Preserving our mission. Public Health Nursing, 19(5), 319-320.
5. Gebbie, K. M., & Hwang, I. (2000). Preparing currently employed public health nurses for changes in the health system. American Journal of Public Health, 90(5), 716-721.
6. Lenburg, C. B. (1999). The framework, concepts, and methods of the competency outcomes and performance assessment (COPA) model. Online Journal of Issues in Nursing. Available: http:// www.nursingworld.org/ojin/topic10/tpc10_2.htm
7. Luttrell, M. F., Lenburg, C. B., Scherubel, J. C., Jacob, S. R., & Koch, R. W. (1999). Competency outcomes for learning and performance assessment: Redesigning a BSN curriculum. Nursing and Health Care Perspectives, 20(3), 134-141.
8. Kaiser, K. L., & Rudolph, E. J. (2003). Achieving clarity in evaluation of community/public health nurse generalist competencies through development of a clinical performance tool. Public Health Nursing, 20(3), 216-227.
9. American Nurses Association. (1986). Standards of community health nursing practice. Kansas City, MO: Author.
10. American Nurses Association. (1999). Scope and standards of public health nursing practice. Washington, DC: Author.
11. Hawranik, P. (2000). The development and testing of a community health nursing clinical evaluation tool. Journal of Nursing Education, 39(6), 266-273.
12. Day, M. (1998). Community education: A portfolio approach. Nursing Standard, 13(10), 40-44.
13. Wenzel, L. S., Briggs, K. L., & Puryear, B. L. (1998). Portfolio: Authentic assessment in the age of the curriculum revolution. Journal of Nursing Education, 37(5), 208-212.
14. Jasper, M. (1995). The potential of the professional portfolio for nursing. Journal of Clinical Nursing, 4(4), 249-255.
15. Ramey, S. L., & Hay, M. L. (2003). Using electronic portfolios to measure student achievement and assess curricular integrity. Nurse Educator, 28(1), 31-36.
16. McMullan, M., Endacott, R., Gray, M. A., Jasper, M., Miller, C. M. L., Scholes, J., & Webb, C. (2003). Portfolios and assessment of competence: A review of the literature, journal of Advanced Nursing, 41(3), 283-294.
17. Bell, S. K. (2001). Professional nurse’s portfolio. Nursing Administration Quarterly, 25(2), 69-73.
18. Kelly, J. (1995). The really useful guide to portfolios and profiles. Nursing Standard, 9(32), 3-12.
19. Sorrell, J. M., Brown, H. N., Silva, M. C., & Kohlenberg, E. M. (1997). Use of writing portfolios for interdisciplinary assessment of critical thinking outcomes of nursing students. Nursing Forum, 32(4), 12-25).
20. Tracy, S., Marino, G. J., Richo, K. M., & Daly, E. M. (2000). The clinical achievement portfolio: An outcomes-based assessment project in nursing education. Nurse Educator, 25(5), 241-246.
21. Karlowicz, K. A. (2000). The value of student portfolios to evaluate undergraduate nursing programs. Nurse Educator, 25(2), 82- 87.
22. Sedlak, C. A. (1999). Differences in critical thinking of nontraditional and traditional nursing students. Nurse Educator, 24(6), 38-44.
About the Authors Marjorie A. Schaffer, PhD, RN, is a professor at Bethel University; St. Paul, Minnesota, where Pamela Nelson, MS, RN, is an assistant professor. Emily Litt, MS, RN, PHN, is a planner with the Minnesota Department of Health.
The authors wish to thank the Center for Public Health Nursing at the Minnesota Department of Health for partial funding through their Linking Public Health Nursing Practice and Education to Promote Population Health grant (#1D11HP00330-01) from the Division of Nursing, Bureau of Health Professions, Health Resources and Services Administration. For more information, contact Dr, Schaffer at m- [email protected].
Copyright National League for Nursing, Inc. Mar/Apr 2005
Comments