Designing a contextualized PIAAC assessment environment for Indonesian adults: An educational design research study
##plugins.themes.bootstrap3.article.sidebar##
##plugins.themes.bootstrap3.article.main##
Abstract
The Programme for the International Assessment of Adult Competencies (PIAAC) assesses adults’ literacy, numeracy, and problem-solving competencies in functional situations. However, PIAAC-oriented assessment resources that reflect Indonesian adult contexts remain limited. This study aimed to design and evaluate a contextualized PIAAC assessment environment for Indonesian adults. Educational Design Research was employed using Plomp’s preliminary research, prototyping, and assessment phases, integrated with formative evaluation through focus group discussions, one-to-one evaluations, small-group evaluations, and field testing. The study produced 93 contextualized items comprising 32 literacy, 31 numeracy, and 30 problem-solving items, which were incorporated into a computer-based testing system and disseminated through an item book and social media content. Iterative evaluation identified and addressed issues related to contextual relevance, competency alignment, instructional clarity, response-format flexibility, and technical functionality. A field test involving 30 teachers showed mean achievement percentages of 84.33% in literacy, 80.33% in problem solving, and 74.33% in numeracy. The findings indicate that international adult competency constructs can be operationalized within authentic Indonesian contexts and iteratively transformed into a practical, usable digital assessment environment. The study offers design-based guidance for developing culturally relevant adult competency assessments while maintaining alignment with the PIAAC framework.
INTRODUCTION
Rapid technological, economic, and social transformations have fundamentally changed the competencies adults need to participate effectively in contemporary life(Díaz-García et al., 2022). The expansion of digital technologies has increased the volume, speed, and diversity of information that adults encounter, while digitalization has changed how information is accessed, communicated, and used in both work and everyday contexts. Adults are therefore increasingly expected to interpret multimodal information, evaluate quantitative evidence, navigate digital systems, and make decisions in situations characterized by changing information and multiple constraints(Yamashita et al., 2019). These changes are also reshaping the workplace, where mamy tasks increasingly require workers to process information, communicate effectively, and adapt changing condition, and solve non-routine problems alongside occupation-specific knowledge(Carnevale & Smith, 2013). Consequently, literacy, numeracy, and problem solving are becoming increasingly important not only for employment but also for adults’ capacity to make informed decisions in everyday context, including accessing and using information related to health and other aspects of daily life(Yamashita et al., 2019). Moreover, persistent low levels of these foundational competencies may contribute to cumulative social and employment disadvantages, particularly among young adults who experience prolonged disconnection from education and employment(van Vugt et al., 2024). These developments underscore the need for assessment approaches that capture how adults actually mobilize competencies in functional situations rather than merely measuring the retention of school-based knowledge.
The Programme for the International Assessment of Adult Competencies (PIAAC) was developed in response to this changing conception of adult competence and the growing need to assess how adults apply skills in real-life situations. PIAAC represents a large-scale international effort to examine how adults use information-processing skills across personal, social, and work-related situations(OECD, 2013; Schleicher, 2008). Its assessment framework focuses on literacy, numeracy, and problem solving in technology-rich environments, with subsequent developments increasingly emphasizing adaptive problem solving in dynamic and changing contexts(Kirsch & Lennon, 2017; OECD, 2024). This orientation is particularly relevant to contemporary adult competence because PIAAC does not treat these skills simply as bodies of knowledge to be recalled. The design of PIAAC is distinctive because assessment tasks are constructed around the interaction between cognitive processes, task characteristics, contexts, and information representations rather than around conventional disciplinary content alone(Kirsch et al., 2020). Research examining PIAAC through cognitive taxonomies has similarly demonstrated that its tasks encompass different levels and forms of cognitive engagement, highlighting the importance of understanding what respondents are required to do with information rather than simply whether an answer is correct(Boeren & Íñiguez-Berrozpe, 2022). Accordingly, PIAAC provides an important conceptual and methodological basis for assessing adult competencies as functional capacities that are mobilized in response to contextual demands, rather than as the mere retention of school-based knowledge.
A central implication of this approach is the importance of authenticity and context in assessment. Authentic assessment requires meaningful correspondence between assessment activities and the ways knowledge and skills are applied beyond formal testing situations. Critical elements include task realism, cognitive challenge, the use of knowledge in context, and opportunities to demonstrate performance through meaningful actions(Ashford-Rowe et al., 2014). This orientation is highly compatible with adult competency assessment because adults encounter literacy, numeracy, and problem-solving demands through activities such as interpreting public information, comparing prices, managing schedules, evaluating financial options, reading workplace documents, and navigating digital services. Studies using PIAAC data have shown that adult competencies are closely intertwined with educational, occupational, and social experiences(Kim, 2018; Lopes et al., 2020). From this perspective, the authenticity of an assessment context is not merely a decorative feature of an item; rather, it shapes the nature and complexity of the cognitive processes through which competence is demonstrated(Abbasi, 2025; Makeham & Lee, 2012; Villarroel et al., 2018). In authentic tasks, respondents must determine which information is relevant, select an appropriate strategy, and apply competencies within the constraints of a particular situation(Makeham & Lee, 2012; Oroh et al., 2025; Villarroel et al., 2018).
However, the use of international adult competency frameworks raises an important contextual challenge. Literacy practices are socially and culturally situated, and the meaning of competent performance cannot be fully separated from the contexts in which adults encounter and use information.Perry et al. (2020)have critically argued that large-scale literacy assessments may insufficiently represent the diversity and situated nature of everyday literacy practices. Differences in educational pathways and opportunities can also produce substantially different experiences of skill development among adult populations(Cosser, 2018), while cross-national and intergenerational variations demonstrate that literacy gaps are embedded within broader social structures(Kim, 2018). These concerns are particularly relevant to Indonesia, where improving adult competencies and aligning skills with changing labour-market demands remain significant policy priorities(OECD, 2021). Although PIAAC offers a robust international framework, assessment tasks developed for international comparison cannot simply be treated as ready-made learning or assessment resources for Indonesian adults. There is a need to translate the underlying competency constructs into authentic tasks that preserve the cognitive characteristics of PIAAC while drawing on information, services, occupations, and situations recognizable within Indonesian adult life.
The challenge becomes more complex when problem solving is situated in technology-rich environments. Digital problem solving involves more than technical proficiency; performance depends on how individuals search for information, sequence actions, evaluate alternatives, and modify strategies as a task unfolds.Vörös and Rouet (2016)demonstrated that strategy use
##plugins.themes.bootstrap3.article.details##
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
The author is responsible for acquiring the permission(s) to reproduce any copyrighted figures, tables, data, or text that are being used in the submitted paper. Authors should note that text quotations of more than 250 words from a published or copyrighted work will require grant of permission from the original publisher to reprint. The written permission letter(s) must be submitted together with the manuscript.References
Abbasi, H. (2025). Educator perspectives on the transformative power of authentic assessments on student engagement, empowerment, and inclusivity. Educational Practice and Theory, 47(1), 41–54. https://doi.org/10.7459/ept/470104
Ashford-Rowe, K., Herrington, J., & Brown, C. (2014). Establishing the critical elements that determine authentic assessment. Assessment & Evaluation in Higher Education, 39(2), 205–222. https://doi.org/10.1080/02602938.2013.819566
Boeren, E., & Íñiguez-Berrozpe, T. (2022). Unpacking PIAAC’s cognitive skills measurements through engagement with Bloom’s taxonomy. Studies in Educational Evaluation, 73, 101151. https://doi.org/10.1016/j.stueduc.2022.101151
Carnevale, A. P., & Smith, N. (2013). Workplace basics: The skills employees need and employers want. Human Resource Development International, 16(5), 491–501. https://doi.org/10.1080/13678868.2013.821267
Chen, Y., & Nunes, B. P. (2025). Quantifying assessment difficulty in GenAI-enhanced education: Integrating reinterpreted RBT and SOLO taxonomy. In 2025 IEEE International Conference on Advanced Learning Technologies (ICALT), (pp. 382–386). https://doi.org/10.1109/icalt64023.2025.00118
Cosser, D. M. (2018). Differential pathways of South African students through higher education. International Journal of Educational Development, 59, 100–109. https://doi.org/10.1016/j.ijedudev.2017.10.003
De Wever, B., Hämäläinen, R., Nissinen, K., Mannonen, J., & Van Nieuwenhove, L. (2023). Teachers’ problem-solving skills in technology-rich environments: A call for workplace learning and opportunities to develop professionally. Studies in Continuing Education, 45(1), 86–112. https://doi.org/10.1080/0158037x.2021.2003769
Díaz-García, V., Montero-Navarro, A., Rodríguez-Sánchez, J.-L., & Gallego-Losada, R. (2022). Digitalization and digital transformation in higher education: A bibliometric analysis. Frontiers in psychology, 13, 1081595. https://doi.org/10.3389/fpsyg.2022.1081595
Elwy, A. R., Wasan, A. D., Gillman, A. G., Johnston, K. L., Dodds, N., McFarland, C., & Greco, C. M. (2020). Using formative evaluation methods to improve clinical implementation efforts: Description and an example. Psychiatry Research, 283, 112532. https://doi.org/10.1016/j.psychres.2019.112532
García-Peñalvo, F. J. (2022). Desarrollo de estados de la cuestión robustos: Revisiones Sistemáticas de Literatura [Developing robust state-of-the-art reviews: Systematic literature reviews]. Education in the Knowledge Society (EKS), 23, e28600. https://doi.org/10.14201/eks.28600
Goldhammer, F., Hahnel, C., & Kroehne, U. (2020). Analysing log file data from PIAAC. In D. B. Maehler & B. Rammstedt (Eds.), Large-scale cognitive assessment: Analyzing PIAAC data (pp. 239–269). Springer. https://doi.org/10.1007/978-3-030-47515-4_10
González-Such, J., Sancho-Álvarez, C., & Sánchez-Delgado, P. (2016). Cuestionarios de contexto PISA: Un estudio sobre los indicadores de evaluación [PISA context questionnaires: A study on assessment indicators]. RELIEVE - Revista Electrónica de Investigación y Evaluación Educativa, 22(1), 1–26. https://doi.org/10.7203/relieve.22.1.8274
Gorges, J., Koch, T., Maehler, D. B., & Offerhaus, J. (2017). Same but different? Measurement invariance of the PIAAC motivation-to-learn scale across key socio-demographic groups. Large-scale Assessments in Education, 5(1), 13. https://doi.org/10.1186/s40536-017-0047-5
He, Q., & von Davier, M. (2015). Identifying feature sequences from process data in problem-solving items with N-grams. In L. van der Ark, D. Bolt, W. C. Wang, J. Douglas, & S. M. Chow (Eds.), Quantitative Psychology Research (Vol. 140, pp. 173–190). Springer. https://doi.org/10.1007/978-3-319-19977-1_13
Jakubowski, M., & Pokropek, A. (2019). Piaactools: A program for data analysis with PIAAC data. The Stata Journal: Promoting communications on statistics and Stata, 19(1), 112–128. https://doi.org/10.1177/1536867x19830909
Kim, S. (2018). Literacy skills gaps: A cross-level analysis on international and intergenerational variations. International Review of Education, 64(1), 85–110. https://doi.org/10.1007/s11159-018-9703-4
Kirsch, I., & Lennon, M. L. (2017). PIAAC: A new design for a new era. Large-scale Assessments in Education, 5(1), 11. https://doi.org/10.1186/s40536-017-0046-6
Kirsch, I., Yamamoto, K., & Khorramdel, L. (2020). Design and key features of the PIAAC survey of adults. In D. B. Maehler & B. Rammstedt (Eds.), Large-scale cognitive assessment (pp. 7–26). Springer. https://doi.org/10.1007/978-3-030-47515-4_2
Leighton, J. P., Heffernan, C., Cor, M. K., Gokiert, R. J., & Cui, Y. (2011). An experimental test of student verbal reports and teacher evaluations as a source of validity evidence for test development. Applied Measurement in Education, 24(4), 324–348. https://doi.org/10.1080/08957347.2011.607061
Liao, D., He, Q., & Jiao, H. (2019). Mapping background variables with sequential patterns in problem-solving environments: An investigation of United States adults’ employment status in PIAAC. Frontiers in psychology, 10, 00646. https://doi.org/10.3389/fpsyg.2019.00646
Lopes, T., Scully-Russ, E., Zarestky, J., & Collins, J. C. (2020). The effects of social characteristics of jobs on the cognitive skills of adults in the United States: A PIAAC analysis. Adult education quarterly, 70(2), 140–174. https://doi.org/10.1177/0741713619884567
Makeham, S., & Lee, C. (2012). Making the aural presentation of examination papers student friendly: An alternative to a reader in examinations. Assessment & Evaluation in Higher Education, 37(2), 237–243. https://doi.org/10.1080/02602938.2010.527915
McKenney, S., & Reeves, T. C. (2025). Educational design research for relevant & robust scholarship. Journal of Computing in Higher Education, 37(2), 614–638. https://doi.org/10.1007/s12528-025-09456-2
Mokkink, L. B., Herbelet, S., Tuinman, P. R., & Terwee, C. B. (2025). Content validity: Judging the relevance, comprehensiveness, and comprehensibility of an outcome measurement instrument – a COSMIN perspective. Journal of clinical epidemiology, 185, 111879. https://doi.org/10.1016/j.jclinepi.2025.111879
OECD. (2013). OECD skills outlook 2013: First results from the survey of adult skills. OECD Publishing. https://doi.org/10.1787/9789264204256-en
OECD. (2016). Skills matter: Further results from the survey of adult skills. OECD Publishing. https://doi.org/10.1787/9789264258051-en
OECD. (2021). Investing in competences and skills and reforming the labour market to create better jobs in Indonesia. OECD Economics Department Working Papers, No. 1670, OECD Publishing. https://doi.org/10.1787/fd54e6be-en
OECD. (2024). Do adults have the skills they need to thrive in a changing world?: Survey of adult skills 2023. OECD Publishing. https://doi.org/10.1787/b263dc5d-en
Oroh, E. Z., Ali, M. I., Pelenkahu, N., Usman, H., & Rorintulus, O. A. (2025). Authentic assessment in higher education to increase critical thinking and develop metacognitive awareness. Studies in English Language and Education, 12(2), 827–844.
Perry, K. H., Shaw, D. M., & Saberimoghaddam, S. (2020). Literacy practices and the programme for the international assessment of adult competencies (PIAAC): A conceptual critique. International Review of Education, 66(1), 9–28. https://doi.org/10.1007/s11159-019-09819-9
Plomp, T. (2013). Educational design research: An introduction. In T. Plomp & N. Nieveen (Eds.), Educational design research (pp. 10–51). Netherlands Institute for Curriculum Development (SLO).
Rammstedt, B., Perry, A., & Maehler, D. B. (2015). Zentrale ergebnisse von PIAAC aus deutscher perspektive [Key findings from PIAAC from a German perspective]. Zeitschrift für pädagogik, 61(2), 162–191.
Ruiz-Primo, M. A., & Li, M. (2016). Ítems contextualizados de PISA Ciencias: La conexión entre las demandas cognitivas y las características de contexto de los ítems [Contextualized PISA science items: The connection between cognitive demands and item context characteristics]. RELIEVE - Revista Electrónica de Investigación y Evaluación Educativa, 22(1), 13. https://doi.org/10.7203/relieve.22.1.8280
Sagita, L., Utami, N. W., Dwipa, N. M. S., Zulkardi, Z., Putri, R. I. I., & Fauzan, A. (2026). Designing a contextual numeracy assessment for adult learners: Evidence from readability testing based on the PIAAC framework. Mathematics Education Journal, 20(3), 671–692. https://doi.org/10.22342/mej.v20i3.pp671-692
Schleicher, A. (2008). Piaac: A new strategy for assessing adult competencies. International Review of Education, 54(5-6), 627–650. https://doi.org/10.1007/s11159-008-9105-0
Turk, N. (2021). Metodologija priprave sistematičnih preglednih člankov [Methodology for preparing systematic review articles]. Slovenian Medical Journal, 90(7-8), 432–442. https://doi.org/10.6016/ZdravVestn.3138
van Vugt, L., Levels, M., & van der Velden, R. (2024). The low skills trap: The failure of education and social policies in preventing low-literate young people from being long-term NEET. Journal of Youth Studies, 27(2), 217–251. https://doi.org/10.1080/13676261.2022.2118036
Venancio, S. I., Bortoli, M. C., Frias, P. G., Giugliani, E. R. J., Alves, C. R. L., & Santos, M. O. (2020). Development and validation of an instrument for monitoring child development indicators. Jornal de Pediatria, 96(6), 778–789. https://doi.org/10.1016/j.jped.2019.10.008
Villarroel, V., Bloxham, S., Bruna, D., Bruna, C., & Herrera-Seda, C. (2018). Authentic assessment: Creating a blueprint for course design. Assessment & Evaluation in Higher Education, 43(5), 840–854. https://doi.org/10.1080/02602938.2017.1412396
Vörös, Z., & Rouet, J.-F. (2016). Laypersons’ digital problem solving: Relationships between strategy and performance in a large-scale international survey. Computers in human Behavior, 64, 108–116. https://doi.org/10.1016/j.chb.2016.06.018
Yamashita, T., Bardo, A. R., Cummins, P. A., Millar, R. J., Sahoo, S., & Liu, D. (2019). The roles of education, literacy, and numeracy in need for health information during the second half of adulthood: A moderated mediation analysis. Journal of Health Communication, 24(3), 271–283. https://doi.org/10.1080/10810730.2019.1601303
Zulkardi, Z., Putri, R. I. I., Fauzan, A., Sagita, L., Alfajri, R., Silvhiany, S., Adelia, V., Anggeraini, Y., Putri, D. A., Utami, N. W., Dwipa, N. M. S., Utari, R. S., Sukma, Y., Azka, D. A., Ismail, R. N., Sari, N., Asra, A., & Fitriani, N. (2026). Assessment frameworks utilized by PIAAC for evaluating adult problem-solving skills. Mathematics Education Journal, 20(2), 423–448. https://doi.org/10.22342/mej.v20i2.pp423-448