Introduction

Writing in digital environments intensifies the inclusion of multimodal representations, many of which are drawn from the youth’s spare-time activities. Focusing on multimodal composition with iPads in a Norwegian school, this article seeks to elicit information about students’ multimodal design choices in text composition and queries the relations between in-school and out-of-school designs. The research questions are directed toward the characteristics of the multimodal representations in four students’ planet books and towards the students’ talks around their texts in order to find out how their participations in various learning sites influence their planet book designs and writer roles. Here, Kress’ (2010) concept of the motivated sign is relevant. In brief, this concept places agency on the sign maker and draws attention to the interests and intentions that motivate the students’ choices of semiotic resources.

Creating multimodal texts is a common practice in contemporary classrooms, and the curriculum states that students should be able to create a range of increasingly complex written, spoken, and multimodal texts for different purposes and audiences, with accuracy, fluency, and purpose (Kunnskapsdepartementet [KD], 2020). In science, graphics, tables, symbols, and figures are essential semiotic resources in the construction and representation of curricular knowledge.

Skilled multimodal composition also requires students’ knowledge of the technology and the processes necessary to produce digital texts, often referred to as digital literacy. Digital literacy can cross online–offline and material–immaterial boundaries and create complex communication trajectories across time and space (Marsh, 2019). In other words, learning spaces and practices are no longer bound by the four walls of the classroom and are shaped by movements across boundaries.

Previous research

The wealth of research conducted over the last decades has informed academic practitioners’ understanding of the continuities and the discontinuities between home and school literacy practices (Bronkhorst & Akkerman, 2016; Jewitt, 2008; Marsh, 2010). While some studies have identified home practices that do not always transfer to classroom practice (Loerts & Heydon, 2017; Marsh, 2005), Bronkhorst and Akkerman’s (2016) reviewed studies show that schools are increasingly developing an outward orientation, seeking to connect to students’ out-of-school participations. Akkerman and Bakker (2011, p. 133) describe boundary crossing as the process of negotiating and combining ingredients from different contexts and define boundary objects as a means of translation between these contexts, leading to learning.

Norwegian studies examining the in-school/out-of-school binary have shown valuable findings. Investigating children’s literacy in a leisure-time context, Michelsen (2015) examines the potential of bridging the gap between the school’s text culture and the way that children read and write in their spare time. Focusing on the children’s use of multimodal texts online, Michelsen (2015) shows how children’s literacy serves the purposes of acquiring expert knowledge of a personal subject and displaying identity, findings relevant for my own research. Also foregrounding leisure-time literacies, Erstad and Silseth (2019) examine how digital technologies and media practices in which youths participate outside school become resources for educational prospects and document instances of personal interests intertwining with more formal pathways of learning.

Like my study, Juuhl’s PhD-study (2014) is situated in school, and shows examples of bridges, as well as gaps, between in-school and out-of-school literacy practices. Examining texts produced in both settings, Juuhl identifies gaps due to “a schooled usage of digital media” (2014, p. 422, my translation). This finding aligns with that of Australian research on the relationship between tablets and literacy, where the use of tablets in school is reported to be “less cognitively challenging and more disciplined than the more exploratory usage seen outside of school” (Simpson & Walsh, 2017, p. 258). According to Simpson and Walsh (2017), research into the modal complexity offered by tablets that considers the context in which learning takes place is still scarce. Krumsvik, Berrum, and Jones (2018) also identify the lack of research-based knowledge about the learning outcome of educational technology using tablets, both internationally and in Norway.

Theoretical perspectives

New Literacy Studies (NLS) have informed numerous studies that explore issues of identity and agency in relation to digital and multimodal practices in homes and schools (Gee, 2005; Jewitt, 2005, Kress, 2010; Lankshear & Knobel, 2007; Marsh, 2019).

Kress, Jewitt, Ogborn, and Tsatsarelis (2001) emphasize that students’ interests and intentions unfold in tension with the norms and expectations in the academic text culture and institutional settings. Kress and colleagues view students’ semiotic work as expressions of their different stances toward the genres of schooled knowledge and conventional literacy practices, their response to “the protocol of being scientific”, and their visual and linguistic realization of this (p. 132).

Lankshear and Knobel (2007) describe new literacies as more participatory, collaborative, and distributed in nature than conventional literacies, as well as less expert dominated. New literacy practices draw on elements of prior cultural production (e.g. through remixing, gaming, blogging, creating fan fiction). The interest in students’ media culture opens broader notions of learning beyond the classroom walls (Apperley & Walsh, 2012; Consalvo, 2007; Erstad & Silseth, 2019; Gee & Hayes, 2012). Consalvo (2007) introduced the concept of paratexts to the study of digital game culture. Digital game paratexts comprise the system of media products that emerge on and about digital games and include a wide variety of print and digital texts. Some are produced by the game industry, others are created by gameplayers themselves (fan fiction, chats, walkthroughs, and YouTube-videos).

The ways that gameplayers co-design and engage in games often resemble more traditional school-based texts and practices and can potentially enhance literacy teaching and learning (Gee & Hayes, 2012; Walsh, 2010). Based on studies of fan sites associated with popular computer games, Gee and Hayes have identified features of what they call nurturing affinity spaces that are supportive of learning. Affinity spaces honor craft knowledge and facilitate identity play and reflections on audience and process. Within these interest-based online communities, people can make, comment on, and share cultural artifacts such as images, videos and texts, and through these, adapt and add to the multimodal designs of these artifacts. Gee (2005) stresses that these spaces are semiotic, which refers to the ways that people interact in the space to create meaning that might be unique to, or descriptive of, the space in question.

Computer-mediated communication (CMC) gives rise to linguistic and semiotic changes and variants related to specific media. Androutsopolous (2011) group these media-specific usages into three themes: conceptual orality, compensation, and linguistic economy. According to Androutsopolous (2011, p. 5), orality includes “all aspects reminiscent of casual spoken language in written discourse.” The semiotics of compensation involve attempts to compensate for the absence of facial expressions or intonation patterns by the standardized means of keyboard and typeface. These include the new generation of emoticons, called emojis, and various imitations of expressiveness (e.g., by repetition of letters and punctuation marks). The third theme, linguistic economy, comprises any strategy of shortening the message form due to the necessity of speed or size limitations of the given medium. Researchers (Androutsopolous, 2011; Hougaard, 2014) consider media-specific language usage as a resource connected to the user’s digital literacy.

Research design, methods, and data

Against this backdrop, my study takes on an ethnographic approach to literacy and focuses on the multimodal representations of students across different sites of learning. I spent 18 months with fifty 11–12-year-old students and their teacher, limited to the writing classes (usually twice per week). I used the modern ethnographic methods: becoming familiar with the context in which the data are collected, conducting participant observations and interviews, and taking field notes.

The entire ethnographic fieldwork encompasses six writing projects in total. In this article, I present a study of the first one, the planet book-project in grade six, which covers nine double-lessons and lasted 2 months. This project has been selected because it serves the purpose of presenting the themes and the topics that have also emerged from the study of the additional writing projects: an innovative use of semiotic resources, crossovers between academic and vernacular literacy practices, the iPad as a boundary-crossing object, and participation in affinity spaces (Strømman, forthcoming). My time spent in the classroom enabled me to look for recurrence and establish patterns within and across the different writing projects. Furthermore, the selected writing project provides enough detail of the contexts of the larger study. In the words of Purcell-Gates (2011, p. 152), it is “a mini-study still linking to the larger ethnography of which it is an essential piece.”

The classroom

The classroom can be described as a tablet-mediated environment, and my study’s participants are well beyond the ‘early adopter’-phase. The school implemented a 1:1 iPad program for Grades 5–7 in 2012/2013. Currently, all students have access to iPads. Grades 5–7-students always have their personal iPad available, even at home, using it for homework, gaming, communicating in social media, and so on, which is why this school and this class were selected for my study. The class also uses Minecraft as a game-based learning platform for curricular activities in most subjects and has its own school blog. Equally important, the iPads are fully integrated into literacy instruction. The concept mapping app iThoughts and the composition app BookCreator (BC) were used in the planet book-project, with PhotoGene (PG) as an editing tool. The core subject application SolarWalk. Explore the Universe and Planets (SW) was also provided. It presents information about astronomy and space as a three-dimensional (3D) solar system that could be rotated and zoomed, resembling a planetarium. SW also allows screen shots of images.

Participants

Participant observation is a key ethnographic method. Four focal students were selected – Jon, Daniela, Martin, and Nina. This focus group was selected together with the teacher, who stated that they were proficient in multimodal composition, the phenomenon of interest in my study. As the class had been creating multimodal texts on their iPads for more than 2,5 years, this was a trait they shared with most of their classmates. Furthermore, most of the sixth graders in this class took part in various affinity spaces, some of them digital. The main reason for selecting Jon, Nina, Martin and Daniela, was that these four students demonstrated both a willingness to participate, and to communicate their experiences to a researcher, which are not traits shared with all twelve-year-olds. My study also deliberately includes students who reported that they enjoyed creating multimodal texts, although not necessarily “the writing part” (Martin) or “the rigid writing tasks” (Daniela). Like most of their classmates, the four students were rather skilled producers of online content, although they had different investments in technology. Nina wrote for the school blog, Daniela produced music videos, Jon and Martin identified themselves as gamers, and all took part in various affinity groups related to these activities. Purcell-Gates (2011) refers to this type of participant selection as a combination of purposive and representational sampling.

Talks around texts

To elicit insider information and to explore the literacy events in detail, individual and focus group interviews were held. “Talk around texts” is a form of semi-structured interviewing that uses texts or artifacts as a starting point and a focus for the interview (Page, Barton, Unger, & Zappavigna, 2014). In these interview settings, the four students had their planet texts (e.g., mind maps and planet books) in front of them and were asked to reflect on their writing decisions. The questions asked were designed to help identify transferences of literacy practices between home and school (cf. boundary crossing) and to further examine how different design principles re-appear and re-combine across different semiotics. Because literacy events are frequently constituted through dialogue, often relating to texts or images, this method provides clues about the participants’ perceptions of the context of writing in school. The perspectives and research questions in this article foreground the students’ texts and talks. The teacher’s viewpoints are gathered as observational data regarding the introduction of the project and do not encompass his assessment of the finished texts.

Researcher’s role

Classroom ethnography positions me as a teacher educator/researcher immersed in the practices and cultures, taking an emic perspective, but this does not obviate the need “to view each classroom as a new and unfamiliar community” (Purcell-Gates, 2011, p. 142). Initially, my research position was close to ‘the observer’. While I was taking an increasing interest in the students’ activities and perceptions, my researcher role developed. For the final part of the research process, I asked the four focal students “to construe the researcher” (Barton & Papen, 2010, p. 111), and one of the interview answers (Daniela’s) identified me as “one of those teachers who knows a lot about writing and makes us super-heroes.” The student’s answer highlights the researcher’s dual role in the classroom; by studying literacy events, influencing these events is inevitable.

Research questions

As ethnographic approaches to literacy seek to describe and provide insights into activities in their contexts, my research questions are directed toward the characteristics of the planet book-project, the students’ planet books, and the situated talks around texts. According to NLS researchers, this set of elements represents a chain of literacy events linked together (Barton & Papen, 2010). The analysis queries how curricular knowledge is represented and communicated by the sixth graders, aims to identify home–school literacy transitions, and is guided by the following research questions:

1. What characterizes the multimodal representations in the students’ planet books?
2. How does the students’ participations in various learning sites influence their planet book-designs and writer roles?

The data

The analysis presented here is based on (1) field notes from nine lessons (observations of the teacher’s introduction of the planet book-project, including hand-outs, and of the students’ text production), (2) four planet books, and (3) the students’ talks around their texts. The three-level description of the data sources (Table 1) shows the different types of data collected in relation to the project, what they are used for, and how they are analyzed.

Table 1. Data from the planet book project.

Social semiotic multimodal analysis (SSMA)

As ethnographers of writing want to know where, when, and for what purposes writing occurs, examining the planet book-project was the first step of the analysis. To help frame the environment of text composition, the planet book project and writing task is presented in the analysis section.

The second step of the analysis was directed toward the students’ selections and arrangements of semiotic resources in the planet books and toward the resources available and required (e.g., in the writing task). I applied social semiotic multimodal analysis (SSMA; Jewitt, 2016) of the texts, which involved categorizing the selection of modes by applying semiotic grammar. Social semiotic multimodal theory (SSMT) provides a descriptive framework that has been used as a tool for this deductive, or theory-guided, analysis of the multimodal representations. Several researchers have analyzed social semiotic processes and dimensions of representations (Jewitt, 2017; Kress & van Leeuwen, 2006) and I relate key analytical points to the central SSMA concepts mode, modal affordances, aptness, salience, and multimodal cohesion. Mode is a socially and culturally given resource for meaning making. The differing possibilities and limitations (affordances) of modes enable specific semiotic work to draw on these affordances. An important point here, is that modal affordances are constantly reshaped depending on the need of those who make meanings (what they consider apt), their rhetorical assumptions in the environment of communication, and what is foregrounded in the texts (what they choose to make salient). The analyses focused on the students’ use of visual and linguistic resources to realize distinct meanings and draw attention to their representations of ‘scientificness,’ resulting in five multimodal categories: images, layout, fact boxes, fonts, and linguistic features. SSMA focuses on shared semiotic practices, and the analyses showed that these five categories are realized as modes in the students’ texts, and that the modal affordances are negotiated and reshaped to fit the communicational task and/or the sign-makers’ interests (Kress, 2010).

In the third step, the students’ talks-around-texts were examined to help recognize the transferences of literacy practices between home and school and identify possible out-of-school influences. This phase of the data analysis involved coding and categorizing the transcribed interview data. In this inductive or bottom-up coding (Purcell-Gates, 2011), the coding decisions were based on the students’ own words and phrases. For example, when two of the focal students (Martin and Jon) talked about the built-in fonts in BC, they discussed “the real gaming text,” “letters that grown-ups don’t understand,” and the desire to “create one’s own ‘text type’” [sic]. All these codes relate to the font as a semiotic resource and a paratextual feature (influenced by gaming), a category of interest to the research focus and a central concept in SSMT. Given the students’ uses of fonts as representational means and their shared assumptions about the fonts’ meaning-potentials, I treat font as a mode in the SSMA. In addition to font, the students were asked to comment on their use of other resources (photos, models, figures, colors, etc.) and the websites and photo-sharing platforms they applied. In the analysis section, the students’ comments taken from these talk-around-text sessions are provided as quotations.

Analysis

Planet book project and writing task

The project was assigned in a science class where the students had to write descriptive texts about a chosen planet, due to an agreed upon fact in class: the textbook chapter on the solar system was incomplete because it did not present the different planets properly. The teacher emphasized that the new planet books would make great supplements to the textbook, and he stressed, “Your books will be read by next year’s sixth–graders.” The class spent two weeks learning about the solar system, which encompassed studying library books, making animation films about the moon landing, and exploring the solar system through the iPad app SW. In addition to books, films, and SW, the teacher had prepared a list of relevant websites, ranging from nysgjerrigper.no to nasa.gov. This is a translated version of the writing task: Decide on one of the planets. Prepare for writing by finding facts and images about your planet. Make a structured concept map from the list of planet features.

To help students plan and organize their texts, the teacher gave them a grid (Table 2), showing how they could arrange the text into six paragraphs corresponding to the concept map. The six planet features listed in the grid are looks like, size, moons, temperature, made of and other. The teacher also demonstrated how the students could use fact boxes for the most interesting planet facts by comparing Mars’ ruddy surface of oxidized iron dust to ‘a baked potato.’ Here, the teacher used the verbal metaphor ‘eye candy,’ that is, the fact box as a visual that could grab the reader’s attention.

Table 2. The teacher’s grid of numbered paragraphs and planet features.

The teacher gave direct instructions on three areas of making the text: reader orientation, composition and labeling (i.e., the grid), and the use of fact boxes. The assignment served the purpose of connecting writing to authentic purposes and positioned the students as “planet experts,” writing an informative text directed toward a less informed reader (next year’s sixth graders). Although not highlighted by the teacher in class, the grid offered genre aspects in terms of content and structure and the instructional verbs ‘tell,’ ‘describe,’ ‘compare,’ and ‘summarize.’ Finally, the project functioned as a prompt for multimodal composition (Kress, 2010).

The following section focuses on the images and layouts of the four students’ title pages and discusses possible out-of-school influences.

Title pages

Figure 1 shows the planet books written by Martin, Daniela, Nina and Jon, respectively. When comparing the four title pages, the observer can easily recognize that they are colorful and equally inviting to a reader. When navigating the iPad, the BC-books are all book-like with page-like pages. The layouts are traditional, each mode’s heading and image share the communicative work of telling the reader what the book is about. Regarding functional load, there is a division of labor. The titles differ in both content and design. Three of them (excluding Daniela’s book Uranus) offer additional information. Nina’s book, The Seventh Planet Uranus, states the planet’s position from the sun. Her book has more images illustrating Uranus’ position in the solar system, and this image–text relation establishes multimodal cohesion. Daniela is the only one that does not use one of the built-in BC fonts. Her blue and porous font is imported from Cooltext.com, a website that she also uses when she produces titles for her music videos.

The two girls have chosen different images of Uranus. Daniela’s Pinterest photography shows Uranus tipped on its side and with rings. In her book, she returns to both the tipping and the rings in her description of what the planet looks like. Nina has chosen a free picture of Uranus from a website called Science-for-Kids. Using the app PG, she foregrounds the image against a blue and starry sky. The white circle breaks the illusion of space, disconnecting the planet from the universe. The font “Bangers” is from the fun-font gallery in BC. The font and image combination makes Nina’s title page look a bit Disney-like and would probably attract many of the Grade 6 readers. Marsh (2005) describes how global discourses such Disney’s mediate children’s everyday literacy practices.

In Martin’s title page, salience is realized through the mode of writing, visual prominence, and capitalization and color. The red color is paramount, both in his bold capital title Red Mars and in the image. In his book, he returns to the red color in describing Mars’ iron oxide surface. Martin’s green byline against the black background can be associated with the retro-aesthetics of the 1980s’ computer games. According to Garda (2014), these retro tendencies manifest themselves in people’s fascination with vintage games and fonts and are now often recycled in modern contexts. In the interview, Martin confirmed that his green byline was inspired by gaming fonts (e.g., Minecraft) and that he also tried, although unsuccessfully, to download what he called “the real gaming text” for this school project.

Jon’s interesting title On Jupiter’s Surface corresponds to the image of the space drone that seems to have landed on a rocky surface somewhere. Regarding science, the problem is that Jupiter is a gas giant without a surface. In his book (Figure 2), Jon quite correctly states, “there’s just loads, loads, and loads of gas layers.”

Since Jon had trouble finding a picture of Jupiter’s surface, he photo-edited an image from NASA to make it apt for his title page and title:

I think that the images that I have chosen are really good because I have filtered and cropped them properly. […]. I made them look a bit more real than the images that I screen casted from Solar Walk or downloaded from the Internet.

In the title pages, students use fonts and colors as resources to make the planet features salient. Furthermore, their design choices are visual signs of individuality – whether these are connected to gaming, music videos, personal taste, and a sense of design – and the desire to capture the reader’s interest. Their chosen images and layouts relate to specific elements in their texts, mainly regarding what the planet looks like and why, and this establishes multimodal cohesion.

The students’ use of websites, photo-sharing platforms, and font galleries demonstrates their cross-over skills as digital writers in school and in various affinity spaces, as well as how these practices often intersect. This is also evident throughout the students’ books. In the following section, analytical points related to images, layout, fact boxes, fonts, and linguistic features, and possible influences are further examined through a selection of planet book-excerpts and the students’ comments on the multimodal representations in these excerpts.

Martin’s Mars

The first excerpt (Figure 3) is from Martin’s book about Mars where he explores the possibility: “Is there life on Mars…?” The question leads to the following chain of arguments: “No They are pretty sure that there isn’t. But they know that there were rivers on Mars once, they have spotted unusual things when they have examined photos taken by robots, they have seen strange formations and things resembling life!👽 ” Here, Martin’s choice of the pronoun “they,” instead of a more academically apt noun (scientists), and his use of repetitive punctuation marks give his text a casual spoken tone, referred to by Androutsopolous (2011) as the conceptual orality often found in chats or text messages. The emoji that Martin has inserted to complement his arguments visually underscores this informal, humoristic style. This is an example of how a media-specific sign (the Extraterrestrial Alien emoji) is used to compensate for the writer’s attitude toward his own statements. In the second paragraph, Martin’s comment on the photographs showing traces of life on Mars shows a more critical position: “These could of course be animated images that they/someone has made.” Martin continues: “It is pretty exciting to look at pictures of Mars! Go on the Internet and search for Mars […]!” Neither this personal account about studying images nor the reader request is what to be expected in an academic genre but is commonly used in interest-based online communities.

In the layout, Martin continues his chosen color scheme, green for subtitles and red for encircling the “traces of life” on Mars and for framing a fact box. He uses one of the built-in fonts called Luckiest Guy. It is hard to read due to a random mix of capital and lower-case letters. Furthermore, Martin has added a shadowing effect. In a talk-around-text session, I asked him to comment on his choice of font. Below, Martin assesses the affordance of fonts and comments on both the font’s possibilities (coolness, collective practice among the boys in class) and limitations (lacks the bold effect):

All of us boys used this font when we discovered it because it’s cool! Later, the teacher told us not to use it. I still think it is cool, although it is a bit too thick. Important words can’t be emboldened.

The font is used “because it’s cool” but is also a visual mark of the community of “us boys” employing it. This demonstrates Martin’s awareness of font as a meaning-making resource (e.g., attracting reader attention, resembling gaming fonts) and how he positions himself as a writer. The teacher that Martin refers to instruct his students to avoid fancy fonts, oversized print, or anything that might distract readers.

Martin’s fact box shows a colorful spaceship breaking the frame to bring the reader “Rocket tips from Mars”. Here, he addresses the size of Mars compared to Earth, as well as the distance between them. Both the fact box information and format are in keeping with the assignment and most science textbooks. In contrast, Martin’s planet book also has linguistic features resembling gaming paratexts. Furthermore, Martin’s alien emoji and choice of photos signal a different tradition, namely popular science and science fiction. In this paragraph, Martin also assesses the information value of the photos (“These could of course be animated images that they/someone has made.”). Like Jon, Martin knows that photos can be manipulated, an important step in developing a critical reading of websites.

Nina’s Uranus

Nina produces texts for the school’s blog, and in many ways her 12-page planet book resembles her blogs. Pages 5–7 (Figure 4) show her book as print-heavy. The readability is somewhat reduced by both her choice of white text on black and the tiny font size. To assist the reader, Nina addresses him or her directly throughout the text using the pronoun “you,” a guiding technique that she also employs in her blogs. Another textual feature drawn from blogging is presenting the images with informative captions: “Voyager 2 and Uranus” and “Rocketship leaving Uranus.” The first image is a photo that illustrates the written text, while the second image (a collage of Uranus and a spaceship emoji) expands the text by adding a humoristic connection to the final title “Goodbye to Uranus.”

Like Martin, Nina names her fact boxes (“Star facts”) and employs emojis and repetitive punctuation marks, which give her text a personal tone. In the excerpt (Figure 4), the winking face emoji follows Nina’s conclusion to why no one has ever visited Uranus: “It’s probably because it’s too cold… .”😉 Winky Face is commonly used to imply humor; here, the emoji also corresponds to the thermometer wearing earmuffs.

The combination of text and different images represents a mix of scientific facts and fun facts. Nina’s extensive description of the moons of Uranus (Figure 5) has references to Titania in Greek mythology, Titania in Shakespeare’s A Midsummer Night’s Dream, and Christina Grimmie’s song Titanium. She also includes how to pronounce these names in a phonetically correct manner. In this case, Nina’s linking of the planet moon to pop culture, Shakespeare, and mythology overshadows the scientific discourse. When asked about her representation of the moon, Nina’s answer shows her familiarity with the dynamics of intertextuality and her reader-orientation:

I used the photography of Christina to make the children remember the name of the largest moon. […] Using good pictures makes children want to read.

On one hand, the combination of emojis, popstars, and fun facts aligns with the website (Science-for-kids) that Nina has listed but is also typical of teen communication in social media and of the dynamic layout of blogs. On the other hand, she uses appropriate copyright material, lists sources, and creates a book incorporating concepts, graphics, and still images in relation to information about the planet, demonstrating academic literacy. In other words, she positions herself as both a peer and an expert vis-á-vis her audience.

Daniela’s Uranus

On pages 3 and 4 (Figure 6) of Daniela’s book, she writes about Uranus in a narrative style: “When William discovered Uranus, he observed a small, green planet with lots of large rings”. The underlined words in the caption state that the planet has 11 rings.

Visiting the same website (Wikipedia) as Daniela did, I learned that Uranus’ rings constitute an area of scientific dispute and that NASA states there are 13 rings. Daniela comments on this dispute by simply stating in one of her fact boxes: “Uranus’ 11 rings have a rather unknown explanation” (Figure 7). As one of the world’s (and perhaps classrooms’) most trafficked websites, Wikipedia has become both a space for writers to compose and a space for understanding how writing is composed (Hicks & Perrin, 2014). Readers are presented with quick pieces of information and hyperlinks, a pattern that Daniela seems to have adopted, although her highlighted words are not clickable. Daniela explained that she used bold “to make the sixth-graders want to read and lines to highlight the most important facts.”

Daniela has collected all her facts on one page (Figure 7), so her readers “don’t need to read through the whole book to find them.” Figure 8 shows a translated version of her first fact box, and here, the narrative elements in “when we turn around eighty, we can imagine being only 1 year old […] on Uranus” might assist her nonexpert audience in comprehending planetary time.

In Daniela’s design, esthetics is important:

I noticed that the website images of Uranus were blue and chose blue for the byline and titles in my book. I chose the font because of the star-like brightness of the letters. […] It is important that the images in my book match this and that they are clear.

Daniela’s text is an example of how transforming the spatial arrangement of the text elements and choosing a coherent color scheme and font make the layout prominent. Daniela uses high-resolution images, and her chosen font and type size make her text easily scanned by the eye. Using the blue Cooltext font, Daniela had to import the letter “Ø” from another font for the subtitle “Størrelse” (Size, Figure 6). Her representational practices in creating the layout and her use of narrative elements align with those employed in her music videos. In personifying both the planet explorer (simply called “William”) and the abstract concept of planetary time (Figure 8), Daniela’s text challenges the science textbook genre.

Jon’s Jupiter

In Jon’s book, images and layout are also given a design prominence. To describe the planet’s size scientifically, Jon has made an illustrated mathematical graphic template, “1 Jupiter equals 121 Earths” (Figure 9). This “fact” is rendered in capital letters and three exclamation points in the text, expressive markers not common in science but typical of personal communication on social media (Hougaard, 2014). On page 5 (Figure 10) Jon presents his own theory, a scientific reasoning grounded on the fact that when there is water, there is a possibility for life. This theory is presented with an image of Jupiter superimposed with a question mark. Here, Jon revisits and questions Jupiter as a gas planet, tying the end of his book to the title page that shows the drone “on Jupiter’s surface.” Making models and formulating hypotheses about scientific phenomena are curricular objectives in science (KD, 2020) and examples of Jon’s emerging sense of the scientific protocol.

All of Jon’s images are collages, re-designed in combination with his new infographics (e.g., an arrow, a 3D model of Jupiter and Earth, a mathematical equation). Jon spent a double lesson making these collages, choosing among previously stored images on his iPad photo gallery and using PG. The three-dimensional models afford the potential for more complex spatial arrangements between the elements than textbook images (e.g., layering and depth), connect with the curriculum, and align with the semiotic registers in science. However, they also resemble the game invitations that Jon makes, and the employed techniques are also used in his production of posters and game walkthroughs.

Many of Jon’s gaming videos are about Minecraft. For his fact boxes, he uses his Minecraft avatar Baa. In the fact box below (Figure 11), Jon lets Baa present the facts, and the arrow points to the fact in question: “Did you know that this red spot on Jupiter is a storm that has lasted for 300 years?” Jon uses Baa in several of his texts for various purposes and he has also printed his own Baa T-shirt online. When asked about his use of Baa, Jon explains:

I always use my avatar Baa. In school projects too. It’s like ‘no problem’ and some of the teachers even think that it’s cool. And Baa-tips? Well, if you watch my You Tube channel, you’ll recognize him.

The Minecraft avatar Baa is a semiotic feature residing in both worlds – the school’s Minecraft platform and in Jon’s YouTube channel – an example of how game-based semiotics make their way into school texts. The examples from Jon’s book also show his abilities to select and combine resources from a semiotic register typical of science, in keeping with textbook conventions. Baa stretches this register but still functions as an apt semiotic resource, establishing cohesion and functioning as a signature or as eye candy for the readers.

Table 3 provides an overview of the SSMA categories and sums up the main findings in the four planet texts.

Table 3. An overview of the SSMA categories and findings.

Discussion

The planet books demonstrate the students’ abilities to convey meaning, combining several semiotic resources and drawing on a range of available designs and digital proficiencies that bridge in-school and out-of-school experiences. In the designs, various aspects (typography, color schemes, collages, models, and verbal text) are foregrounded, and the analysis has shown examples of multimodal representations influenced by the crossing of boundaries. Through interpretation and transfer of content into other contexts and formats, the students can make use of different semiotic resources as ‘tools for thinking’ and learn to be critical of different forms of representation (Jewitt, 2008). The analysis indicates that Jon and Martin are in the process of becoming critical readers, especially when “reading” and editing images. That said, the analysis has also shown examples of tensions arising when various linguistic and visual features drawn from out-of-school literacy practices conflict with the registers in science.

The mentioned out-of-school literacy activities are closely connected to the students’ life-worlds and thus integral parts of their identity construction as writers. All the texts show signs of being reader oriented and “academic,” although not with the clarity and focus of most science textbooks. Regarding linguistic features, there are differences in the students’ use of scientific discourse: Martin adopts an informal, humoristic tone, while taking a critical stance toward photo editing. Nina’s text has a personal, blog-like voice, incorporating intertextual resources. Daniela uses narrative elements and underlines important facts, while Jon’s text has both academic-like and expressive features. In presenting scientific knowledge about planets, the students differ in linguistic and visual accuracy and in their emerging sense of ‘scientificness’ and its expression.

Some of the semiotic resources used are drawn from different affinity spaces. These are above all evident in images, fonts, and the layout, although there are instances of media-specific language by using emojis, a casual style, capital of letters, and punctuation (Androutsopoulos, 2011). The students use fonts, emojis, figures (e.g., spaceships, stars), popstars and avatars as resources to visually mark their individuality, a finding supporting Jewitt’s (2005) and Michelsen’s (2015) studies. The examples above illustrate how these resources are aptly and playfully used as a shorthand to express concepts, sentiments, and ideas. Although some of the resources conflict with textbook norms, they mainly function as reader-oriented and thus expand the dialogical space by helping less informed readers grasp the content of the text.

The analysis also shows examples of how the students “borrow” from different website genres (e.g., the blog format [Nina], Wikipedia [Daniela], and game designs [Jon and Martin]). Lankshear and Knobel (2007) discuss the potential of new forms of literacy for learning by including blogging and the use of wikis in schools. Studies have also shown that when students are encouraged to represent curricular knowledge through multimodal designs drawn from game-based literacy practices, their design skills and proficiencies are improved (Erstad & Silseth, 2019; Walsh, 2010). However, other studies problematize the bridging of home and school literacy activities, as well as point out that not all students participate in literacy activities like blogging or gaming (Juuhl, 2014; Michelsen, 2015).

The concept of modal affordance refers to what is possible to express and represent easily and what is considered apt. The students were involved in the task of producing a text appropriate for the planet book-project, which entailed actively remaking the signs available to them as resources motivated by the context of the lessons and by their own interests, which may factor in their sense of the teacher’s interest but is not identical to it (Kress et al., 2001). As mentioned, the teacher instructed the students to avoid fancy fonts and ensure reader friendliness, something disregarded by Martin and Nina. In contrast, Jon’s comments on his use of the avatar Baa for the fact boxes indicate that this specific semiotic practice is actively endorsed by the teacher.

Marsh (2010) identifies situations where the practices flow in the direction of school to home, referred to as “one-way traffic.” Other researchers report a schooled usage of digital technologies (Juuhl, 2014; Simpson & Walsh, 2017). In contrast, my study’s participants transfer representational and digital practices from one domain to another in a more fluid manner. This is partly due to how the tablet-mediated classroom connects with the students’ tablet-mediated spare time, and to how the teacher mostly confirms the students’ semiotically sophisticated toolkits drawn from various learning sites. In my study, this “two-way traffic” results in representational practices and traversal skills that are more freely combinative with an increasingly multifaceted semiotic landscape. It is then a paradox that many of the semiotically complex affinity spaces are limited to out-of-school activities, thus remaining hidden from teachers and practitioners (Apperley & Walsh, 2012).

Categorizing the selection of modes and applying semiotic grammar make visible the means used by students to communicate meaning, providing the meta-tools needed to understand the students’ semiotic work. This is a valuable insight regarding literacy pedagogy as it pays attention to sign-making as a motivated social process. More importantly, the talk-around-texts method gives students opportunities to articulate their views and discuss their multimodal texts by identifying where their design choices stem from and assessing the strategies used to appeal to their readers. This helps develop a meta-language about (multimodal) texts and composition practices, which are vital factors in literacy development. The talks-around-texts sessions were the researcher’s main classroom contribution, as they created literacy events in which the four students happily participated (cf. “super-heroes”).

The selection of only four students allowed me to concentrate on information-rich literacy events where I could monitor their writing processes and carry out frequent talk-around texts sessions. The research questions focus on multimodal design choices and do not aim to discover a diverse cross section of skills and proficiencies among twelve-year-olds. This small-scale study’s results cannot be generalized. Nonetheless, it still provides a window into how students may negotiate representing complex topics and expresses the complexity of entering a new school genre and the relatively new concept of ‘scientificness’ for sixth-grade students. The writing project presented here was selected because it was the first of six and is similar to the additional five in terms of contextual frame and findings. The prolonged period in the field and the use of SSMA helped identify similarities and patterns in the data, which makes them representative for this class in this school at that time.

Conclusion

My conclusion foregrounds the importance of utilizing the porosity of the classroom as a resource. Extending literacy education beyond the school means including and addressing the linguistic, semiotic, and digital assets that the students possess in various affinity spaces. Researchers (Jewitt, 2008; Kress, 2010) have long argued for the need to incorporate multimodal literacy in the curricula. The intersecting multimodal and digital literacy practices that I observe in my study are essential in contemporary communication to enable students to become skilled producers and discerning consumers of multimodal texts. Thus, my study suggests the possibilities for curricular assignments in writing that connect to students’ out-of-school multimodal repertoires. These semiotic registers are less fixed and less constrained by convention, which also underscore the need for explicit teaching of how modes and semiotic resources construct meaning in specific genres. This includes class discussions on how to design genre-appropriate curricular representations.

My study also points to the importance of school projects that connect to students’ technologically mediated literacy life-worlds, as this is significant in helping them acquire more traditional academic literacies. Continuity across contexts is not self-evident. Bronkhorst and Akkerman’s (2016) review shows that teachers increasingly demonstrate abilities as boundary brokers by accepting the use of everyday literacy next to academic literacy. Supporting students in learning across contexts and validating out-of-school participations by including them in assignments and assessments are possible ways to achieve this (Strømman, forthcoming).

My study identifies implications for educators, curriculum developers and researchers concerning the need for broader discussions and classroom understandings about literacy instruction with new digital technologies. Employing qualitative analyses to interrogate multimodal representations, the study provides insights into students’ multimodal learning with iPads and highlights how they draw on interest-based affinity spaces to represent curricular knowledge. There are several similarities between iPads for entertainment use and as educational technology for use in school (Krumsvik et al., 2018), and iPads are key sites for multimodal investigation because they make a wide range of modes available, often in new inter-semiotic relationships (Jewitt, 2016). In my study, iPads’ connecting function spans the school, the home, and various virtual spaces. Other types of technological tools and different educational software might be equally apt for literacy instruction and multimodal text production. This research area must be revisited and re-evaluated in the light of a wider range of empirical cases.