Multimodal Methods for Researching Digital Technologies 论文

摘要

This chapter provides an introduction to the field of multimodality and discusses its potential application for researching digital data and environments. It begins by outlining what multimodality is, its theoretical origins in social semiotics, and its underlying assumptions. A number of concepts central to multimodality are introduced: these include mode, semiotic resource, materiality, modal affordance, multimodal ensemble and meaning functions. The scope and potential of multimodality for researching digital technologies are then discussed. The chapter sets out an illustrative example of multimodal research. It concludes with a discussion of the limitations and challenges of a multimodal approach for researching digital technologies. What is multimodality? Multimodality is an inter-disciplinary approach drawn from social semiotics that understands communication and representation as more than language and attends systematically to the social interpretation of a range of forms of making meaning. It provide concepts, methods and a framework for the collection and analysis of visual, aural, embodied, and spatial aspects of interaction and environments (Jewitt, 2009; Kress, 2010). While other modes of communication, such as gesture, have been recognized and studied extensively (e.g. McNeill, 1992), multimodality investigates the interaction between communicational means and challenges the prior predominance of