The integration of the verbal and visual components of multiform arguments (MFAs) into a cohesive utterance is possible by juxtaposing and entangling the semantic and visual aspects of words and images. This is feasible since printed words and printed images have the capacity to be both meaningful and visible at the same time. Further, their visibility and meaningfulness can very well exist while the other medium is also visible and meaningful. The mechanisms that coordinate and integrate the visibility and meaningfulness of words and images within an MFA are “multiform references” (MFRs).
Noa Yaari, Communication, 2018.
Since MFRs are a kind of reference, they have a potential to influence the readers to shift their attention from words to images and vice versa; however, there is nothing intrinsic in MFRs that guarantees a change in readers’ behavior. Thus, when we discuss MFRs, we relate to a rhetorical device that has a potential to change readers’ mode of reading and observing the text. The actual power of MFRs to change readers’ behavior will be clearer after we track readers’ eye movements while they consume MFAs. Tracking readers’ eye movements will help us reveal the varied patterns involved in reading illustrated texts, among them the elements that drive readers to shift their attention between the verbal and visual components; the velocity of readers’ reaction to visual and semantic stimuli (that is manifested in rapid eye movements, i.e., saccades, that are measured in milliseconds); and the duration in which their gaze rests upon the different components. Finally, tracking readers’ eye movements will also allow us to understand better the “compelling” element that thinkers have ascribed to word and image relation.
Noa Yaari, Business, 2018.
Indeed, Saussure, Wittgenstein and Bal have pointed out the power embedded in word and image relation to forcefully evoke thoughts in one’s mind. Charles Sanders Peirce’s classification of signs into three modes: symbolic, iconic and indexical, may be considered as a framework to think about the compelling or involuntary effect word and image relation has upon readers. According to Peirce, indexical signs are connected with the things or objects they represent by a physical, organic connection (2.229). Therefore, the direct and contiguous connection between indexical signs and the things they represent enables indices to “direct the attention to their objects by blind compulsion” (2.306). Peirce also argues that indices can be thought of “as a fragment torn away from the object, the two in their existence being one whole or a part of such whole“ (2.230). If words and images have pushing and pulling forces between them, that can shift individuals’ attention from the words to the images and vice versa, across the MFA’s space, it implies that words and images may be physical parts of a whole. This conceptual framework suits well my understanding of MFAs and especially MFRs, that might be a whole whose verbal and visual poles are “torn” from its body.
Noa Yaari - The Multiform Grammar Lab 