Pulling and Pushing Forces in Multiform References

Explicit, implicit and indeterminate multiform references (MFRs) maintain pulling and pushing forces between their verbal and visual poles. These forces are the mechanisms that potentially move readers to shift their attention between words and images across a multiform argument (MFA) or message. They are the invisible elements that make MFAs cohesive and coherent; the core of multiform grammar (MFG). These pulling and pushing forces, however, exist due to semantic relationships between, on the one hand, words in the main text, and on the other, words or conventional signs in the captions. In an explicit MFR, for example, “(Fig. 1)” in the main text and “(Fig. 1)” at the beginning of the caption pull and push readers’ attention in the same direction, at the same time. The sign in the main text pushes readers’ attention from the main text towards the caption, and ultimately, the image. The sign in the caption simultaneously pulls readers’ attention from the main text towards itself and the image. Thus, the signs’ identical semantic meaning, as well as their identical appearance, enable readers to effortlessly shift their attention between the MFA’s verbal and the visual components. This is how readers navigate throughout the physical and literal space of the hybrid argument, using the same mechanism.

Pulling and pushing forces in explicit MFRs, in Douglas Biow’s On the Importance of Being an Individual in Renaissance Italy. p. 185.

In implicit MFRs, the pulling and pushing forces between the verbal and the visual poles are mostly based on the semantic and the visual similarity between words in the main text and words in the caption, rather than on the convention “(Fig. 1).” It should be noted that when we discuss the verbal and visual poles, as the two ends of an implicit MFR, the visual pole may be specific words in the caption, rather than purely the image. This condition, however, might change as the reading of the MFA continues, and an association between the words in the main text and the image is established. For example, as often happens, titles of portraits are the names of the people who are depicted in them; therefore, when authors discuss the depicted figures in the main text, they inevitably mention the titles of the portraits. In the image below, the titles of the two portraits are written in the captions of the two illustrations, as well as twice in the main text. Thus, “Julius II” and “Clement VII” in the captions, on the one hand, and in the main text, on the other, create poles between which pulling and pushing forces take place. My assumption is that when the two names are mentioned in the main text the second time, the readers find it easier to associate these names directly with the images. However, since in our example one caption is below the other, this association might take longer. At any rate, the semantic and the visual similarity between the names in the main text and the names in the captions, constitute pulling and pushing forces that enable readers to navigate throughout the MFA.

Pulling and pushing forces in implicit MFRs, in Douglas Biow’s On the Importance of Being an Individual in Renaissance Italy. p. 185.

What happens when an MFA uses explicit as well as implicit MFRs? What happens when its space is saturated with varied pulling and pushing forces, between its words and images? A large number of referential conventions, words, as well as images that can serve as verbal or visual poles for MFRs increases the probability that readers will shift their attention between the MFA’s verbal and visual components. These shifts, in turn, strengthen readers’ comprehension and retention of the MFA, since they create associations between the representational level of the MFA and the phenomenon it discusses. I would like to argue that each MFA has an ideal number of MFRs to reach its ultimate comprehension and retention among its readers. Grasping this number is to use MFG effectively and creatively since, at the end of the day, MFRs are rhetorical devices.

Pulling and pushing forces in explicit and implicit MFRs, in Douglas Biow’s On the Importance of Being an Individual in Renaissance Italy. p. 185.
Posted in Contemplations, Dissertation, Teaching

Indeterminate Multiform References

Indeterminate multiform References (MFRs) begon with terms whose semantic relatedness with terms in the caption or the image is vague and challenging to classify; it is neither a clear semantic similarity, meronym-holonym or any other relatedness; however, indeterminate MFRs seem to influence the semantic, spatial and epistemological aspects of the MFA. In some cases, a term or the verbal pole of an indeterminate MFR signifies a phenomenon into which the image or visual pole may become and therefore cannot entirely represent that phenomenon (i.e., that is represented by the verbal pole). In such cases, the two poles of the MFR suggest potentially a shift or transgression from one state to another or a limbo. These liminal situations can be relevant to either the phenomenon that the MFA explores or the MFA itself, that might change perspectives on that phenomenon. In both cases, our attempt to hypothesize MFRs encounters a moment – and point in the space of the MFA – that manifests a “twist” or “gap” in multiform representation.

We may hypothesize an indeterminate MFR when we read, for example, the term “outdoors” in the main text and see an image of a “dog” in vicinity to it. What is the semantic relatedness between “outdoors” and “dog”? It can only be determined by the context in which both are used; the way the author sees the surrounding both signs share. At the same time, we should consider that the author’s outlook over the relation between “outdoors” and “dog” may change as the MFA progresses. Clearly, in some cases, the context plays a central role in clarifying the semantic relatedness between the MFA’s verbal and visual components; this raises our sensitivity to how authors use MFRs as rhetorical devices. It also makes us aware of the possibility that authors use implicit and indeterminate MFRs in varied degrees of consciousness, and that MFAs are not fully under their control.

Posted in Dissertation, Teaching

Implicit Multiform References

Implicit multiform references (MFRs) generate shifts of attention through semantic relatedness between the MFA’s verbal and visual components and their visual features without announcing their operation. For example, the presence of both the word “cat” and an image of a “cat” on the same page establishes a point or moment within the consumption of the illustrated text, in which it would make sense to shift the attention between the two. After all, what could be the motivation to embed both kinds of representations in proximity if not to discuss the same phenomenon?

Noa Yaari, Permanent Residency, 2018

The semantic relatedness between the term “cat” and an image of a “cat” is similarity. In addition to similarity, terms and images within an MFA can relate to each other by being meronyms or holonyms to each other. A meronym is a term that signifies part or a member of whole and therefore can represent it. For example, “tail” and “cat.” Holonym is the opposite of meronym; it signifies the whole in relation to its part or member. In addition to linking verbal and visual components of a specific MFA, implicit MFRs may operate between different MFAs within a single publication (e.g. all the written “cats” in a publication relate to an image of a cat in that publication), as well as between various publications (e.g. any written “cat” in any publication relates to an image of a certain cat in a certain publication). The function of implicit MFRs is always based on the same principles; their direction of readers’ attention depends on the semantic and visual features of both the verbal and visual components, and on the readers’ own inclination to shift their attention.

The main difference between explicit and implicit MFRs is that while we recognize the first through a well-known convention, we can only hypothesize the last. Hypothesizing implicit MFRs demonstrates that they can come in varied degrees of semantic similarity. For example, the word “tiger” and an image of a “cat” on the same page may establish an MFR that is perhaps more implicit (or less explicit) than the one between the word “cat” and an image of a “cat,” within the same field of vision. When a meronym-holonym relationship is considered, the distance between the term “tail” and an image of “cat” (with or without a tail) is arguably shorter than the distance between the terms “pet” or “animal” and an image of a “cat.” I will examine this premise in my next blogpost.  At any rate, the gradation in implicit MFRs suggests that there could be multiple implicit MFRs within an MFA. It also implies that even if there is an explicit MFR (that uses brackets) within an MFA, it is highly probable that there are also implicit MFRs within that MFA (Fig. 1).

Fig. 1 Noa Yaari, I Love Permanent Residency, 2018
Posted in Contemplations, Dissertation, My Art

Explicit Multiform References

Multiform references (MFRs) are classified into three main types: explicit, implicit and indeterminate. Within a multiform argument (MFA), an explicit MFR announces its own juxtaposition of the verbal and visual components, usually through brackets in the main text, and at the beginning of the caption of the illustration. In the brackets, the term ”fig. n.” – “fig.” for “figure” and “n.” for “number,” that are usually written in Arabic or Roman numerals – comes to signify that these are “the right” time and place, along the consumption of the illustrated text, to direct attention to the illustration, that is displayed above its corresponding caption (Fig. 1). It is important to note that “fig. n.” is a hybrid sign, as it starts with a verbal anacronym and ends with a number; the “fig.” represents the kind of representation to which the author directs the readers (i.e., figure, plate or illustration), while the “n.” is the number of the cases in which that kind of representation has been used so far within the publication.

Fig. 1 Noa Yaari, Figure One, 2018

The presence of two identical-hybrid signs, one embedded in the main text and one at the beginning of the caption, establishes an explicit MFR that can generate shifts of attention in a bidirectional way: from words to images and vice versa. When readers encounter “fig. n.,” that is, an explicit MFR, they have the freedom to cooperate with the author and shift their attention from the main text to the caption and the illustration or ignore the conventional sign (Fig. 2). Explicit MFRs operate between different MFAs within a single publication and between various publications as they operate between words and images within an MFA; their direction of readers’ attention somewhere else is done consciously and clearly. Although explicit MFRs are quite conventional, my study of MFAs shows that they can come in different styles.

Fig. 2 Noa Yaari, Figure Two, 2018

 

Posted in Dissertation

Multiform References

The integration of the verbal and visual components of multiform arguments (MFAs) into a cohesive utterance is possible by juxtaposing and entangling the semantic and visual aspects of words and images. This is feasible since printed words and printed images have the capacity to be both meaningful and visible at the same time. Further, their visibility and meaningfulness can very well exist while the other medium is also visible and meaningful. The mechanisms that coordinate and integrate the visibility and meaningfulness of words and images within an MFA are “multiform references” (MFRs).

Noa Yaari, Communication, 2018.

Since MFRs are a kind of reference, they have a potential to influence the readers to shift their attention from words to images and vice versa; however, there is nothing intrinsic in MFRs that guarantees a change in readers’ behavior. Thus, when we discuss MFRs, we relate to a rhetorical device that has a potential to change readers’ mode of reading and observing the text. The actual power of MFRs to change readers’ behavior will be clearer after we track readers’ eye movements while they consume MFAs. Tracking readers’ eye movements will help us reveal the varied patterns involved in reading illustrated texts, among them the elements that drive readers to shift their attention between the verbal and visual components; the velocity of readers’ reaction to visual and semantic stimuli (that is manifested in rapid eye movements, i.e., saccades, that are measured in milliseconds); and the duration in which their gaze rests upon the different components. Finally, tracking readers’ eye movements will also allow us to understand better the “compelling” element that thinkers have ascribed to word and image relation.

 

Noa Yaari, Business, 2018.

Indeed, Saussure, Wittgenstein and Bal have pointed out the power embedded in word and image relation to forcefully evoke thoughts in one’s mind. Charles Sanders Peirce’s classification of signs into three modes: symbolic, iconic and indexical, may be considered as a framework to think about the compelling or involuntary effect word and image relation has upon readers. According to Peirce, indexical signs are connected with the things or objects they represent by a physical, organic connection (2.229). Therefore, the direct and contiguous connection between indexical signs and the things they represent enables indices to “direct the attention to their objects by blind compulsion” (2.306). Peirce also argues that indices can be thought of “as a fragment torn away from the object, the two in their existence being one whole or a part of such whole“ (2.230). If words and images have pushing and pulling forces between them, that can shift individuals’ attention from the words to the images and vice versa, across the MFA’s space, it implies that words and images may be physical parts of a whole. This conceptual framework suits well my understanding of MFAs and especially MFRs, that might be a whole whose verbal and visual poles are “torn” from its body.

 

Bal, Mieke. Reading “Rembrandt”: Beyond the Word-Image Opposition. Cambridge: Cambridge University Press, 1991.
de Saussure, Ferdinand. Course in General Linguistics. Edited by Charles Bally, Albert Sechehaye and Albert Riedlinger. Translated and annotated by Roy Harri. London: Duckworth, 1983.
Peirce, Charles S. Collected Papers of Charles Sanders Peirce, 8 vols. Edited by Charles Hartshorne and Paul Weiss, Cambridge, MA: The Belknap Press of Harvard University Press, 1931-58.
Wittgenstein, Ludwig. Philosophical Investigations. Translated by G.E.M. Anscombe, P.M.S. Hacker, and Joachim Schulte. Malden: Wiley-Blackwell, 2009.
Posted in Uncategorized

Multiformity vs. Multimodality

The concept “multiformity” resonates with the concept “multimodality,” not only in the way it looks and sounds, but also the broad social and cultural contexts from which it develops and to which it is addressed. Both concepts allow us to explore forms of communication that mix what we apprehend as distinct channels of transmitting information: written and spoken words; still and moving visual images; sound; gestures and so on. Both concepts ponder the great influence new technologies have on human interaction, and their implications in the arts and education, especially in the digital era. Therefore, it is important to clarify the main difference between the “multiformity” that I offer and Gunther Kress’s “multimodality” (2010).

The difference between the two concepts or approaches lies in the weight the two ascribe to social versus biological factors in human communication, and consequently the approaches’ use of the concepts “mode” versus “form.” “Mode,” according to Kress, is a semiotic resource for making meaning; its potential is in the affordance of the material it is made of (i.e. sound in speech or graphic stuff in writing), while its realization is the practical use of materialistic potentials by members of a given society. Further, modes differ in their underlying “semiotic logic;” the organizing principle under which they are conveyed. Kress claims, for example, that the logic of words is to follow each other in temporal sequence, and the logic of images is to display their elements simultaneously in space (2010: 81-82).

Noa Yaari, Multimodality, 2014.

Kress’s approach to multimodality is social; it sees the sign-makers’ use of signs as the mechanism that generates potential semantic meanings, rather than grammar, that can be thought of as an abstract and fixed system of rules originating in the brain. According to the social approach, the process that refashions lingual resources and practices is humans’ motivation to frame meanings in social context; this enables lingual interactions to reflect the present, with its instability: its social and technological transformations (Kress 1996, 2010).

Noa Yaari, Multiformity, 2018.

“Form,” on the other hand, relates to the most fundamental level that one can explore in relation to verbal and visual, or any other type of communication, while the level itself is defined according to the question in hand. For example, while exploring written language, we may focus on forms such as a graphic element in a letter, a letter, a sentence, a shape of a paragraph, etc. Forms in spoken language can be a phoneme, a word, a whole speech, and even a sound in the background of the speech. Forms in visual (non-verbal) communication can be a brush stroke, an artwork, a garden or building, but also a graphic element within a letter or a gesture of a speaker, during their speech.

Thus, the use of the concepts “form” and “multiform” attempts to open a theoretical framework to immediate and direct semiotic phenomena of any type and scale, so that they can shape and convey meanings synergistically, before any established logic interferes between them. It implies that “forms” function in high velocity; they do not have time to distinguish between temporal sequence, on the one hand, and simultaneous display in space, on the other, nor between ”time” and “space.” The reason why they are fast is that they function on both the social and biological levels. The body’s reaction to any kind of stimuli and processing them into meanings, given probable social conditioning and contexts, cannot rule out survival mechanisms.

 

Kress, Gunther. Multimodality: A Social Semiotic Approach to Contemporary Communication. New York: Routledge, 2010.
Kress, Gunther, and Theo van Leeuwen. Reading Images: The Grammar of Visual Design. New York: Routledge, 1996.

 

Posted in Dissertation, My Art

What are Multiform Arguments?

Multiform arguments (hereafter MFAs) are arguments that are composed of words and images. MFAs claim that a certain phenomenon has happened or explain the reasons for its occurrence. I became aware of MFAs while reading illustrated historiography, especially of cultural history. I noticed that when I wanted to point out how cultural historians use visual evidence in their publications, I lacked terms to describe what I read and observed. While it was easy to recognize and pinpoint textual units such as “words,” “images,” “sentences,” “captions” and ”list of illustrations” in those publications, it was challenging to describe combinations of words and images in the same argumentative sequence and space.

Noa Yaari, This is My Home, 2018.

The lack of terms that denote multiform components in historiography, I believe, has deterred authors and readers from exploring these hybrid forms, and subsequently constructing them more sophisticatedly, creatively and effectively. Moreover, since studies have shown that language shapes thinking, it would be reasonable to assume that without copious nomenclature not only the language of illustrated arguments might stay restricted, but also the notions this language is meant to convey. In addition, without adequate nomenclature it is impossible to establish a critical discourse about using multiformity in historical arguments. The concept “MFA” seems to be a proper beginning of a solution, since it sets forth the multiplicity of the semiotic systems and their combination within an argument, as well as the argumentative context in which these combinations function. Moreover, the concept “MFA” allows us to investigate the semantic, spatial and epistemological aspects of verbal-visual insights; to examine their cohesiveness, focusing on the way they expand in space and the time that it takes to consume their body, and their coherence, paying attention to their logical consistency. “MFA,” therefore, promotes the creation and communication of historical knowledge through diverse and hybrid forms of thought and expression. It cultivates an experimental, theoretical and critical approach to textual multiformity; and another perspective on life, in the past and right now.

Posted in Dissertation, My Art

History Sells

            

Noa Yaari, History Sells (homage to Andy Worhol), 2015. A bookmark. 20.5 x 5 cm. Toronto
Posted in My Art

In the

Noa Yaari,  I Lose Control, 2018. Ink on paper. 21 x 27 cm. Toronto

 

Image source – Sandro Botticelli’s The Birth of Venus (1482-1485):
https://en.wikipedia.org/wiki/The_Birth_of_Venus#/media/File:Venus_botticelli_detail.jpg
Posted in My Art

What Makes Visual Evidence “Evidence”?

Alfa Romeo 4C Spider

In April’s post, I asked if – within the industry of creating and communicating historical knowledge – there is any epistemological significance to visual evidence that was photographed by the historian who writes about it. I claimed that participating in the production of the visual evidence is an “opportunity to compare the production of the artwork to the production of its reproduction;” and that, based on this comparison, historians may become more aware of the dependence of the visual evidence on the context in which it is produced, seen and recorded, and consequently to its limitation in representing broader cultural and social phenomena.

Ferrari 488 GTB

So, if producing visual evidence in illustrated historiography is no different than artistic activity, as both processes are deeply influenced by the ever-changing conditions around them, what makes the visual evidence “evidence”? What is in an object or image that makes it a source of historical knowledge, within and outside of the publication of the historical research? If we imagine an artwork and illustrated-history book as two artworks or two pieces of evidence, that exist at the same time, and in either case the illustrated book reproduces the artwork as an indication of a broader phenomenon, then the book has to “freeze” the artwork and itself at a certain point in time. It must do so to claim that what it creates and communicates is “knowledge.” What power does the book have over the artwork if they are two simulations artworks or pieces of evidence? And whatever the answer may be, does the artwork have the same or similar power over the book?

Lamborghini Huracán Performante Spyder

The Lamborghini above is driving left to the white line. Also the Alfa Romeo. What does it mean? Does is have to mean anything?

 

Alfa Romeo 4C Spider – image source: http://www.carmag.co.za/car-reviews/road-tests/road-test-alfa-romeo-4c-spider/
Ferrari 488 GTB – image source: https://auto.ferrari.com/en_EN/sports-cars-models/car-range/ferrari-488-gtb/
Lamborghini Huracán Performante Spyder – image source: https://www.lamborghini.com/en-en/models/huracan/huracan-performante-spyder
Posted in Contemplations