- March 2020 (1)
- February 2020 (1)
- January 2020 (1)
- December 2019 (1)
- November 2019 (1)
- October 2019 (1)
- September 2019 (1)
- August 2019 (1)
- July 2019 (1)
- June 2019 (1)
- May 2019 (1)
- April 2019 (1)
- March 2019 (1)
- January 2019 (1)
- December 2018 (1)
- November 2018 (1)
- October 2018 (1)
- September 2018 (1)
- August 2018 (1)
- July 2018 (1)
- June 2018 (1)
- May 2018 (1)
- April 2018 (1)
- March 2018 (1)
- February 2018 (1)
- January 2018 (1)
- December 2017 (1)
- November 2017 (1)
- October 2017 (1)
- September 2017 (1)
- August 2017 (1)
- July 2017 (1)
- June 2017 (1)
- May 2017 (1)
- April 2017 (1)
- January 2017 (1)
- December 2016 (1)
- November 2016 (1)
- October 2016 (1)
- September 2016 (1)
- July 2016 (2)
- June 2016 (1)
- May 2016 (3)
- April 2016 (3)
- March 2016 (3)
Today I gave a talk at the conference “World History Association of Texas – Phi Alpha Theta” at St. Edward’s University in Austin, Texas (via Zoom). My talk entitled “Approaching History and the Arts with Multiform Grammar” introduced my academic and artistic exploration of the communication of ideas through combinations of words and images.
In my talk, I presented a scan of a double spread (pp. 148-49) from the chapter “The Perfecting of the Individual” in Jacob Burckhardt’s The Civilization of the Renaissance in Italy, that he wrote in 1860 and Johannes Jahn illustrated in 1926. I showed how Jahn embedded two images in this double spread: one of the artist Andrea Mantegna, and the other of the polymath Leon Battista Alberti. I pointed out that, in this chapter, while Burckhardt uses Alberti as an example of the development of what he observed as the Renaissance, “all-sided” man, he doesn’t write anything about Mantegna. My highlights of Alberti’s name, image, and pronounce in blue, throughout this double spread, visualize this imbalance.
Jacob Burckhardt, The Civilization of the Renaissance in Italy (New York, 1958), pp. 148-49
During the Q&A session, we discussed Jahn’s decision to embed the two images, although the text refers only to Alberti and the influence this might have on Burckhardt’s argument. We wondered if the section in which Mantegna is “doesn’t work” while the one that includes Alberti does. I claimed that we could benefit from suspending our judgment of this multiform situation and that the fact that Mantegna and Alberti are present in this chapter in different ways may even raise useful questions. For example, would it be possible that the use of various strategies of presentation in a single chapter increases the possibility that we read this chapter every time differently? Would this effect, in turn, encourage us to return to this chapter, to look for information that we thought we could find there? Moreover, perhaps applying various modes of presentation in a single document has some merit. It reminds us that the individuals involved in the making of illustrated literature have some freedom and that it’s fun to see them practicing it even if the result looks odd.
Jacob Burckhardt, The Civilization of the Renaissance in Italy (New York, 1958), pp. 148-49
I’m currently working on an art project in the library of the Centre for Reformation and Renaissance Studies at the University of Toronto. The project is an installation entitled “Image-Text Relationships at the CRRS Library,” and it’s taking place in the hallway that leads to the library.
Welcome to the CRRS Library!
The project has three main goals. First, to make the “multiform grammar” (MFG) accessible in higher education and to the general public. Second, to use a “body of knowledge” as tangible art material. And third, to use a space on campus for the practice of the MFG. These goals raise the questions: What is MFG? What is the “body of knowledge”? How will MFG be practiced at the CRRS library?
MFG is a grammatical system that I developed for the combination of words and images. One of its main concepts is “multiform arguments” (MFAs); these are arguments that are composed of both words and images. Here’s an example of an MFA that discusses Duke of Anjou’s chain of command in 1567-1568 through a combination of words and a diagram. This double-spread is only part of this MFA.
James B. Wood, The King’s Army, 1996, pp. 78-79
Another central concept is “multiform references” (MFRs); these are the rhetorical devices that authors use to shift their readers’ attention between words and images to create a new, unified representation. As we can see, Wood uses terms such as “Anjou,” “chain of command,” “figure 11,” “diagram,” “Swiss,” and “1567” and “1568” to integrate the verbal and the visual components of his MFA.
Noa Yaari, MFRs in James B. Wood’s The King’s Army, 1996, pp. 78-79, 2019, Toronto
Lastly, the concepts “verbal” and “visual poles” describe the points between which the MFRs extend. In Wood’s MFA, the visual component is a diagram that represents by analogy the chain of command. Thus, some of the MFRs in this MFA extend between the term “diagram” and the diagram itself.
Noa Yaari, Verbal and Visual Poles in James B. Wood’s The King’s Army, 1996, pp. 78-79, 2019, Toronto
The body of knowledge that I use is MFAs in the books that the CRRS library holds. This is what I do: scan thirty MFAs, print the scans, remove the captions of the illustrations from the prints and, consequently, from the MFAs, paint on the prints, scan the paintings, print the scans, laminate the prints, and create the installation from the reproductions of the paintings and the independent captions.
Noa Yaari, CRRS 4/30. 2019. Mixed media. 21 x 28 cm. Toronto
Noa Yaari, CRRS 6/30. 2019. Mixed media. 21 x 28 cm. Toronto
Noa Yaari, CRRS 7/30. 2019. Mixed media. 21 x 28 cm. Toronto
Noa Yaari, CRRS 8/30. 2019. Mixed media. 21 x 28 cm. Toronto
I will practice the MFG by extending MFRs across the hallway that leads to the CRRS library.
Noa Yaari, Welcome to MFG: Welcome to the CRRS Library! (simulation), 2019, Toronto
How do speakers who use both words and images employ MFG? And how can they do so intentionally and effectively? As you can see in the illustration below, the speaker refers to the image on the screen in three different ways. These are three types of multiform references (MFRs) that integrate words and images into a unified meaning. The first MFR is an explicit one, in which the speaker refers to the image directly and overtly by saying: “as the image shows…” When the audience hears this phrase, it knows that anything that will be said afterward will address what the image on the screen represents in the eyes of the speaker. By using an explicit MFR, the speaker develops a clear expectation among the audience and, consequently, enables it to agree or disagree with the speaker’s multiform argument (MFA).
The second phrase says: “the modern perception of Earth…” In this case, the MFR between the words and the image is an implicit one as it doesn’t refer directly to what the image shows. Instead, this implicit MFR builds on the semantic similarity that the term “Earth” and the image on the screen maintain. Does the image on the screen represent “Earth”? Does “Earth” represent the image? Not necessarily, since there are various ways to visually represent “Earth,” and there are various meanings that the image on the screen may signify, which we can describe through terms other than “Earth.” Put differently, the term “Earth” and the image on the screen are not “synonyms,” although they may denote, in specific contexts, the same meaning. Thus, in implicit MFRs, the context plays a role; we do need to know how “the modern perception of…” in this phrase operates within the whole MFA, to develop any critical view of this multiform rhetoric. Importantly, even with such a critical view, in this example, there is no explicit reference between the spoken component of the MFA and its visual one.
The third phrase says: “we are the world…” Do the phrase and the image on the screen create a new, unified meaning? And if so, how do they do this? The semantic distance between the term “world” and the image is shorter than the one between “we” and the image. Yet, to what extent the term “world” and the image denote the same meaning? When the speaker says “world” or “the world,” do they refer to what the image on the screen represents? Well, also here, it depends on the context. We need to know how the speaker uses the term “world” and this image, within the MFA, to assess the connection between the two. At the same time, since the phrase “we are the world” is a metaphor that describes who “we” are, and the image may (in specific contexts) denote “the world,” we may interpret the image as “we.” If we do so – if this is the new unified meaning that this multiform rhetoric implies – then we should look for ideologies that are subtly expressed between words and images, through indeterminate MFRs.
What happens to a punctuation mark that transforms into a representation of a pair of eyes or a smiling mouth? Is it still a punctuation mark, or is it a sign of a different kind?
Let’s examine a colon that signifies a pair of eyes and a curved bracket that signifies a smiling mouth. Noticeably, as representations of eyes and mouth, the colon and bracket become facial features of a horizontal head. However, like punctuation marks, their verticality still aligns with the verticality of the letters, that seem to stand or sit straight in their line (Fig. 1). The fact that the colon and the bracket are vertical and horizontal at the same time creates a “double paradox:” on the one hand, they align with the verticality of the letters but also with the horizontality of the head; and one the other, they contradict the verticality of the letters but also the horizontality of the head.
Fig. 1. : )
The composition of the eyes and mouth within both the face and the written line is significant as well. For example, when it comes to English, the question of whether the eyes are left to the mouth or vice versa influences the meaning of the verbal-visual phrase. If the eyes are left to the mouth and the whole face is right to the written words, the text and consequently the reading become bidirectional; the words direct the readers’ attention from the left to the right and the face from the right to the left (Fig. 2). Moreover, even if there are no words in that line, the face directs the readers “backward” since the colon and the bracket imply that this is a written environment, within which the readers automatically move from left to right. How would a smiling face look like in, for instance, Hebrew in a way that makes sense (Fig. 3)?
Visual Rhetoric : )
Fig. 2. Visual Rhetoric : )
( : רטוריקה ויזואלית
Fig. 3. Visual Rhetoric : ) in Hebrew
Looking at figure 2 and 3, we could argue that the vicinity of the eyes to the written words implies that the brain that is ostensibly above the eyes has uttered the words; that the owner of the face is the author of the text. Would the vicinity of the mouth to the text imply that the mouth has uttered the text? Would it signify authorship as well? I think that, in English, when the mouth is left to the eyes and the text is left to the face, the face looks like someone who anticipates something; that, from their point of view, there is more to come (Fig. 4).
Visual Rhetoric ( :
Fig. 4. Visual Rhetoric ( :
Finally, the transformation of a punctuation mark into a representation of a facial feature that signifies a specific meaning results from a synchronized transformation of at least two punctuation marks. For example, only the vicinity of the colon to the concave side of the bracket makes the latter a smiling mouth, which, in turn, makes the colon eyes of a smiling face. Without this synchronicity – that happens in space and time – these signs will stay punctuation marks. As such, they will operate only when words or other punctuation marks are on both their sides (Fig. 4).
Visual: Rhetoric (!).
Fig. 4. Visual: Rhetoric (!).
So, when the colon and the bracket signify a pair of eyes and a smiling mouth, are they still punctuation marks or signs of a different kind? It’s hard to tell because after they transform into representations of facial features, they still clarify the meaning of phrases and sentences just as punctuation marks do. Could they be a new kind of punctuation marks? Perhaps, because as our technology and society rapidly change, so do our communicative practices.
Noa Yaari, Into My Arms I, 2019. Ink on Paper, 17.7×25.4 cm. Toronto.
Noa Yaari, Into My Arms II, 2019. Ink on Paper, 17.7×25.4 cm. Toronto.
In 1974, psychologists Alan Baddeley and Graham Hitch offered a model of the working memory, which was revised by Baddeley in 2000. According to the 1974 model, the working memory is a system that enables temporary storage and manipulation of information for complex tasks. These could include language acquisition, reading, learning, reasoning, and comprehension (Baddeley 2000). This model features the working memory as composed of three elements: the central executive and two subsidiary systems, i.e. the phonological loop and the visuospatial sketchpad.
The central executive, which lacks storage capacity, directs attention to a given source of information, whether perceptual, the two subsidiary systems, or the long-term memory (LTM). The central executive also reflects upon and modifies that information (Baddeley 2000). The two subsidiary systems, unlike the central executive, can store information and combine it with both sensory input and information from the central executive (Baddeley 2000). The phonological loop stores and rehearses acoustic and speech-based information, while the visuospatial sketchpad holds and manipulates visual images (Baddeley 1992).
Noa Yaari, The Tripartite Model of the Working Memory, 2019. Toronto
The tripartite model of the working memory has been found useful for studying cognitive differences among individuals and for examining both neuropsychological deficits and normal cognitive functioning (Baddeley 1992). At the same time, studies have illuminated the limitations of this model indicating certain cognitive phenomena which it could not explain. One such phenomenon is the higher number of words that subjects recall when those words are presented to them “chunked” into meaningful sentences or prose rather than when those words are presented unrelated (Baddeley 2000). Another phenomenon is the integration of information from different sensory channels and the LTM, that is necessary for the creation of a complex and yet coherent representation of the world and prose. Thus, the need to explain processes such as chunking and binding that integrate new and old information, and that combine information that is held in varied codes and in different components of the working memory, required the addition of another component to the 1974 model (Baddeley 2000).
Noa Yaari, Chunking and Binding in the Working Memory, 2019. Toronto
The new component that Baddeley offered is the “episodic buffer;” an interface between the sub-systems and the LTM that integrates verbal, visual and other kinds of information into a common, multimodal code. Controlled by the central executive, the episodic buffer enables a temporary, unitary and coherent representation of a multidimensional environment (2000). Interestingly, the episodic buffer resonates with the concept “multiform reference” (MFR), which I developed as an essential component of the multiform grammar (MFG). MFRs are devices that link words and images within a multiform argument (MFA) integrating their individual meanings into a multiform one, considering the MFA’s semantic and visual qualities. MFRs may be terms in the main text and in captions that have the potential to shift readers’ attention from the MFA’s verbal component to its visual one. For instance, the word “sky” in the main text has the potential to shift readers’ attention to an image of the sky, especially if this word and this image are presented near one another.
Noa Yaari, Multiform Grammar in the Working Memory, 2019. Toronto
The tripartite model of the working memory and Baddeley’s introduction of the episodic buffer help us expand the MFG by adding analysis of cognitive capacities and processes to the analysis of verbal-visual sources and especially of MFAs. Baddeley’s addition will enable us to understand better how readers consume MFAs and consequently, to offer insights into effective combination of words and images in multiform arguments and teaching.
Baddeley, Alan. “The Concept of Episodic Memory.” Philosophical Transactions: Biological Sciences 356.1413 (2001): 1345-50.
—. “The Episodic Buffer: A New Component of Working Memory?” Trends in Cognitive Science 4.11 (200): 417-23.