Welcome to MFG: Welcome to the CRRS Library!

I’m currently working on an art project in the library of the Centre for Reformation and Renaissance Studies at the University of Toronto. The project is an installation entitled “Image-Text Relationships at the CRRS Library,” and it’s taking place in the hallway that leads to the library.

       

Welcome to the CRRS Library!

 

The project has three main goals. First, to make the “multiform grammar” (MFG) accessible in higher education and to the general public. Second, to use a “body of knowledge” as tangible art material. And third, to use a space on campus for the practice of the MFG. These goals raise the questions: What is MFG? What is the “body of knowledge”? How will MFG be practiced at the CRRS library?

MFG is a grammatical system that I developed for the combination of words and images. One of its main concepts is “multiform arguments” (MFAs); these are arguments that are composed of both words and images. Here’s an example of an MFA that discusses Duke of Anjou’s chain of command in 1567-1568 through a combination of words and a diagram. This double-spread is only part of this MFA.

James B. Wood, The King’s Army, 1996, pp. 78-79

 

Another central concept is “multiform references” (MFRs); these are the rhetorical devices that authors use to shift their readers’ attention between words and images to create a new, unified representation. As we can see, Wood uses terms such as “Anjou,” “chain of command,” “figure 11,” “diagram,” “Swiss,” and “1567” and “1568” to integrate the verbal and the visual components of his MFA.

Noa Yaari, MFRs in James B. Wood’s The King’s Army, 1996, pp. 78-79, 2019, Toronto

 

Lastly, the concepts “verbal” and “visual poles” describe the points between which the MFRs extend. In Wood’s MFA, the visual component is a diagram that represents by analogy the chain of command. Thus, some of the MFRs in this MFA extend between the term “diagram” and the diagram itself.

Noa Yaari, Verbal and Visual Poles in James B. Wood’s The King’s Army, 1996, pp. 78-79, 2019, Toronto

 

The body of knowledge that I use is MFAs in the books that the CRRS library holds. This is what I do: scan thirty MFAs, print the scans, remove the captions of the illustrations from the prints and, consequently, from the MFAs, paint on the prints, scan the paintings, print the scans, laminate the prints, and create the installation from the reproductions of the paintings and the independent captions.

Noa Yaari, CRRS 4/30. 2019. Mixed media. 21 x 28 cm. Toronto

 

Noa Yaari, CRRS 6/30. 2019. Mixed media. 21 x 28 cm. Toronto

 

Noa Yaari, CRRS 7/30. 2019. Mixed media. 21 x 28 cm. Toronto

 

Noa Yaari, CRRS 8/30. 2019. Mixed media. 21 x 28 cm. Toronto

 

I will practice the MFG by extending MFRs across the hallway that leads to the CRRS library.

Noa Yaari, Welcome to MFG: Welcome to the CRRS Library! (simulation), 2019, Toronto
Posted in My Art, Research

Using Multiform Grammar in Presentations

How do speakers who use both words and images employ MFG? And how can they do so intentionally and effectively? As you can see in the illustration below, the speaker refers to the image on the screen in three different ways. These are three types of multiform references (MFRs) that integrate words and images into a unified meaning. The first MFR is an explicit one, in which the speaker refers to the image directly and overtly by saying: “as the image shows…” When the audience hears this phrase, it knows that anything that will be said afterward will address what the image on the screen represents in the eyes of the speaker. By using an explicit MFR, the speaker develops a clear expectation among the audience and, consequently, enables it to agree or disagree with the speaker’s multiform argument (MFA).

The second phrase says: “the modern perception of Earth…” In this case, the MFR between the words and the image is an implicit one as it doesn’t refer directly to what the image shows. Instead, this implicit MFR builds on the semantic similarity that the term “Earth” and the image on the screen maintain. Does the image on the screen represent “Earth”? Does “Earth” represent the image? Not necessarily, since there are various ways to visually represent “Earth,” and there are various meanings that the image on the screen may signify, which we can describe through terms other than “Earth.” Put differently, the term “Earth” and the image on the screen are not “synonyms,” although they may denote, in specific contexts, the same meaning. Thus, in implicit MFRs, the context plays a role; we do need to know how “the modern perception of…” in this phrase operates within the whole MFA, to develop any critical view of this multiform rhetoric. Importantly, even with such a critical view, in this example, there is no explicit reference between the spoken component of the MFA and its visual one.

The third phrase says: “we are the world…” Do the phrase and the image on the screen create a new, unified meaning? And if so, how do they do this? The semantic distance between the term “world” and the image is shorter than the one between “we” and the image. Yet, to what extent the term “world” and the image denote the same meaning? When the speaker says “world” or “the world,” do they refer to what the image on the screen represents? Well, also here, it depends on the context. We need to know how the speaker uses the term “world” and this image, within the MFA, to assess the connection between the two. At the same time, since the phrase “we are the world” is a metaphor that describes who “we” are, and the image may (in specific contexts) denote “the world,” we may interpret the image as “we.” If we do so – if this is the new unified meaning that this multiform rhetoric implies – then we should look for ideologies that are subtly expressed between words and images, through indeterminate MFRs.

Posted in Contemplations, Research, Teaching

Zooming in on Multiform References

Posted in Research, Teaching

Visual Rhetoric : )

What happens to a punctuation mark that transforms into a representation of a pair of eyes or a smiling mouth? Is it still a punctuation mark, or is it a sign of a different kind?

Let’s examine a colon that signifies a pair of eyes and a curved bracket that signifies a smiling mouth. Noticeably, as representations of eyes and mouth, the colon and bracket become facial features of a horizontal head. However, like punctuation marks, their verticality still aligns with the verticality of the letters, that seem to stand or sit straight in their line (Fig. 1). The fact that the colon and the bracket are vertical and horizontal at the same time creates a “double paradox:” on the one hand, they align with the verticality of the letters but also with the horizontality of the head; and one the other, they contradict the verticality of the letters but also the horizontality of the head.

: )

Fig. 1. : )

The composition of the eyes and mouth within both the face and the written line is significant as well. For example, when it comes to English, the question of whether the eyes are left to the mouth or vice versa influences the meaning of the verbal-visual phrase. If the eyes are left to the mouth and the whole face is right to the written words, the text and consequently the reading become bidirectional; the words direct the readers’ attention from the left to the right and the face from the right to the left (Fig. 2). Moreover, even if there are no words in that line, the face directs the readers “backward” since the colon and the bracket imply that this is a written environment, within which the readers automatically move from left to right. How would a smiling face look like in, for instance, Hebrew in a way that makes sense (Fig. 3)?

Visual Rhetoric : )

Fig. 2. Visual Rhetoric : )

( : רטוריקה ויזואלית

Fig. 3. Visual Rhetoric : ) in Hebrew

Looking at figure 2 and 3, we could argue that the vicinity of the eyes to the written words implies that the brain that is ostensibly above the eyes has uttered the words; that the owner of the face is the author of the text. Would the vicinity of the mouth to the text imply that the mouth has uttered the text? Would it signify authorship as well? I think that, in English, when the mouth is left to the eyes and the text is left to the face, the face looks like someone who anticipates something; that, from their point of view, there is more to come (Fig. 4).

Visual Rhetoric ( :

Fig. 4. Visual Rhetoric ( :

Finally, the transformation of a punctuation mark into a representation of a facial feature that signifies a specific meaning results from a synchronized transformation of at least two punctuation marks. For example, only the vicinity of the colon to the concave side of the bracket makes the latter a smiling mouth, which, in turn, makes the colon eyes of a smiling face. Without this synchronicity – that happens in space and time – these signs will stay punctuation marks. As such, they will operate only when words or other punctuation marks are on both their sides (Fig. 4).

Visual: Rhetoric (!).

Fig. 4. Visual: Rhetoric (!).

So, when the colon and the bracket signify a pair of eyes and a smiling mouth, are they still punctuation marks or signs of a different kind? It’s hard to tell because after they transform into representations of facial features, they still clarify the meaning of phrases and sentences just as punctuation marks do. Could they be a new kind of punctuation marks? Perhaps, because as our technology and society rapidly change, so do our communicative practices.

Posted in Research

Into My Arms: A Blogpost in Progress

Noa Yaari, Into My Arms I, 2019. Ink on Paper, 17.7×25.4 cm. Toronto.

Noa Yaari, Into My Arms II, 2019. Ink on Paper, 17.7×25.4 cm. Toronto.

 

 

Posted in My Art

Multiform Grammar

Noa Yaari, Multiform Grammar is Making Us Rich, 2019. Mixed media, 17.7×25.4 cm. Toronto.
Posted in My Art

Multiform Grammar and the Working Memory

In 1974, psychologists Alan Baddeley and Graham Hitch offered a model of the working memory, which was revised by Baddeley in 2000. According to the 1974 model, the working memory is a system that enables temporary storage and manipulation of information for complex tasks. These could include language acquisition, reading, learning, reasoning, and comprehension (Baddeley 2000). This model features the working memory as composed of three elements: the central executive and two subsidiary systems, i.e. the phonological loop and the visuospatial sketchpad.

The central executive, which lacks storage capacity, directs attention to a given source of information, whether perceptual, the two subsidiary systems, or the long-term memory (LTM). The central executive also reflects upon and modifies that information (Baddeley 2000). The two subsidiary systems, unlike the central executive, can store information and combine it with both sensory input and information from the central executive (Baddeley 2000). The phonological loop stores and rehearses acoustic and speech-based information, while the visuospatial sketchpad holds and manipulates visual images (Baddeley 1992).

Noa Yaari, The Tripartite Model of the Working Memory, 2019. Toronto

The tripartite model of the working memory has been found useful for studying cognitive differences among individuals and for examining both neuropsychological deficits and normal cognitive functioning (Baddeley 1992). At the same time, studies have illuminated the limitations of this model indicating certain cognitive phenomena which it could not explain. One such phenomenon is the higher number of words that subjects recall when those words are presented to them “chunked” into meaningful sentences or prose rather than when those words are presented unrelated (Baddeley 2000). Another phenomenon is the integration of information from different sensory channels and the LTM, that is necessary for the creation of a complex and yet coherent representation of the world and prose. Thus, the need to explain processes such as chunking and binding that integrate new and old information, and that combine information that is held in varied codes and in different components of the working memory, required the addition of another component to the 1974 model (Baddeley 2000).

Noa Yaari, Chunking and Binding in the Working Memory, 2019. Toronto

The new component that Baddeley offered is the “episodic buffer;” an interface between the sub-systems and the LTM that integrates verbal, visual and other kinds of information into a common, multimodal code. Controlled by the central executive, the episodic buffer enables a temporary, unitary and coherent representation of a multidimensional environment (2000). Interestingly, the episodic buffer resonates with the concept “multiform reference” (MFR), which I developed as an essential component of the multiform grammar (MFG). MFRs are devices that link words and images within a multiform argument (MFA) integrating their individual meanings into a multiform one, considering the MFA’s semantic and visual qualities. MFRs may be terms in the main text and in captions that have the potential to shift readers’ attention from the MFA’s verbal component to its visual one. For instance, the word “sky” in the main text has the potential to shift readers’ attention to an image of the sky, especially if this word and this image are presented near one another.

Noa Yaari, Multiform Grammar in the Working Memory, 2019. Toronto

The tripartite model of the working memory and Baddeley’s introduction of the episodic buffer help us expand the MFG by adding analysis of cognitive capacities and processes to the analysis of verbal-visual sources and especially of MFAs. Baddeley’s addition will enable us to understand better how readers consume MFAs and consequently, to offer insights into effective combination of words and images in multiform arguments and teaching.

 

Baddeley, Alan. “The Concept of Episodic Memory.” Philosophical Transactions: Biological Sciences 356.1413 (2001): 1345-50.
—. “The Episodic Buffer: A New Component of Working Memory?” Trends in Cognitive Science 4.11 (200): 417-23.
—. “Working Memory.” Science, 255.5044 (1992): 556-59.

 

Posted in Contemplations, My Art

Mother

Noa Yari, Mother, 2019. A poster. Toronto
Posted in Uncategorized

Queen

Noa Yaari, Queen, 2019. A poster. Toronto
Posted in Uncategorized

Pulling and Pushing Forces in Multiform References

Explicit, implicit and indeterminate multiform references (MFRs) maintain pulling and pushing forces between their verbal and visual poles. These forces are the mechanisms that potentially move readers to shift their attention between words and images across a multiform argument (MFA) or message. They are the invisible elements that make MFAs cohesive and coherent; the core of multiform grammar (MFG). These pulling and pushing forces, however, exist due to semantic relationships between, on the one hand, words in the main text, and on the other, words or conventional signs in the captions. In an explicit MFR, for example, “(Fig. 1)” in the main text and “(Fig. 1)” at the beginning of the caption pull and push readers’ attention in the same direction, at the same time. The sign in the main text pushes readers’ attention from the main text towards the caption, and ultimately, the image. The sign in the caption simultaneously pulls readers’ attention from the main text towards itself and the image. Thus, the signs’ identical semantic meaning, as well as their identical appearance, enable readers to effortlessly shift their attention between the MFA’s verbal and the visual components. This is how readers navigate throughout the physical and literal space of the hybrid argument, using the same mechanism.

Pulling and pushing forces in explicit MFRs, in Douglas Biow’s On the Importance of Being an Individual in Renaissance Italy. p. 185.

In implicit MFRs, the pulling and pushing forces between the verbal and the visual poles are mostly based on the semantic and the visual similarity between words in the main text and words in the caption, rather than on the convention “(Fig. 1).” It should be noted that when we discuss the verbal and visual poles, as the two ends of an implicit MFR, the visual pole may be specific words in the caption, rather than purely the image. This condition, however, might change as the reading of the MFA continues, and an association between the words in the main text and the image is established. For example, as often happens, titles of portraits are the names of the people who are depicted in them; therefore, when authors discuss the depicted figures in the main text, they inevitably mention the titles of the portraits. In the image below, the titles of the two portraits are written in the captions of the two illustrations, as well as twice in the main text. Thus, “Julius II” and “Clement VII” in the captions, on the one hand, and in the main text, on the other, create poles between which pulling and pushing forces take place. My assumption is that when the two names are mentioned in the main text the second time, the readers find it easier to associate these names directly with the images. However, since in our example one caption is below the other, this association might take longer. At any rate, the semantic and the visual similarity between the names in the main text and the names in the captions, constitute pulling and pushing forces that enable readers to navigate throughout the MFA.

Pulling and pushing forces in implicit MFRs, in Douglas Biow’s On the Importance of Being an Individual in Renaissance Italy. p. 185.

What happens when an MFA uses explicit as well as implicit MFRs? What happens when its space is saturated with varied pulling and pushing forces, between its words and images? A large number of referential conventions, words, as well as images that can serve as verbal or visual poles for MFRs increases the probability that readers will shift their attention between the MFA’s verbal and visual components. These shifts, in turn, strengthen readers’ comprehension and retention of the MFA, since they create associations between the representational level of the MFA and the phenomenon it discusses. I would like to argue that each MFA has an ideal number of MFRs to reach its ultimate comprehension and retention among its readers. Grasping this number is to use MFG effectively and creatively since, at the end of the day, MFRs are rhetorical devices.

Pulling and pushing forces in explicit and implicit MFRs, in Douglas Biow’s On the Importance of Being an Individual in Renaissance Italy. p. 185.
Posted in Contemplations, Dissertation, Teaching