Skip to main content

Music And Comics: Two Great Tastes That Taste Great Together?

You like comics, right? Sure. That's why you're here. Chances are you like music too. And you've probably used the web as a means to enjoy both. The Internet has proved a stunningly efficient instrument for distribution of both comics and music separately, but what happens when technology brings the two together? Shouldn't two good things combine to make something great?

Well... maybe, maybe not...

 

A MATTER OF DEBATE

Whenever the subject of combining music and comics comes up, there are two arguments almost invariably arise against it:

  1. The addition of sound to comics is an unnecessary gimmick and only results in distraction and inconvenience for the reader, and
  2. the addition of music takes something away from the essence of what comics is in the first place.

The former argument boils down to personal preference. Some people would rather to choose their own soundtrack for their reading experience than have one prescribed, and some prefer no soundtrack at all. There's not much to argue against that. The latter argument, however is open to considerable debate.

Try as they might no one has come up with a clear-cut, irrefutable definition of what "comics" is. The advent of web technology and the possibility of integrating elements such as animation and sound has blurred the picture even more. What is it that makes a comic a comic, and how does the addition of these other elements affect that? Is there something essential to comics that is lost in the attempt to integrate sound?

It is beyond the scope of this article to propose any new definitions, but when discussing comics in combination with music it's important to note some distinctions of form.

 

PACE YOURSELF

On the surface comics and music are not dissimilar forms. Both rely extensively on rhythm and timing to create their desired effects. The disparity of image versus sound is not an insurmountable distinction either. After all we both see and hear thing simultaneously all the time. The main difference between comics and music, and the stumbling block toward integration, is the way we process the information given by each. Music is assimilated passively - tones and rhythms are processed in real time with the performer in complete control. Comics are processed more dynamically. The audience is in control as the interplay of elements is deciphered, or read. And everyone reads at their own pace.

So what happens when we mix the two distinct forms? Can the differences be reconciled, and if so what does it mean for comics?

Let's take a look at some techniques.

 

LAYING OUT THE BEAT

As stated comics and music have a common reliance on "beat." Comics' rhythm comes from a division of space while music's is temporal. The obvious idea for depicting the rhythm of music in comics is a visual abstraction.

If time can be represented by space in comics then the division of space can mimic divisions in time. All we need are visual cues. We have a head start in finding them in fact, as a visual abstraction of music has already existed for hundreds of years.

 

The staff and bars are used to represent tone. The type of note gives us the cue to the rhythm. It's a visual depiction of sound. Musical notation is not comics, though it too is read and deciphered. But it gives us the necessary divisions of time. If we imagine the note's duration as panels of varying size and imagine a measure as one tier of panels we get this layout:

Still not quite a comic, but if we introduce elements that reinforce the notion of the panels duration...

Voila... music as comics

It's similar to the theory (apparently expressed by Matt Madden and recently put into practice by Derik Badman) of translating poetic rhyme scheme to comics form. Sound is interpreted visually by putting into effect a series of design constraints on the images. Here, music suggests the rhythm but the result is pure comics.

How effective is it as integration? An example such as our dancing friend falls well inside established boundaries of comics form. Music played a part in it's creation, but it's not quite a combination. Ultimately the strip stands alone. Comics? Yes. Music? No.

The next step, then, is to bring the sound into the equation - simultaneous presentation. Not actually new to comics (have you ever listened to music while reading?) it's a method made more convenient by the web.

 

MOOD MUSIC

The technique of embedding an MP3 on the page with a comic is the most common of combinations currently employed in webcomics. Most of the stories in Kean Soo's Exit Music use this technique, as do Jonathan Altschuler and Colleen Macisaac's Music Comics and many of the installments in Cat Garza's Those Were the Salad Days. The music is meant to be played as you read and acts as a kind of emotional setting or background for the narrative. The theory is that the emotional timber of the song reflects and intensifies the sentiment of the comic. The technique can be quite effective.

Take Colleen MacIsaac and Jonathan Altschuler's music-comic, #1. The narrative is simple and disquieting. Altschuler's looping, discordant of piano progression is an excellent set piece for the eerie, dream-like sequence depicted by MacIsaac's brushwork. Similarly, Kean Soo's In a Lonely Place uses the background of Joy Division's song of the same name to reflect a half-conscious emotional resonance. In each comic image and sound reflect on each other and the result can be quite striking.

In reading a combination of this type one might unconsciously fall in step to the rhythm of the music, but essentially the strip can be read at it's own pace. The music serves as background only. Overall, it retains its identity as "comics" but there are other limitations to this type of combination.

The songs tend to go on long after you've finished reading the comic. And one is left wondering if the addition of music is really vital to the reading. All of Kean Soo's Exit Music comics have appeared in print (sans music, of course). Neal Von Flue's music/comic experiments (scroll down to Synesthesia Drive In) highlight the uncertainty by offering up the comics and songs as separate elements on separate web pages. Th Music is essentially unnecessary. Still, it can make for interesting effect and the combination retains the reader controlled pacing of comics.

Sometimes it's necessary to give up a little of that control, however, for a song to better reflect a narrative. Rarely does a story (or song) only express one mood throughout. But to express any shifts in emotion comic and song must work in tandem. To synch these shifts a greater reliance on creator controlled pacing is needed.

N'SYNC - whoops, I mean, IN SYNCH

Kean Soo uses visual cues to pace the reader in Bottle Up and Explode. Printing the lyrics at key spots in the flow of the strip forces the reader to follow along at the pace of the music. The controlled pacing attempts to better match certain parts of the song to key moments in the narrative. It's a clever solution, and uniquely suited to the visual nature of comics. But it only works if the reader consciously uses the cues. Even then it can make for a clumsy reading experience as you stop and wait at each chunk of lyrics for the song to catch up with your reading.

 

MUSIC VIDEOS?

The next step in synching music to narrative is limiting the duration the reader is focused on any given panel. Using programs such as Flash (as in Broken Saints) or QuickTime (as the first few installments of Cat Garza's Salad Days) places a set time for each panel to appear and animated transitions take you to the next panel. This is where the distinction of comics/not comics really starts to make itself evident. The images themselves are static, and in the case of Broken Saints all dialogue and exposition are presented as text within the panels. But the experience of "reading" is lost with the animation, and it becomes a more passive experience of "viewing." Image and sound are perfectly synched but at the total loss of reader controlled pacing that is a hallmark of comics.

Animation can have it's place in comics, though. Garza sees the QuickTime files in Salad Days as animated "meta-panels." Taken in context with the rest of the strip, the quicktime file acts as an attention getting splash panel, or a music video embedded into the course of the comics narrative. The "meta-panel" is a concept that works equally well in a gag-strip format. Take this example from Kidd Radd. The second panel is an interactive animation (through a click-through exchange of animated gif images) set to an 8-bit soundtrack (note - apparently the sound does not work on some browsers). It's a fun little gag to play around with and it works as comics due to its simple panel-to panel-context.

 

SUPER SCIENCE

Up to this point any attempts at a true synching narrative to soundtrack in a whole-comic scale have largely been failures. Kean Soo's lyric-printing method makes for clumsy reading and the creator driven pacing of Broken Saints results in something that's not quite comics anymore. It's been said that it would take some kind of super-science to find a method that could match a soundtrack to any readers preferred pacing. So what would that "super-science" be?

One could imagine sound loops linked to certain areas of a comics page triggered by some sort of eye-tracking technology. Or perhaps with a greater understanding of brain activity, certain electro-chemical reactions used in the process of reading could be tracked and exploited to this measure. There are a number of science-fiction-like possibilities that could be put forward. The uncertainty of process, inconvenience of any needed hardware, and considerable expense of even studying the credibility of these methods renders them no more than idle speculation. Besides a simpler answer may already be right in front of us.

 

THE READER-DRIVEN SOUNDTRACK

Adobe/Macromedia's Flash, a web-native program, has proved fertile ground for animators. Flash animations proliferate on the web and offer streamlined integration of motion and sound. But the program also offers something traditional animation cannot - interactivity. It's a feature that is not lost on some of the more technologically ambitious creators of webcomics. From the experimental animated panel transitions of John Barber and the special effects work of Patrick Farley, to the clever embedded panels B. Shur uses at I Am A Rocket Builder the interactive possibilities of web animation are making their mark on comics storytelling. With click-scroll interfaces such as Daniel Merlin Goodbrey's Tarquin Engine the reader still chooses the pace but with panel-to -panel navigation, the reader's focus becomes a lot more predictable. Certain elements can be matched to specific panels. Flash has the capability to link up mouse-clicks to audio files as well, and that's where the idea of an integrated webcomic soundtrack becomes a real possibility.

Flash's sound integration has been used in webcomics before, often to somewhat irritating effect. Adding sound effects to a comic's click-through navigation can be unnecessary and distracting. Dark Horse Comics' initial online presentation of Mike Mignola's Hellboy stories offers a somewhat clumsy first look at the technique. It's a somewhat jarring combination here, the background "music" is clumsy and repetitive drone (later additions to the series dropped the music altogether) but it begs an interesting point. Perhaps with a little imagination, using layering and repeating tones and sound loops, an ambitious cartoonist could design a real-time interactive soundtrack for their comic. Dynamic, adaptive musical scores have been a staple of the video game industry for years. Given the technology available on the web it's not hard to imagine them for comics as well.

Of course even click-through, animated transitions, such as those found in Tarquin and Infinite Canvas, have their opponents in the comics/not comics debate. It seems the closer we get to true music/narrative integration the further we stray from comics. Perhaps a fully integrated soundtrack just isn't possible. But when you get down to it, all these questions of form are really beside the point.

 

WHO CARES?

Comics or not comics there are still those who prefer no sound. The deciding factor in the accessibility of these experiments in form is how compelling the content is, not how well it meets the definition of comics. Ultimately it's the entire reading experience that matters.

Does the combination create an interesting effect? Are the elements integrated in a clever way? Most importantly, is the addition of music truly necessary to the overall message? These are the only questions we need to answer. Whether or not the result retains the "purity" of comics is of interest only to nerds like me.

In order to be effective a music/comics combination only needs to be interesting. Whatever way you mix it, it just has to make sense. The music must be absolutely integral to the overall piece. That is the only measure of whether a combination works. If a cartoonist can then pull that off it just might convince the other half to turn up the volume in the first place.

 

 

Tym Godek does occasionally make webcomics, but he's never tried one with music so what does he know anyway?

Re: Music And Comics: Two Great Tastes That Taste Great Together

You really off the ball on this whole sound comics thing.

There gere you are looking for is Sound Novel.

Personally I wish it was more like that. but that kinda defeats the point of animations.

1. Whenever a reader makes up noises and sound effects in there head. This is the basic idea in all of the following.

2. Whever they practically show the comic going page by page with sound  effects,  and vioce material. Note they do this on channel PBS all the time. Back then if was a kid like me in thre 1990's and 1980's we had books with a tapes and would turn the page of the book whenever it was needed. There is also a couple of programs on television that did this. I also have some more mature programs that does the whole sound effects with scripting.

3. This has been done on the radio ( I forget the heros ) with the Golden Age comicbooks. Again I forget the characters. It was something like Dick Tracy but it was another name and had a femine heroine called the pussy cat something. Remember the famous Radio program that made people hide in there basements.

4. About Flash this is a FLASH toon AKA Power Point presentation. The whole purpose of Flash was to replace power point. The problem is that Power point has too many requirments while FLASH can be a self expanding exe file.

TRY AGAIN??

 

 

Great article, Tim!!!

Joe Zabel's picture

Great article, Tim!!!

But how about that dancing bug?

tymmi's picture

Is that cute, or what?

The Yellow Light comics and stuff.

 

 

Layouts and all

Neil Cohn's picture

Yes, the bug is pretty cute, I'll give you that. The essay was good. Regarding the layouts as pacing similar to music, this is essentially the reasoning that P. Craig Russell uses in translating operas to comic form. I'm sure he did it for Ring of the Neibelung at least, because he talks about it in the sketchbooks section.

You also tap into a theory I've never really worked to develop that much regarding layouts. Rather than panel sizes/shapes being indicators of time (which really makes no sense), they work instead to create a rhythmic pacing (think "visual intonation" in a sense).

Its not that a long stretched out panel creates a "longer narrative time" but it creates a "longer rhythmic unit" that is also subject to the internal composition of the image. Of course, we'd need to do some eyetracking experiments to prove any of this, but I think the results of such theorizing could lend to some interesting results in praxis, no?

---------------------
Studying the Visual Language of "Comics" - www.emaki.net

------- Studying the Visual Language of "Comics" - http://www.emaki.net

Why stop at music?

Why stop at music? If sound can be added, why not add voices too? And hey ... while we're at it - we only need to add extra panels and we could make it look as if the characters can move ...

Oh wait ... haven't Disney et al been doing that for 70 years or something?

Broken Voice Comics
Because comics are not just for kids

Website: www.brokenvoice.co.uk
Contact: edit_bvc@yahoo.co.uk

Broken Voice Comics
Because comics are not just for kids

i think you're missing the point

cayetano garza's picture

the point being, adding META to comics whilst still keeping them within the context of BEING comics.

animation and sound aren't exclusive to movies/film, in case you hadn't heard. ;)

year of the rat

Yeah, What Cat said.

tymmi's picture

And yours is the exact argument I'm trying to address in the article.

The Yellow Light comics and stuff.

 

 

Quote:It seems the closer we

Quote:
It seems the closer we get to true music/narrative integration the further we stray from comics.

And into territory already covered quite thoroughly long ago. Interactive storytelling with full soundtrack(often with full voice now) have existed for decades. Some aren't particularly interactive, they're just stories that are clicked through, told through a series of still images. They have a genre title: "Digital comics". It's not a rare secret genre or anything. Frank Miller is apparently working on a Metal Gear one for PSP, and there were plenty before he became involved with that projec. Granted the same "is it a comic?" questions can be applied to those as they can with webcomics. Some play up the panel by panel style of comic books. Others are pretty animated and come with mini games and other more video-gamey conventions.

I don't think that's terribly important though. If they're not comics, they're still worthwhile creative endeavors. When something slightly breaks away from what someone considers a conventional comic and into that territory, what they're doing ought to be compared to what has already been done.

<a xhref="http://www.kiwisbybeat.com" target=blank>Kiwis by beat!</a>

And that's not all, but...

tymmi's picture

There are a few other existing parallels you can draw. Video games themselves have been using comics-like (text and image), "click-through" interfaces for narrative exposition for decades - most times with music (or voice) included. This article focuses on the webcomic-specific aspects of form. I absolutely agree that it doesn't really matter if it's a "comic" or not (in anything but a discussion of comics form). And that's pretty much exactly what I said.

As for comparing these web experiments with what already exists...

I'm familiar with a few of these "digital comics." Most of what I've seen amounts to a click-through comic with music playing in the background and sometimes panel transition-timed sound effects. I haven't seen any kind of narrative-responsive, synched soundtrack (sound loops that shift and conform to specific points in the narrative). If you know of something like that I'd like to see it.

It's not really clear from the video of that Miller/Metal Gear thing how much of comics' reader interactivity it retains. What it shows is pretty similar to the Broken Saints technique (of course it's just a brief promotional video).

A lot of attempts to integrate comics and multimedia content end up like this, which is no longer comics (and pretty ugly to boot) in my opinion.

The Yellow Light comics and stuff.

 

 

I can't think of any

I can't think of any offhand, but I mostly play really old ones with novel concepts(dodgeball game played with menus and stuff like that). Some ambient sound scenes had a similar effect. I
I think a lot of it has to do with the scope of the games and the fact that they're not really working with really dynamic scenarios. They know at which point someone will reach a certain scene, and at that point is when the music shifts. Soundtracks where the music is tied directly to the action aren't as common as they used to be, so people are fine with simply having something that fits the general mood of a scene playing in the background.

...Actually, there was a part in Metal Slader Glory(NES) in the middle of the game, where you go to an abandoned...moon colony or something, to find a missing person. The BGM is just a droning noise while the characters explore it, and brief pieces of music are played to match key scenes. There are only a few, but as a matter of timing music with events I thought it worked pretty well. Albeit it's a pretty old game.

Looking at a few games, I also noticed there is a bit of jumping back and forth being comic and video styles of presentation depending on what was happening. I think that's a good approach. Playing up the strengths of both elements.

I agree that that Star Wars thing looks kind of sloppy. If enough people want to call that sort of thing a comic, it'll be one.Using what I would consider older definitions, none of these things would be, but then, "comic" once just referred to how strips were funny, right? So I try not to think about it too much.

Maybe people will end up giving these multimedia things a new title-

<a xhref="http://www.kiwisbybeat.com" target=blank>Kiwis by beat!</a>

Yirmumah teaches you Ukulele

djcoffman's picture

Yirmumah teaches you how to play ukulele starting here: http://www.yirmumah.net/archives.php?date=20050627