ARTS 2610: Designing Musical Games

Arts Department, Rensselaer Polytechnic Institute

Monday/Thursday, 12:00-1:50pm, VAST Lab, SAGE 2411

Instructor: Rob Hamilton, West Hall 305/6

e: hamilr4 [at] rpi [dot] edu

w: https://robertkhamilton.github.io/

Office Hours: TBD

COURSE SYLLABUS - Fall, 2024

PREREQUISITE

None

COURSE DESCRIPTION

One of the most exciting areas of music technology development is happening in the realm of gaming and interactive virtual space. Music and Sound Design play crucial roles in the design of gaming environments, narratives and flow. And as designers create ever more innovative game experiences featuring rich graphics, fast multiplayer networking and next-generation controllers, new techniques for creating immersive music and sound for games to complement and showcase these advances are not only possible but necessary.

This Studio class will explore cutting edge techniques for building interactive sound and music systems for games and 2D/3D rendered environments. To better understand the link between virtual space and sound, students will learn the basics of designing sound and composing music for interactive game spaces by designing and implementing rich musical games within the Unity and Unreal gaming engines. Coursework will require the ability and desire to code game logic and design game environments. Techniques for integrating sound and music within games including game-centric middleware tools like FMOD and WWISE, interactive sound synthesis and computer networking using Open Sound Control may be explored.

Working in teams or on their own, students will design their own music-rich game experience, compose music, design sound material, and implement their own playable interactive musical game experiences.

CATALOG DESCRIPTION

Students will explore the artistic role of music and sound in gaming by building their own interactive sound and music-rich games and 2D/3D rendered environments. Within the context of their own creative game projects, students will learn the basics of designing sound and composing music for interactive game spaces. Using workflow programming languages and software tools, students will program basic gaming interactions, link them to interactive audio software, and create a musical gaming experience.

LEARNING OUTCOMES

    Students who successfully complete this course will demonstrate...
  1. an understanding and appreciation of game design and musical creation through an awareness of the many disciplines underlying the field including: software design and programming, interaction design, listening skills, musical theory, musical acoustics, digital audio theory, and digital signal processing.
  2. basic technical facility in the areas of game development, audio recording, editing, sound synthesis, software development and post production.
  3. creativity and resourcefulness through the creation of musical game environments and composition of your own sonic projects

EVALUATION

Evaluation is based on the following:

CLASS PARTICIPATION

You will be required to present some of your assignments to the class, to show your work within the software environment you used to create it, and to engage the class in discussion of your work. When you are not presenting your own work, you need to be attentive to whoever is presenting, and to engage them in discussion of their work. Failure to participate in class will lower your grade.

ATTENDANCE

You must attend class to succeed in this course.

  1. Since much of the class is focused on listening to and discussing work in class, attendance is mandatory.
  2. ** More then two unexcused absences will affect your grade, detracting 1/2 grade each additional 2 unexcused absences. **
  3. Late arrivals are very disruptive - continued late arrival will affect your grade.
  4. It should go without saying but no use of mobile devices or personal computers during class time (except for as required by the coursework itself) is acceptable. Continued violations will be treated as an unexcused absence.

STATEMENT REGARDING ACADEMIC INTEGRITY

Collaboration between students in this course is strongly encouraged. Likewise, students are encouraged—indeed, to some extent required—to exchange ideas, opinions and information . You are also encouraged to help each other in the lab and with performance, production, and presentation of composition projects.

Plagiarism of any kind is in direct violation of University policy on Academic Dishonesty as defined in the Rensselaer Handbook, and penalties for plagiarism can be severe. In this class you will be expected to attribute due credit to the originator of any ideas, words, sounds, or music which you incorporate substantially into your own work. This applies particularly to citation of sources for sonic "samples" included in your compositions.

The use of automated technical aids (e.g. ChatBots, AI code plugins) is not strictly forbidden but if used MUST be documented in great detail and discussed with the Professor prior to use in a graded project/assignment.

Submission of any assignment that is in violation of this policy may result in a grade of F for the assignment in question. Violation of this policy will be reported, as defined in the Rensselaer Handbook

DISABILITY SERVICES FOR STUDENTS

Students requiring assistance are encouraged to contact Disability Services: http://doso.rpi.edu/dss to discuss any special accommodations or needs for this course.

OFFICE HOURS

Office Hours for Fall, 2024 will be TBD

TEXTBOOKS / REPOSITORIES / FOUNDATIONAL MATERIAL

Designing Sound, by Andy Farnell (book missing)

Musimathics (Vol. I), by Gareth Loy

Musimathics (Vol, II), by Gareth Loy

Theory and Techniques of Electronic Music, by Miller Puckette


 

FILE UPLOAD LINK (BOX)


 

COURSE SCHEDULE

The proposed course topics and schedule will be as follows (take note of project due dates!). Based on class progress and interests, this schedule is subject to change. Special topics, guest lectures, supplemental reading, listening and additional assignments to be announced.

Week 1:
Week 2:
Tuesday, 9/3

Intro music:
      Idioteque (Radiohead cover) by Amanda Palmer
      Idioteque by Radiohead
      Mild und Leise by Paul Lansky (sample source)

Lecture: Designing Musical Games :: Gaming Musical Design
The role of sound and music in games; filmic influences; spectra of interactivity

- Destiny Trailer: https://www.youtube.com/watch?v=VFh5ArG46_M
- "Hope for the Future" https://www.youtube.com/watch?v=163_C5UVU-I
- Destiny 2 musical lore: https://gamerant.com/destiny-2-paul-mccartney-hope-for-the-future-song-lore/
- Marty O'Donnell talks Destiny/McCartney: https://www.youtube.com/watch?v=xsmRNGrh7Vk

- Journey Trailer: https://www.youtube.com/watch?v=2FZkTXs3Td8
- Journey Grammy Nomination: Forbes
- Austin Wintory Interview (BAFTA): https://www.youtube.com/watch?v=WcuMLbKTgwc

- Portal: Still Alive by Jonathan Coulton

- Smule Magic Piano: https://apps.apple.com/us/app/magic-piano-by-smule/id421254504
- Smule Musical Scoring paper: Social Composition: Musical Data Systems for Expressive Mobile Music

- Zatoichi: The Blind Swordsman: https://www.imdb.com/title/tt0363226/

- Q3OSC (Quake 3) paper: Q3OSC: or How I Learned to Stop Worrying and Love the Game
- Nous Sommes Tous Fernando: https://ccrma.stanford.edu/~rob/portfolio/fernando/

Week 3:
Monday, 9/9

Frequency       Root       MIDI (0-127)       Pitch       Speed       Sharp(#)       Octave       Flat(b)       Timbre       Partial       Equal Temperament       C4, D, E, F, G...       Loudness       Amplitude       Harmonic       Volume       Fifth       Chromatic       Gain       Fourth       Diatonic       Note       Third       Consonance       Scale       Major       Dissonance       Interval       Minor       Cent       Mode       Perfect       Microtone       Chord       Sample       Period       Sample       44.1 kHz       Compact Disc       Sampling Rate       48 kHz       96 kHz       Bit Depth       Nyquist Theorem       Quantization       Aliasing/Foldback       Resolution       Distortion       Signal       Clipping       Noise       PCM

 
Intro music: Mortal Kombat!!!!!!!

The Art and Science of Foley
aka: MOR-TAAL--KOMMMM-BAAAAT-day

Video : Digital Foley: "Mortal Kombat Foley: The Sound of Violence"
Video: "The Hunger Games" & "Frozen" Foley Artists Turn the Sound of Junk into Miracles

Week 4:
Monday, 9/16

Intro Music: Journey Playthough (Spoiler Warning)

Playing Audio in Unity: Overview
        - AudioSource.PlayOneShot: Plays an AudioClip, and scales the AudioSource volume by volumeScale.
        - Play: Plays the clip.
        - PlayDelayed: Plays the clip with a delay specified in seconds. Users are advised to use this function instead of the old Play(delay) function that took a delay specified in samples relative to a reference rate of 44.1 kHz as an argument.
        - PlayOnGamepad: Enable playing of audio source though a specfic gamepad.
        - PlayScheduled: Plays the clip at a specific time on the absolute time-line that AudioSettings.dspTime reads from.
Week 5:
Monday, 9/23

NO CLASS

READING:
        - Chunity Paper
        - MIDI.CITI: Designing an Experience-oriented Musical Cityscape

Links:
        - Chunity on Unity Asset Store
        - Chunreal Git repository

Week 6:
Monday, 9/30

Intro Music:
        - U2 With or Without You @ The Sphere, Las Vegas (2023)
        - Brian Eno Music for Airports (1978)

Bloom by Brian Eno (app, 2008)
Polyfauna by Radiohead (app, 2014)
Biophilia by Bjork (app, 2011)

Spore: Will Wright and Brian Eno (2008)
GDC 2008: Kent Jolly on Procedural Music systems in Spore
Usage of PD in Spore and Darkspore (paper, 2011)
Spore audio team interview

Proteus by David Kanaga (2013)
Generative Music Systems: Leonard Paul's music from "Sim-Cell": youtube
Martin Brinkmann patches: http://www.martin-brinkmann.de/pd-patches.html
RJDJ Abstractions: https://github.com/rjdj/rjlib

GG Music (Soundcraft/Starcraft II)
Paper: Soundcraft: Transducing StarCraft 2 (2013)
ChucK: https://chuck.cs.princeton.edu/

Week 7:
Monday, 10/7

Intro Music: The Hand That Feeds (cover - chamber ensemble version)

Unity Third Person audio project: RPI Box download (1.1 GB)
- Snapshot Transition types: Linear, Smoothstep, Squared, SquareRoot, BrickwallStart, BrickwallEnd
- AudioMixer Snapshot Weighted transitions: AudioMixer.TransitionToSnapshots
(same Snapshot video above)

Unity Audio Tutorials:
Audio Random Container: tutorial (Project Download (1 GB)
Audio Timelines: Working with Audio Tracks in Timeline

Perpetual Alpha (?): Unity 23.2 alpha: Audio Random Container
Download other Unity Versions...

Timeline Signals: Asset, Emitter, Receiver Overview video

Assignment #2: Sample-based game audio scene (- DUE DATE, FRIDAY, OCTOBER 18, 11:59:59 pm EST )

Using your favorite samples, either from our in-class recording challenge or new samples you make yourself, modify a Roll-A-Ball project OR the First or Third-Person Starter Assets scenes OR some other cool scene that has no sound, with sound and MAKE IT YOUR OWN. You can modify and add functionality or assets to the scene as needed/desired to make the game do what you'd like it to.
This assignment is looking to see you flex both your tech/implementation chops (following the examples we've played with in class) and your creative chops, with special focus on how you make your project personal and engaging through the use of sound and music.

The success of your scene will revolve around whether or not your scene is sonically-coherent, i.e. all the sounds exist and belong in the same space. Coherence can be helped by creative use of reverberation in game, by careful editing of sounds in Audacity, or simply through artistic choices you make in your role as sound-designer.
In addition to the project itself, create a detailed "call-sheet" list of all the sounds you will use/are using in the project, including variations. List file-names, extensions, durations, and any notes that would be helpful or useful to your programming team*. Please include a short paragraph or description of the overall-zeitgeist of your scene, and why we should care.
Please include examples of the following Unity audio processes in your scene:

    * Simple Audio Source playback (multiple)
    * Reverb Zones
    * Mixer Groups with exposed parameters and plugins/effects
    * C# scripting of audio playback events
    * 3D sound/distance-based attenuation
    * Varying pitch, amplitude, etc.
    * Sounds triggered through collision
    * An awesome looping stereo background "musical" track
    * Timeline audio tracks with signal-driven game events
    * Emergent gameplay that instigates some kind of interesting sound/music scape
    * EXTRA-CREDIT: Make your background track a multi-channel "stem"-based playback example
    * EXTRA-CREDIT: Use Chunity with ChucK audio scripts connected to your scene
    * EXTRA-CREDIT: include MIDI file control and/or Microphone input

Please upload your entire project to the Box dropbox folder using the same naming conventions as Assignment #1, i.e:
HOUSEKEEPING: box.rpi.edu file submission folder
- You all received an invitation to a shared "DMG Assignment Dropbox" Box folder. - zip up all your files into one .zip archive
- use the following naming convention: "a" + assignment # + "_" + rcs username + ".zip" (e.g. "a2_hamilr4.zip")
- if files are too crazy large, you can break your submission into multiple .zip archives. For example if you have 3 large .zip files, each would be titled as: "a2_hamilr4_1_of_3.zip", etc.


* the programming team is a lie...
Week 8:
Monday, 10/14
NO CLASS (Indigenous Peoples' Day)
Week 9:
Monday, 10/21
Week 10:
Monday, 10/28
Week 11:
Monday, 11/4

Week 12:
Monday, 11/11
Week 13:
Monday, 11/18
Week 14:
Monday, 11/25
Week 15:
Monday, 12/2
Week 16:
Monday, 12/9
FINAL DAY OF CLASS
FINAL PROJECT SUBMISSION

TBD