Notebook from class

Download .ipynb Open in the browser

“Open in the browser” runs the notebook on your computer with JupyterLite: no installation, no account. Your changes stay in that browser: download the notebook to keep them.

Sound in real time · notebook from class

One way to make sound is to compute a whole wave, then play it: write, run, listen. Here the sound card asks for the next few samples, again and again, and we answer while it plays. We can change things from a cell and hear them a moment later, as in a REPL.

In a local Jupyter this needs %pip install sounddevice. In the browser a stand-in with the same interface is next to this notebook: run the first cell and press ▶ audio.

Setup

import sounddevice as sd
import numpy as np
import asyncio

SR = 48000

1 · The sound card asks

An OutputStream calls a function of ours, the callback, whenever it needs samples. It passes outdata, an empty array to fill, and frames, how many samples it wants.

def callback(outdata, frames, time, status):
    outdata[:, 0] = 0.1 * np.random.uniform(-1, 1, frames)      # noise

out = sd.OutputStream(samplerate=SR, channels=1, callback=callback)
out.start()
out.stop()

outdata has one row per sample and one column per channel. outdata[:, 0] is the whole first column: all the rows, column 0. A small array shows how this indexing works:

r = np.random.random((2, 3))
r
r.shape
r[:, 0]               # the first column
r[0]                  # the first row
r[:, 0] = [10, 20]    # a column takes as many values as there are rows
r
try:
    r[:, 0] = [1, 2, 3]
except ValueError as e:
    print(e)

Two more things:

  • write into outdata, do not rebind it. outdata = ... only moves a local name: the sound card never sees it. The return value of the callback is ignored;
  • pass channels and samplerate. Without them the stream uses the device's: often two channels, and the second column, never written, plays whatever was in memory.

2 · A sine, block by block

def sine_callback(outdata, frames, time, status):
    t = np.arange(frames) / SR
    outdata[:, 0] = 0.2 * np.sin(2 * np.pi * 440 * t)

out = sd.OutputStream(samplerate=SR, channels=1, blocksize=512, callback=sine_callback)
out.start()
out.stop()

It buzzes. Every block starts again from t = 0, so the sine jumps back to the start of its cycle every 512 samples: 48000 / 512 ≈ 94 times a second. A click that repeats 94 times a second is no longer a click: it is a low note.

With blocks of 48000 samples the jump comes once a second, and we hear it as a click:

out = sd.OutputStream(samplerate=SR, channels=1, blocksize=48000, callback=sine_callback)
out.start()
out.stop()

Bigger blocks are not the fix: a change made now would be heard up to a second later (the latency). The callback needs a memory.

3 · A sine that remembers

Memory between calls means state, and state calls for an object. A method can be the callback: s.callback remembers its s.

class Sine:
    def __init__(self, f):
        self.f = f
        self.t = 0.0                         # where we are, in seconds

    def callback(self, outdata, frames, time, status):
        t = self.t + np.arange(frames) / SR
        outdata[:, 0] = 0.2 * np.sin(2 * np.pi * self.f * t)
        self.t += frames / SR

s = Sine(440)
out = sd.OutputStream(samplerate=SR, channels=1, blocksize=512, callback=s.callback)
out.start()

No more buzz. Now change the frequency while it plays:

for i in range(20):
    s.f += 5 * i
    await asyncio.sleep(0.5)
out.stop()

It clicks at every change. The phase is 2π · f · t, with t growing all the time: change f, and the phase of the whole past changes with it, so the wave jumps.

What the oscillator must remember is not the time but where it is in its cycle: the phase. At each block the phase grows by 2π · f · frames / SR, with the frequency of that block.

class Sine:
    def __init__(self, f):
        self.f = f
        self.phase = 0.0

    def callback(self, outdata, frames, time, status):
        phases = self.phase + 2 * np.pi * self.f * np.arange(frames) / SR
        outdata[:, 0] = 0.2 * np.sin(phases)
        self.phase += 2 * np.pi * self.f * frames / SR

s = Sine(440)
out = sd.OutputStream(samplerate=SR, channels=1, blocksize=512, callback=s.callback)
out.start()
for i in range(20):
    s.f += 5 * i
    await asyncio.sleep(0.5)
out.stop()

await asyncio.sleep(...) waits without blocking the sound. It works in a local Jupyter and in the browser, where time.sleep would stop the audio too.

4 · Sources

The sine fills outdata itself, so it can only play alone. We split the two jobs:

  • the sine gives its next samples: block(frames);
  • stream_of builds a stream whose callback writes them into outdata.
class Sine:
    def __init__(self, f):
        self.f = f
        self.phase = 0.0

    def block(self, frames):
        phases = self.phase + 2 * np.pi * self.f * np.arange(frames) / SR
        self.phase += 2 * np.pi * self.f * frames / SR
        return 0.2 * np.sin(phases)

def stream_of(source):
    def callback(outdata, frames, time, status):
        outdata[:, 0] = source.block(frames)
    return sd.OutputStream(samplerate=SR, channels=1, blocksize=512, callback=callback)
s = Sine(440)
out = stream_of(s)
out.start()
s.f = 600
out.stop()

Inside callback, the name must be source, the parameter. With a global name such as s it would play s whatever we pass, without any error.

stream_of uses only one thing of the object: its block method. Anything with block(frames) is a source. It is not a class but a family of classes: a duck type. Python can give it a name with a Protocol. A square wave is another source:

class Square:
    def __init__(self, f):
        self.f = f
        self.phase = 0.0

    def block(self, frames):
        phases = self.phase + 2 * np.pi * self.f * np.arange(frames) / SR
        self.phase += 2 * np.pi * self.f * frames / SR
        return 0.1 * np.sign(np.sin(phases))        # +1 where the sine is positive, -1 where it is negative
sq = Square(440)
out = stream_of(sq)
out.start()
sq.f = 100
out.stop()

5 · A mixer

A mixer holds a list of sources and sums their blocks. It has a block method too, so a mixer is a source: it can go into stream_of, or into another mixer.

class Mixer:
    def __init__(self, sources):
        self.sources = sources

    def block(self, frames):
        out = np.zeros(frames)
        for source in self.sources:
            out += source.block(frames)
        return out
s1 = Sine(400)
s2 = Sine(500)
m = Mixer([s1, s2])
out = stream_of(m)
out.start()
s1.f = 300
s2.f = 301        # 1 Hz apart: they beat
out.stop()

6 · From clips to sources

The tools for clips, scales and instruments:

from IPython.display import Audio
from dataclasses import dataclass

def play(wave):
    return Audio(wave, rate=SR, normalize=False, autoplay=False)

def pluck(f, dur_s=1.0, tau=0.3, amplitude=0.25):
    t = np.arange(round(dur_s * SR)) / SR
    env = np.exp(-t / tau)
    wave = np.sin(2 * np.pi * f * t)
    return amplitude * env * wave

def mix_sounds(sounds):                      # a sound is (onset_s, wave)
    dur_s = max([onset_s + len(wave) / SR for onset_s, wave in sounds])
    out = np.zeros(round(dur_s * SR))
    for onset_s, wave in sounds:
        onset = round(onset_s * SR)
        out[onset:onset + len(wave)] += wave
    return out

note_names = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"]

class Pitch:
    def __init__(self, value):
        if isinstance(value, str):           # a name like "C4": the octave last, the note before
            value = 12 * (int(value[-1]) + 1) + note_names.index(value[:-1])
        self.midi = value

    @property
    def f(self):                             # the frequency in Hz
        return 440 * 2 ** ((self.midi - 69) / 12)

    def __add__(self, semitones):
        if isinstance(semitones, Pitch):
            raise TypeError("Add semitones, not a pitch.")
        return Pitch(self.midi + semitones)

    def __sub__(self, other):
        if isinstance(other, Pitch):
            return self.midi - other.midi    # an interval
        return Pitch(self.midi - other)      # a lower pitch

    def __eq__(self, other):
        return isinstance(other, Pitch) and self.midi == other.midi

    def __repr__(self):
        return note_names[self.midi % 12] + str(self.midi // 12 - 1)

@dataclass
class Clip:
    events: list      # (onset, data): onset in beats
    length: float     # in beats: where the next clip starts

    def __rshift__(self, other):          # then
        moved = [(onset + self.length, data) for onset, data in other.events]
        return Clip(self.events + moved, self.length + other.length)

    def __mul__(self, times):             # again
        out = Clip([], 0)
        for _ in range(times):
            out = out >> self
        return out

    def __add__(self, n):
        return Clip([(onset, data + n) for onset, data in self.events], self.length)

    def __or__(self, other):              # together
        return Clip(self.events + other.events, max(self.length, other.length))

def note(data, length=1):
    return Clip([(0, data)], length)

def rest(length):
    return Clip([], length)

class Scale:
    def __init__(self, steps, root):
        self.root = root
        self.distances = []
        distance = 0
        for step in steps:
            self.distances.append(distance)
            distance += step

    def __getitem__(self, degree):
        octave = degree // len(self.distances)
        step = degree % len(self.distances)
        return self.root + (12 * octave) + self.distances[step]

def play_clip(clip, scale, beat_s=0.5):
    sounds = [(onset * beat_s, pluck(scale[data].f)) for onset, data in clip.events]
    return play(mix_sounds(sounds))

play_clip goes from a clip to a sound, a whole wave. We keep that step on its own:

def render_clip(clip, scale, beat_s=0.5):
    sounds = [(onset * beat_s, pluck(scale[data].f)) for onset, data in clip.events]
    return mix_sounds(sounds)

a_major = Scale([2, 2, 1, 2, 2, 2, 1], Pitch("A4"))
call = note(0) >> note(1) >> note(2) >> note(0)
sound = render_clip(call, a_major)

A wave is an array: it has no block, so it is not a source. We need an adapter that takes a wave and gives it out one block at a time. We call it a voice, as in a sampler: it remembers how far it has got.

class Voice:
    def __init__(self, wave):
        self.wave = wave
        self.i = 0                           # how far we have got

    def block(self, frames):
        out = np.zeros(frames)
        chunk = self.wave[self.i : self.i + frames]     # shorter at the end, then empty
        out[:len(chunk)] = chunk
        self.i += frames
        return out

A slice past the end of an array is not an error: it is shorter, or empty. So the last block is the end of the wave and then zeros, and after that only zeros.

np.arange(10)[8:13], np.arange(10)[20:25]
v = Voice(render_clip(call, a_major))
out = stream_of(v)
out.start()
v.i = 0           # back to the start
v.wave = render_clip(note(1) >> note(2) >> note(1) >> note(0), a_major)
v.i = 0
out.stop()

7 · Playing into a mixer

Start a stream on an empty mixer: silence. Then add sources while it plays.

m = Mixer([])
out = stream_of(m)
out.start()
m.sources.append(Voice(pluck(Pitch("A4").f)))       # run it again and again
for degree in [0, 2, 4, 7]:
    m.sources.append(Voice(pluck(a_major[degree].f, dur_s=2)))     # a chord
len(m.sources)

The notes are over, but the voices are still in the list: the mixer asks each of them for a block of zeros, every block. Add enough and the callback can no longer finish in time: the sound card gets no samples, and we hear cracks (an underflow).

out.stop()

A voice can say when it is over: done, a property. The mixer takes finished sources out of its list. It loops over a copy, list(self.sources): removing from the list it loops over would skip elements. Sources without done, such as Sine, play forever.

class Voice:
    def __init__(self, wave):
        self.wave = wave
        self.i = 0

    def block(self, frames):
        out = np.zeros(frames)
        chunk = self.wave[self.i : self.i + frames]
        out[:len(chunk)] = chunk
        self.i += frames
        return out

    @property
    def done(self):
        return self.i >= len(self.wave)

class Mixer:
    def __init__(self, sources):
        self.sources = sources

    def block(self, frames):
        out = np.zeros(frames)
        for source in list(self.sources):
            out += source.block(frames)
            if getattr(source, "done", False):       # False for sources without done
                self.sources.remove(source)
        return out
m = Mixer([])
out = stream_of(m)
out.start()
for degree in [0, 2, 4, 7]:
    m.sources.append(Voice(pluck(a_major[degree].f, dur_s=2)))
len(m.sources)    # run it again after two seconds
out.stop()

8 · A chiptune arpeggio

One square wave, jumping between the notes of a chord sixteen times a second: the ear hears the chord. Old video game chips had three or four voices, and played chords this way.

[0, 2, 4, 7] are degrees of the scale: the triad (first, third, fifth) and the root an octave up. The four chords are I, vi, IV and V.

sq = Square(440)
out = stream_of(sq)
out.start()

chords = [[0, 2, 4, 7], [5, 7, 9, 12], [3, 5, 7, 10], [4, 6, 8, 11]]
for chord in chords * 2:
    for _ in range(4):
        for degree in chord:
            sq.f = a_major[degree].f
            await asyncio.sleep(0.06)
out.stop()

Try [0, 2, 4, 6]: the fourth note becomes the seventh of the scale, and the colour changes.