Sound in real time · notebook from class
One way to make sound is to compute a whole wave, then play it: write, run, listen. Here the sound card asks for the next few samples, again and again, and we answer while it plays. We can change things from a cell and hear them a moment later, as in a REPL.
In a local Jupyter this needs %pip install sounddevice. In the browser a stand-in with the same interface is next to this notebook: run the first cell and press ▶ audio.
Setup
import sounddevice as sd
import numpy as np
import asyncio
SR = 480001 · The sound card asks
An OutputStream calls a function of ours, the callback, whenever it needs samples. It passes outdata, an empty array to fill, and frames, how many samples it wants.
def callback(outdata, frames, time, status):
outdata[:, 0] = 0.1 * np.random.uniform(-1, 1, frames) # noise
out = sd.OutputStream(samplerate=SR, channels=1, callback=callback)out.start()out.stop()outdata has one row per sample and one column per channel. outdata[:, 0] is the whole first column: all the rows, column 0. A small array shows how this indexing works:
r = np.random.random((2, 3))
rr.shaper[:, 0] # the first columnr[0] # the first rowr[:, 0] = [10, 20] # a column takes as many values as there are rows
rtry:
r[:, 0] = [1, 2, 3]
except ValueError as e:
print(e)Two more things:
- write into
outdata, do not rebind it.outdata = ...only moves a local name: the sound card never sees it. The return value of the callback is ignored; - pass
channelsandsamplerate. Without them the stream uses the device's: often two channels, and the second column, never written, plays whatever was in memory.
2 · A sine, block by block
def sine_callback(outdata, frames, time, status):
t = np.arange(frames) / SR
outdata[:, 0] = 0.2 * np.sin(2 * np.pi * 440 * t)
out = sd.OutputStream(samplerate=SR, channels=1, blocksize=512, callback=sine_callback)
out.start()out.stop()It buzzes. Every block starts again from t = 0, so the sine jumps back to the start of its cycle every 512 samples: 48000 / 512 ≈ 94 times a second. A click that repeats 94 times a second is no longer a click: it is a low note.
With blocks of 48000 samples the jump comes once a second, and we hear it as a click:
out = sd.OutputStream(samplerate=SR, channels=1, blocksize=48000, callback=sine_callback)
out.start()out.stop()Bigger blocks are not the fix: a change made now would be heard up to a second later (the latency). The callback needs a memory.
3 · A sine that remembers
Memory between calls means state, and state calls for an object. A method can be the callback: s.callback remembers its s.
class Sine:
def __init__(self, f):
self.f = f
self.t = 0.0 # where we are, in seconds
def callback(self, outdata, frames, time, status):
t = self.t + np.arange(frames) / SR
outdata[:, 0] = 0.2 * np.sin(2 * np.pi * self.f * t)
self.t += frames / SR
s = Sine(440)
out = sd.OutputStream(samplerate=SR, channels=1, blocksize=512, callback=s.callback)
out.start()No more buzz. Now change the frequency while it plays:
for i in range(20):
s.f += 5 * i
await asyncio.sleep(0.5)out.stop()It clicks at every change. The phase is 2π · f · t, with t growing all the time: change f, and the phase of the whole past changes with it, so the wave jumps.
What the oscillator must remember is not the time but where it is in its cycle: the phase. At each block the phase grows by 2π · f · frames / SR, with the frequency of that block.
class Sine:
def __init__(self, f):
self.f = f
self.phase = 0.0
def callback(self, outdata, frames, time, status):
phases = self.phase + 2 * np.pi * self.f * np.arange(frames) / SR
outdata[:, 0] = 0.2 * np.sin(phases)
self.phase += 2 * np.pi * self.f * frames / SR
s = Sine(440)
out = sd.OutputStream(samplerate=SR, channels=1, blocksize=512, callback=s.callback)
out.start()for i in range(20):
s.f += 5 * i
await asyncio.sleep(0.5)out.stop()await asyncio.sleep(...) waits without blocking the sound. It works in a local Jupyter and in the browser, where time.sleep would stop the audio too.
4 · Sources
The sine fills outdata itself, so it can only play alone. We split the two jobs:
- the sine gives its next samples:
block(frames); stream_ofbuilds a stream whose callback writes them intooutdata.
class Sine:
def __init__(self, f):
self.f = f
self.phase = 0.0
def block(self, frames):
phases = self.phase + 2 * np.pi * self.f * np.arange(frames) / SR
self.phase += 2 * np.pi * self.f * frames / SR
return 0.2 * np.sin(phases)
def stream_of(source):
def callback(outdata, frames, time, status):
outdata[:, 0] = source.block(frames)
return sd.OutputStream(samplerate=SR, channels=1, blocksize=512, callback=callback)s = Sine(440)
out = stream_of(s)
out.start()s.f = 600out.stop()Inside callback, the name must be source, the parameter. With a global name such as s it would play s whatever we pass, without any error.
stream_of uses only one thing of the object: its block method. Anything with block(frames) is a source. It is not a class but a family of classes: a duck type. Python can give it a name with a Protocol. A square wave is another source:
class Square:
def __init__(self, f):
self.f = f
self.phase = 0.0
def block(self, frames):
phases = self.phase + 2 * np.pi * self.f * np.arange(frames) / SR
self.phase += 2 * np.pi * self.f * frames / SR
return 0.1 * np.sign(np.sin(phases)) # +1 where the sine is positive, -1 where it is negativesq = Square(440)
out = stream_of(sq)
out.start()sq.f = 100out.stop()5 · A mixer
A mixer holds a list of sources and sums their blocks. It has a block method too, so a mixer is a source: it can go into stream_of, or into another mixer.
class Mixer:
def __init__(self, sources):
self.sources = sources
def block(self, frames):
out = np.zeros(frames)
for source in self.sources:
out += source.block(frames)
return outs1 = Sine(400)
s2 = Sine(500)
m = Mixer([s1, s2])
out = stream_of(m)
out.start()s1.f = 300s2.f = 301 # 1 Hz apart: they beatout.stop()6 · From clips to sources
The tools for clips, scales and instruments:
from IPython.display import Audio
from dataclasses import dataclass
def play(wave):
return Audio(wave, rate=SR, normalize=False, autoplay=False)
def pluck(f, dur_s=1.0, tau=0.3, amplitude=0.25):
t = np.arange(round(dur_s * SR)) / SR
env = np.exp(-t / tau)
wave = np.sin(2 * np.pi * f * t)
return amplitude * env * wave
def mix_sounds(sounds): # a sound is (onset_s, wave)
dur_s = max([onset_s + len(wave) / SR for onset_s, wave in sounds])
out = np.zeros(round(dur_s * SR))
for onset_s, wave in sounds:
onset = round(onset_s * SR)
out[onset:onset + len(wave)] += wave
return out
note_names = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"]
class Pitch:
def __init__(self, value):
if isinstance(value, str): # a name like "C4": the octave last, the note before
value = 12 * (int(value[-1]) + 1) + note_names.index(value[:-1])
self.midi = value
@property
def f(self): # the frequency in Hz
return 440 * 2 ** ((self.midi - 69) / 12)
def __add__(self, semitones):
if isinstance(semitones, Pitch):
raise TypeError("Add semitones, not a pitch.")
return Pitch(self.midi + semitones)
def __sub__(self, other):
if isinstance(other, Pitch):
return self.midi - other.midi # an interval
return Pitch(self.midi - other) # a lower pitch
def __eq__(self, other):
return isinstance(other, Pitch) and self.midi == other.midi
def __repr__(self):
return note_names[self.midi % 12] + str(self.midi // 12 - 1)
@dataclass
class Clip:
events: list # (onset, data): onset in beats
length: float # in beats: where the next clip starts
def __rshift__(self, other): # then
moved = [(onset + self.length, data) for onset, data in other.events]
return Clip(self.events + moved, self.length + other.length)
def __mul__(self, times): # again
out = Clip([], 0)
for _ in range(times):
out = out >> self
return out
def __add__(self, n):
return Clip([(onset, data + n) for onset, data in self.events], self.length)
def __or__(self, other): # together
return Clip(self.events + other.events, max(self.length, other.length))
def note(data, length=1):
return Clip([(0, data)], length)
def rest(length):
return Clip([], length)
class Scale:
def __init__(self, steps, root):
self.root = root
self.distances = []
distance = 0
for step in steps:
self.distances.append(distance)
distance += step
def __getitem__(self, degree):
octave = degree // len(self.distances)
step = degree % len(self.distances)
return self.root + (12 * octave) + self.distances[step]
def play_clip(clip, scale, beat_s=0.5):
sounds = [(onset * beat_s, pluck(scale[data].f)) for onset, data in clip.events]
return play(mix_sounds(sounds))play_clip goes from a clip to a sound, a whole wave. We keep that step on its own:
def render_clip(clip, scale, beat_s=0.5):
sounds = [(onset * beat_s, pluck(scale[data].f)) for onset, data in clip.events]
return mix_sounds(sounds)
a_major = Scale([2, 2, 1, 2, 2, 2, 1], Pitch("A4"))
call = note(0) >> note(1) >> note(2) >> note(0)
sound = render_clip(call, a_major)A wave is an array: it has no block, so it is not a source. We need an adapter that takes a wave and gives it out one block at a time. We call it a voice, as in a sampler: it remembers how far it has got.
class Voice:
def __init__(self, wave):
self.wave = wave
self.i = 0 # how far we have got
def block(self, frames):
out = np.zeros(frames)
chunk = self.wave[self.i : self.i + frames] # shorter at the end, then empty
out[:len(chunk)] = chunk
self.i += frames
return outA slice past the end of an array is not an error: it is shorter, or empty. So the last block is the end of the wave and then zeros, and after that only zeros.
np.arange(10)[8:13], np.arange(10)[20:25]v = Voice(render_clip(call, a_major))
out = stream_of(v)
out.start()v.i = 0 # back to the startv.wave = render_clip(note(1) >> note(2) >> note(1) >> note(0), a_major)
v.i = 0out.stop()7 · Playing into a mixer
Start a stream on an empty mixer: silence. Then add sources while it plays.
m = Mixer([])
out = stream_of(m)
out.start()m.sources.append(Voice(pluck(Pitch("A4").f))) # run it again and againfor degree in [0, 2, 4, 7]:
m.sources.append(Voice(pluck(a_major[degree].f, dur_s=2))) # a chordlen(m.sources)The notes are over, but the voices are still in the list: the mixer asks each of them for a block of zeros, every block. Add enough and the callback can no longer finish in time: the sound card gets no samples, and we hear cracks (an underflow).
out.stop()A voice can say when it is over: done, a property. The mixer takes finished sources out of its list. It loops over a copy, list(self.sources): removing from the list it loops over would skip elements. Sources without done, such as Sine, play forever.
class Voice:
def __init__(self, wave):
self.wave = wave
self.i = 0
def block(self, frames):
out = np.zeros(frames)
chunk = self.wave[self.i : self.i + frames]
out[:len(chunk)] = chunk
self.i += frames
return out
@property
def done(self):
return self.i >= len(self.wave)
class Mixer:
def __init__(self, sources):
self.sources = sources
def block(self, frames):
out = np.zeros(frames)
for source in list(self.sources):
out += source.block(frames)
if getattr(source, "done", False): # False for sources without done
self.sources.remove(source)
return outm = Mixer([])
out = stream_of(m)
out.start()for degree in [0, 2, 4, 7]:
m.sources.append(Voice(pluck(a_major[degree].f, dur_s=2)))len(m.sources) # run it again after two secondsout.stop()8 · A chiptune arpeggio
One square wave, jumping between the notes of a chord sixteen times a second: the ear hears the chord. Old video game chips had three or four voices, and played chords this way.
[0, 2, 4, 7] are degrees of the scale: the triad (first, third, fifth) and the root an octave up. The four chords are I, vi, IV and V.
sq = Square(440)
out = stream_of(sq)
out.start()
chords = [[0, 2, 4, 7], [5, 7, 9, 12], [3, 5, 7, 10], [4, 6, 8, 11]]
for chord in chords * 2:
for _ in range(4):
for degree in chord:
sq.f = a_major[degree].f
await asyncio.sleep(0.06)
out.stop()Try [0, 2, 4, 6]: the fourth note becomes the seventh of the scale, and the colour changes.