Audio

Audio clips are <audio> elements on the composition's timeline, and Kinetic plays them in the preview and mixes their sound into a render. Markers and a tempo name the moments of a track, and level() and spectrum() read its loudness at any time.

Audio clips

An audio clip is an <audio> element with data-start inside the composition root. Kinetic plays the clips in the preview, mixes them into the file render() writes, and reads their levels for level() and spectrum().

<audio id="music" src="music.mp3" data-start="0" data-duration="20" data-volume="0.8" data-fade-in="1" data-fade-out="2"></audio>
<audio id="whoosh" src="whoosh.wav" data-start="intro - 0.3"></audio>

• A footage clip with data-has-audio="true" is an audio clip too, its sound read from the video file.
• An audio clip is no clip for transitions, so a sound timed from a cut never counts as the clip a data-clip-transition replaces.
• A data-start that names no element starts the clip at 0, and development builds warn (KN-113).

The file

The file is the element's src, or with <source> children the one the element chose, else the first. Kinetic fetches and decodes each file once, at readiness, at 48 kHz, for the preview, the render and the levels.

• The element's own load is a second download of the same file, which preload="none" on the <audio> saves.
• A file that fails to load or decode is KN-135 (Failure).
• A file on another origin needs CORS headers, and a page opened from a file cannot fetch its files at all, so serve the folder over http. KN-135's development message says which when the fetch fails.

Markers and tempo

A piece cut to music names its moments once, on the audio clip, in the file's own seconds. Those seconds go through the clip's data-start, data-media-start and data-playback-rate, so the beat grid and the markers stay on the song however it is trimmed.

<audio id="music" src="music.mp3" data-start="0"
       data-tempo="134.68" data-downbeat="10.712" data-meter="4"
       data-markers="{ drop: 14.28, stop: 48.05 }"></audio>

const { timeOf, tempo } = wildflower.kinetic;
timeOf('stop');          // composition seconds; also 'stop + 0.4', 'intro' (when #intro ends) or 2.5
const grid = tempo();    // the one clip with data-tempo, or tempo('music')
grid.beat;               // a beat's length in composition seconds, at the clip's rate
grid.at(4);              // the time of beat 4, counting the downbeat as 0
grid.of(this.time);      // the beat now; Math.floor(grid.of(t) / grid.meter) is the bar

timeOf() takes a number, a name, or a name plus or minus seconds, as data-start does.

• A name is a marker, else an element's id, which is when that element ends, and 'intro.start' is when it starts.
• A time before 0 reads as 0.
• In data-start itself a marker names nothing, and a name is still an element's id.

tempo() returns { bpm, beat, meter, at(n), of(t) }, and at() takes fractions and negatives, so at(n + 0.25) is a sixteenth after beat n. Both functions read the page on every call.

The tempo is the default for steady music; see Which to use for when a beat file fits instead.

A step's wait stays seconds into the step, so a hold on a marker subtracts the step's start.

const { timeOf } = wildflower.kinetic;
wildflower.kinetic.step('drop', {
    wait: timeOf('stop') + 1.7 - timeOf('intro')   // holds 1.7 s after the 'stop' marker, for a step that starts when intro ends
});

Each of these is KN-100 on every build.

• An expression that is no time, or a name that is neither a marker nor an id.
• A data-markers that is no object of numbers, or does not read, with the character where reading stopped in the message.
• tempo() with no clip, or with two and no id.
• A data-tempo, data-downbeat or data-meter that is missing or wrong, with the clip and the attribute in the message.

Finding the tempo with beatgrid.cjs

The package's scripts/beatgrid.cjs reads a track and prints the values, with the attributes to paste. It runs in Node, and needs ffmpeg for anything but WAV.

• It gives the tempo, the meter and the first downbeat, with how sure it is of each.
• It lists the bar lines where the music changes section, with the change in dB at each.

node node_modules/@wildflowerjs/kinetic/scripts/beatgrid.cjs track.mp3          # --json gives the same as JSON

Which tempo the music is counted in is a matter of feel, so the script can add a second reading.

• When the half beats hit nearly as hard as the beats, it adds a double-time reading beside the first.
• When every other beat hits clearly softer and half the tempo is the runner-up, it adds a half-time reading.
• Noise or a held tone has no pulse to find, so all three readings are low on it, and a silent file is an error.

Sound in the preview

Kinetic plays the clips through Web Audio on its own clock.

• While the clock plays, every clip that has not ended starts so that the sound of a frame is heard while the frame is on screen. A clip added or removed while it plays is heard, or stops, after the next ready(), with no seek.
• Kinetic reads the time the output device is playing from getOutputTimestamp(), schedules each start 0.1 s ahead, and allows one 60 Hz display frame from the animation frame to the screen. When the time on screen and the time heard are more than 45 ms apart, after a seek, the loop back to 0 or a stall, every clip starts again from the time on screen.
• A pause stops every source.
• While the clock is paused, a seek such as a frame step or a scrub plays the sound of that frame alone, from the frame's time for one frame, with a 3 ms ramp at each end, so stepping finds the frame a sound starts on. A seek to the time already shown plays nothing.
• The seeks that check() and render() make play nothing, and neither does the frame the clock holds at the end of play({ to }) or with loop off.

A browser starts sound only after a gesture on the page, so the audio context starts suspended.

• When the composition has an audio clip, the preview bar has a speaker button, off until pressed, then on and off. Pressing it, or the bar's play button, resumes the context.
• Without the bar, Kinetic puts a speaker button of its own over the stage's lower right corner, outside the composition, so no render shows it. The sound stays off until it is pressed, and it turns the sound on and off without pausing, while a click on the picture pauses. It shows for 2 seconds as the page starts, then while the pointer is over the stage and for 3 seconds after a touch, and left alone it always fades, so a page running unattended in a hall or a talk shows nothing over the picture.

config({ sound: false }) plays nothing in the preview and suspends the context, and config({ sound: true }) resumes it. Sound changes nothing in the DOM, so check() is unaffected.

Sound in a render

When the composition has an audio clip, render() writes a sound track beside the video, mixed over the render's range, from to to. The options are in the render table.

await wildflower.kinetic.render({ from: 2, to: 8 });                      // AAC-LC sound track
await wildflower.kinetic.render({ from: 2, to: 8, audioCodec: 'opus' });  // Opus sound track

Kinetic mixes with OfflineAudioContext at 48 kHz in stereo, in segments of 10 s, faster than real time.

• A clip's start and length are whole samples from composition seconds, so the mix is the same on every machine.
• The segments join exactly at a data-playback-rate of 1, and at another rate a join is off by less than 1e-3 of full scale, below -60 dB.

The track is AAC-LC by default, which QuickTime, Safari and other players read, or Opus with audioCodec: 'opus'. When the browser cannot encode AAC, the default falls back to Opus, and development builds warn once (KN-202).

• The AAC encoder puts priming samples before the sound, which Kinetic measures for each render and the file's edit list skips, so the first sample heard is the one at from.
• An Opus track carries its pre-skip in the dOps box alone, since Chrome applies both that and an edit list's start. Kinetic encodes the pre-skip's length past to, so the sound lasts the whole render.
• Chrome's own decode of an Opus track ends on a whole 20 ms packet, so it can run on by less than one packet.
• A delay Kinetic cannot measure is KN-203.

level(t, band, options) and spectrum(t, bins)

const { level, spectrum } = wildflower.kinetic;
level(t, 'bass');                        // 0 to 1: the bass's level at composition time t
level(t, 'all', { clip: 'music' });      // one clip alone, by its id
spectrum(t, 16);                         // 16 levels, 0 to 1, low to high

level() returns 0 to 1 for a band at composition time t, read at the nearest frame, and options.clip reads one audio clip alone, by its id.

spectrum(t, bins) returns bins levels from 20 Hz to 16 kHz on a logarithmic scale, low to high, with bins a whole number from 1 to 64, 16 by default. Each of fewer than 64 bins is the mean of the levels of the 64-bin scale it covers.

The levels are those of the mix the viewer hears, with each clip's start, trim, volume and fades.

• For each frame, a 2048-sample Hann window centered on the frame's time goes through an FFT in JavaScript, so every machine computes the same numbers.
• Each band is mapped from -60 dB to its loudest frame in the composition onto 0 to 1, and a level falls over about 0.15 s rather than in one frame.
• The tables are built once, at readiness, after the decode, so a seek reads a table and frame 0 waits for them.
• A computed that calls level() or spectrum() runs again when the tables are built, through the clock's audio field.

A band, a bins or a clip that is not one of the above throws (KN-100).

See Which to use for the levels beside the beat grids and the markers.