Algorithmic detection of crucial moments

[Peter Billam]

Greetings all :-)

A couple of months ago, I was at a performance where a band of
two horns, two electronic keyboards and I think a midi track,
accompanied a silent film (Die Abenteuer des Prinzen Achmeds).
The midi track seemed to react Instantly Instantly to events on
screen (eg: scene-changes); no delay and very reliably.
Of course that can be done by hand, but it seemed to me you could
do most of it with a simple algorithm that detected the frames
when the total amount by which the pixels changed was highest.
(It was mostly at these moments when the music changed to a new
section.) Then you could either use that:
* in real time to generate some event (a non-standard midi event?)
which could be used to start playing a new midi-file,
* in a semi-by-hand process, to genenerate a list of times
(eg 00:17:32.6 or 1052.6 etc) to which you then produce your
midi-track (but then in performance you have to start the video
and start the midi-track at exactly the same moment...)
* with some more arcane stuff using SMPTE codes ...

People must have tackled this detect-crucial-moments-in-a-video
problem, no ?

Then I got to thinking you could do the same if you were fitting
midi-tracks to a pre-existing audio-recording; you'd have to detect
big changes in dynamic level, and sudden changes in spectrum.
Algorithmically this could be harder than counting pixels in the
video case, but still it could be very useful.

Anyone know anything about this automatic-detection-of-crucial-
-moments problem ? Any software out there already ?

Regards, Peter

[Peter Billam]

Post by Peter Billam
People must have tackled this detect-crucial-moments-in-a-video
problem, no ?
Then I got to thinking you could do the same if you were fitting
midi-tracks to a pre-existing audio-recording; you'd have to detect
big changes in dynamic level, and sudden changes in spectrum.
Algorithmically this could be harder than counting pixels in the
video case, but still it could be very useful.
Anyone know anything about this automatic-detection-of-crucial-
-moments problem ? Any software out there already ?

Aha... well, for the audio problem, there is aubio :-)
apt-get install libaubio-dev libaubio-doc python-aubio
http://aubio.org/documentation

A few command line applications are provided:
aubioonset
extract musical onset times
aubiopitch
extract musical pitch
aubiotrack
outputs detected pitch
aubionotes
uses both onset and pitch to extract symbolic music data
from an audio source and emit MIDI like data.
aubioquiet
detects silence and noisy regions
aubiomfcc
compute Mel frequency Cepstrum coefficients
aubiocut
a python script that takes an input sound and creates one new sample
at each detected onset or beat. The slices produced by aubiocut are
useful for use with a sequencer such as Hydrogen.

There's a python interface already, which could serve as a model
for a Lua translation; but there is already aubionotes
http://aubio.org/manpages/latest/aubionotes.1.html
aubionotes can print pitches in midi, bin, cent, and Hz.
At first glance, if not reading a file it seems to need jack audio,
but still, interesting...
Looks like it's asking for an alsamidi client to eat aubionotes
output, and do something with it, not sure exactly what ... :-)

Regards, Peter