Current Project Updates

The following page includes detailed project updates presented in chronological order from most recent to most distant. (Project notes by Ellery Wolfe.)

09/27/20

I've been feeding more example pieces into the new harmonic analyzer and it has been working surprisingly well. There are some new things in each example that trip up the routine but these were mostly minor issues that were easy to fix.

The big issue that was giving me problems was pickup notes (again!) The rest before a pickup always occurs in note off mode along with the time sig, key sig, and tempo stuff. But the delay that indicates the rest doesn't occur in the same place every time. Compare the scores for "Greensleeves" versus "When the Saints." In one the tempo marking happens right at the beginning and in the other it doesn't happen until the first actual note. This rearranges where the pickup rest occurs in the MIDI info. Also there are often other bytes that look very similar to the delay times (which is a variable length quantity). So there were a lot of false positives happening for identifying pickup rests. I think I finally have the conditions correct.

Here is the first analysis for "Silent Night." Each chord is a quarter note:

CHORDS NO INVERSIONS:
I5,I5,I5,I,I,I,I5,I5,I5,I,iii5,vi5,V5,V5,V5,V,V,V,I,I,I,I5,I5,I5,IV,IV,
IV,IV5,IV5,IV,I5,I5,I5,I,I,I,IV,IV,IV,IV5,IV5,IV,I5,I5,I5,I,iii5,vi5,
V5,V5,V5,V7,V5,V,I,I,I,I,I,I,I,I5,I,V,V,V5,I,I,I,
CHORDS INVERSIONS:
I5,I5,I5,I,I,I,I5,I5,I5,I,iii5(inv),vi5,V5,V5,V5,V,V,V,I,I,I,I5,I5,I5,
IV,IV,IV,IV5,IV5,IV,I5,I5,I5,I,I,I,IV,IV,IV,IV5,IV5,IV,I5,I5,I5,I,
iii5(inv),vi5,V5,V5,V5,V7,V5,V,I,I,I,I,I,I,I,I5,I,V,V,V5,I,I,I,

There were some issues here. First of all, it identified diads (power chords) all over the place. Since this is a two-voice arrangement, the fact that the third isn't always present isn't really significant. Also most of the time there is only one chord per bar, so that output has a lot of redundant information and is really hard fora human to read. Here's the same output after fixes:

CHORDS NO INVERSIONS:
I,I,I,I,iii,vi,V,V,I,I,IV,IV,I,I,IV,IV,I,I,iii,vi,V,V7,V,I,I,I,V,I,
CHORDS INVERSIONS:
I,I,I,I,iii64,vi,V,V,I,I,IV,IV,I,I,IV,IV,I,I,iii64,vi,V,V7,V,I,I,I,V,I,

There is also a list of where each of these chords starts at, so the program still keeps track of all relevant information. Now the info is fairly easy for a human to follow while still keeping the beat-by-beat analysis.

Here is "When the Saints"

CHORDS NO INVERSIONS:
rest,no chord,I,V,I,I,V7,I,V,I,I,no chord,I,I,I,I,V,V7,V,I,V,I,I,I,I,
IVsus9,IV,IV,IV7,IV,V,I,I,V,I,I,I,
CHORDS INVERSIONS:
rest,no chord,I,V,I,I64,V7,I,V,I,I6,no chord,I,I64,I,I6,V,V42,V64,I64,
V,I,I64,I,I6,IVsus9,IV,IV,IV7,IV,V,K64,K64,V,I,I64,I,

"Greensleeves"

CHORDS NO INVERSIONS:
rest,no chord,i,iv,i,VII,VII,VI,VI,v,V,V,i,i,iv,i,VII,VII,VI,V,i,i,i,
III,III,VII,VII,i,i,v,V,V,III,III,VII,VII,VI,V,i,i,i,
CHORDS INVERSIONS:
rest,no chord,i,iv64,i,VII,VII,VI,VI,v,V,V,i64,i,iv64,i,VII,VII,VI,V,
I,i,i,III,III,VII,VII,i,i,v,V,V,III,III,VII,VII,VI,V,I,i,i,

"We Wish You a Merry Christmas"

CHORDS NO INVERSIONS:
rest,no chord,I,IV,ii,V,V/vi,vi,IV,iii,IV,ii,V,I,V,I,V7,V7sus4,V7,ii,I,ii,iii,Isus9,I,I,ii,V,I,
CHORDS INVERSIONS:
rest,no chord,I,IV64,ii,V,V6/vi,vi,IV,iii,IV,ii6,V64,I,V,I,V42,V7sus4(43),V43,ii,I,ii,iii,Isus9(6),I6,I,ii,V,I,

I'm gradually feeding it more and more complicated arrangements so next will be some pieces with four or five simultaneous voices, more sus chords, more accidentals, etc.

greensleeves.pdf | saints.pdf | silent.pdf | we wish.pdf

09/03/20

So I've been working on making a routine that will make a harmonic analysis with ANY arrangement. My old method only worked on 4-part SATB. The old method basically just had an insanely long "library" of chords including inversions, and it would grab the right one based on the scale degrees of the notes it saw. The new method is a lot better because it uses a bunch of smaller helper functions to figure out what is going on. This allows for unusual chords to be ID'd without me having to make a specific entry for it in the library.

So how does it work? When it reaches the end of a stack of notes (if any note changes, there is a new stack that has to be analyzed) it looks at the scale degrees. The stack can be one note, two notes, ten notes, whatever. Then it looks for perfect fifths among the intervals. If a perfect fifth is detected, that is a pretty good indication of what the root note is. Of course, diminished chords needed a different method for finding out the root since there are no perfect fifths. Also some chords, like minor 7 chords, contain two perfect fifth intervals and needed a different method.

Once the root is identified, you can use that as a reference to figure out what type of chord it is: major triad, minor seventh, secondary dominant, etc. You can also figure out the inversion.

So basically it figures out the chord organically by going through a series of steps rather than having to reference a massive list of all possible chords. And that is an analysis of just ONE stack of notes. If multiple stacks happen within one beat, those analyses then get combined into just one option, which is the best analysis for that beat.

Of course, sometimes a beat cannot be analyzed as a standard chord. Once all the normal chords have been figured out, it then goes back to the unidentified stacks and compares them to the following stack. If they are mostly similar, it figures out what the odd note(s) actually was and can use it to figure out sus and extended chords. Pretty cool.

These are the results for "Happy Birthday." Each chord corresponds to a quarter note.

CHORDS NO INVERSIONS:
rest,rest,no chord,vi,I,I,V7,V7,V7,V9,V7,V7,I,I,I,I,I,I,IVsus4,IV,IV5,I,I,V7,I,I,I,
CHORDS INVERSIONS:
rest,rest,no chord,vi6,I,I,V7,V7,V7,V9,V7,V7,I,I,I,I,I,I,IVsus464,IV64,IV5(inv),I,I,V7,I,I,I,

It correctly identified the V9 and IVsus4 based on their similarity to the following chords. During the first pass, these stacks were simply listed as unidentified.

happy birthday.pdf

07/13/20

-Most of the functions worked on the assumption that a piece would have only one key and time signature. This works for almost all simple music (which is what it had been looking at so far) but obviously key changes and time sig changes are not that uncommon. A big issue this caused was harmonic analysis now needs to work on blocks of key and time signatures COMBINED. For example, let's say there is a piece that is all in C but it changes from 4/4 to 6/8 partway through. The analysis has to run twice, once for the part in 4/4 and once for the part in 6/8 (or else it would analyze the 6/8 part by quarter note, when dotted quarters would be best). Each time sig/key sig chunk is now analyzed as its own thing.

-Getting the program to recognize weird durations, like 32nd notes, tuplets, or staccato notes. There is an infinite amount of durations that are technically possible in music. And there is no way to make the program account for every single weird-ass possibility (like an 11-tuplet or a quarter note tied to a 256th note, etc). I could keep adding these durations in, but at a certain point it is just not worth it since they are incredibly rare. The program can now deal with any note down to a 32nd note resolution and any kind of triplet, which covers like 99.9% of music.

-My harmonic analysis used to only work on 4-part SATB music. Ideally it should work with any number of voices. It now has a routine that figures out the range of the number of voices in a piece. A 4-part choral piece will have 4 voices throughout, making analysis fairly easy, but a piano piece will have a constantly changing number of simultaneous voices depending on how many fingers are pressing down at any given time. This uncertainty in how dense the voicing can be makes harmonic analysis a very tricky problem.

-To deal with this, the harmonic analysis has to take into account which notes happen on the downbeat, which notes happen on other (non-downbeat) strong beats, which notes are weak beats, which notes are held over from previous beats, which notes are accidentals, and which note (or notes) are the lowest for that beat. It is very complicated and this new method is still not complete, but these are all elements that are important for getting a solid harmonic analysis for any type of instrumentation.

04/10/20

-The phrase guesser now works a lot differently. Now it identifies phrase endings based on where they actually happen rather than rounding to the nearest barline. If a piece starts with a pickup, then it is normal for the phrases to NOT start on the barline, but rather in the same position as where the pickup begins. A piece that starts on beat 3 will normally have phrases start on beat 3 as well.

-The phrase guesser now counts up all the long note durations and then makes an educated guess about which note length (or lengths) are indicative of a phrase ending. Before the phrase guesser made a lot of assumptions about where phrase endings were but this method seems a lot more precise. For example, in Amazing Grace there are a whole lot of half notes, so this meant that using half notes to look for phrase endings is a terrible idea since it will ID a phrase every single bar. But in a piece with mostly quarter and eighth notes, half notes will often be the giveaway that there is a phrase ending. It all depends on the piece.

-One problem with the harmonizations so far was that in a lot of songs the speed of the chord changes actually speeds up at the cadence points. Listen to Happy Birthday harmonized with bar-long chord choices. It completely erases the cadence at the end because the V in the V-I cadence only lasts for one beat. This is very bad. Now listen to Happy Birthday harmonized by beat. Now it sounds too chordy. This is less bad than erasing the cadence but this harmonization sounds far too complex for such a simple tune.

-The solution to this was to create a new list of chord choices which uses an adaptive speed. Basically it keeps the bar-long chord choices for the beginning and middle parts of a phrase, but then speeds up to beat-long chords when there is a cadence point. The Happy Birthday harmonization using this method sounds spot on.

-Last week I did harmonizations of God Save the Queen/My Country Tis of Thee. The harmonization by beat for that one sounded pretty good, but the harmonization using adaptive speed sounds even better IMO. No need to change chords for all those passing tones and neighbor tones.

-Amazing Grace is another new harmonization using this method. The standard harmonization of this has some IV-I movement but the algorithm wanted to go IV-V. Sounds ok but definitely doesn't sound like the traditional arrangement which holds off on going to V until later.

Amazing Grace adaptive.mp3 | Amazing Grace adaptive.pdf
God Save the Queen adaptive.mp3 | God Save the Queen adaptive.pdf
Happy Birthday adaptive.mp3 | Happy Birthday adaptive.pdf
Happy Birthday bar.mp3 | Happy Birthday bar.pdf
Happy Birthday quarter.mp3 | Happy Birthday quarter.pdf

03/30/20

-Now the harmonizer isn't using the GA to make parts anymore at all. Before the alto and tenor lines were made with the GA and used the soprano and bass for reference. Then I changed it so that the alto line was made with the GA and the tenor line was made by process of elimination by figuring out which scale degree was still needed. That method was bad because if the alto choice wasn't ideal, it would force the tenor into making a bad choice. The current method now makes the alto and tenor lines simultaneously.

-Given the soprano and bass notes (known as the fundamental interval), there are only a limited number of options for how to arrange the alto and tenor voices. For example, if the fundamental interval is an octave and the soprano and bass are both on scale degree 1, the tenor MUST be the third and the alto MUST be the fifth. There is no other way to arrange the notes. But let's say the fundamental interval is two octaves. The ideal way to stack the notes is 1,5,3,1 because there is pretty much an even distance between all the voices. However you could also arrange them 1,3,5,1 (a 10th interval between bass and tenor) or 1,3,3,1 (it is ok in traditional SATB arrangement to leave out the fifth and double the third). The program remembers these alternate voicings for later.

-The first step is to arrange every beat in the piece with the most favored stack of the four voices, the most favored choice being the one with the most even distance between all four voices. The example "God Save the Queen before" shows this. However, just because every configuration uses the most favored one doesn't mean they flow together well. Look at beat 3: the tenor voice leaps up a 7th and then back down a third. This would be very difficult for the tenor singer to perform since it is so non-melodic.

-The next step evaluates how good or bad each alto/tenor combo is given the intervals to the surrounding notes. Then it compares the alternative voicings for that position and swaps it out if it reduces the intervals. The before example had 1,7,3,5 as the configuration for beat three but 1,3,7,5 made the tenor and alto voices flow much more smoothly, so that is what it chose.

-This method works great compared to the previous ones. All the voices are much more melodic when there aren't a bunch of weird leaps everywhere. There is also no possibility of illegal voice-crossing anymore since it is always choosing based on legal stacks of pitches. The only issue I can see with the "after" example is the second to last beat should have a better arrangement. Gotta figure out why it missed that one.

-One other thing I did was to have the chord guesser be able to work beat by beat as an option. God Save the Queen has a very obvious I-V-I cadence at the end and having it harmonize by measure turned it into ii-I.

God Save the Queen (before).mp3 | God Save the Queen (before).pdf
God Save the Queen (after).mp3 | God Save the Queen (after).pdf

03/14/20

-I made a pretty huge change in the way everything works in the program. I got sick of trying to define the length of every type of note and rest. Since note durations are not actually the duration they are supposed to be (due to the gap between notes), durations are extremely varied. For example, a quarter note rest following a quarter note will be a quarter's duration plus the gap time for a quarter note. But a quarter rest following a half note will be slightly longer since the gap following a half note is longer. I had a big list of all the different types of durations and every time I read in a new piece, if there was a note/rest combo it had never seen before, I would have to add it to the list. Making this even worse was the fact that very long notes have very long gap times. A dotted half note's gap time is longer than a sixteenth note so the program would constantly misidentify these as being actual rests. Tons of durations were getting ID'd incorrectly due to the huge amount of overlap.

-So I did some research on how the MIDI files express the durations and I finally wrapped my head around variable length quantity and signed 2s complement (the way it expresses negative values). I made functions that convert integer to variable length quantity or the other way around. Now every note, rest, or gap duration is dealt with as an integer and it converts to variable length when it writes. I also figured out formulas for how to determine gap durations given the note's duration.

-This was a much simpler way of doing things, but it meant I had to change every single step along the way which took a really long time. Before I had my own system for how long each type of note should be but now everything just works with the actual int value that MIDI uses.

-Now the MIDI reader and writer can deal with almost any type of note/rest length without me having to define it beforehand. Like it could see a whole note tied to a quarter note tied to a 32nd note and it will know what is going on rather than shitting bricks. The only thing it will mess up on now are weird tuplet groupings. It would have no idea how to do a seven-tuplet for example. But these are extremely rare so I won't worry about them until I see one.

-The harmonization I have included this time is Daisy Bell. The 6-beat note and the triplet eighths were messing everything up with the old system but now it's all working great! It made some questionable chords choices though. Don't know why it didn't pick any minor chords. The second half of the verse section definitely sounds like it needs some minor in there.

Daisy Bell.mp3 | Daisy Bell.pdf

02/14/20

-I did a few more harmonizations. Check out It's a Small World and Eine Kleine Nachtmusik Bad. As you can see the tenor line was still letting wrong notes creep in and also it has way too many big leaps.

Eine Kleine Nachtmusik bad.mp3 | Eine Kleine Nachtmusik bad.pdf

It's a Small World whole.mp3 | It's a Small World whole.pdf

-The weakness of the GA is that it is bad at giving specific notes that are needed. Since the tenor line is often forced into giving specific scale degrees based on what has already been chosen, it ends up being the worst of the four voices. So I decided to scrap the tenor GA altogether. Now it doesn't choose out of key notes anymore.

-If there is a big leap, it can fix that in various ways. It can go to a 2nd best choice. For example, let's say the soprano, alto, and bass already have scale degrees 1, 3, and 5 covered. The ideal choice would be for the tenor to double scale degree 1, but doubling the 3 is also ok. If the 3 ends up being a smaller interval, it will go to that instead.

-If the tenor is still forced into having only one option, it can swap out adjacent notes to their 2nd best choices. Or it can shift the note up or down an octave.

-Another thing that was going wrong with the GA's was a really stupid mistake on my part. In Eine Kleine Bad, you can see the alto note B in a C major chord, making it into a major 7th chord. This is because the GA always assumed that the soprano note would be a note in the relevant chord. Non chord tones in the melody would make the GA mess up the weightings. The eighth note F on top of the chord C major two bars earlier would trick it into thinking scale degree 7 was an acceptable choice.

-Check out Eine Kleine Nachtmusik Good which is after these fixes. You can see the tenor line looks a lot more melodic now and both the alto and tenor aren't picking wrong notes anymore.

Eine Kleine Nachtmusik good.mp3 | Eine Kleine Nachtmusik good.pdf

-Lullaby sounds ok but there is a problem with the phrasing. The phrase guesser works by measure only. Since every phrase technically starts on beat 3, having the chords change with the bar lines sounds awkward. This is also why it picked a V chord at the end of phrase 1 when it should've been a I. The phrase guesser needs to be a lot more adaptable so that is what I will focus on next.

Lullaby.mp3 | Lullaby.pdf

01/10/20

-I got a better sounding version of Drummer Boy with chord changes every half note. I completely revamped the inputs for the GA in order to do this. Before the GA always ran on a quarter note basis. If there was a rest in the melody, the GA would still output a note for that beat and it would get turned into a rest later. If there was a dotted half note in the melody, it would output 3 different notes. One of those would get turned into a dotted half duration and the other two got erased. This was a redundant method but it worked fine for other songs. But for some reason this was causing serious issues on Drummer Boy.

-So now the GA only outputs on a note by note basis. It no longer cares about note durations. If there is a long note in the melody, it outputs one value for that note. If there is a rest, the GA ignores it.

-This fixed the wrong notes in Drummer Boy. It is still choosing big leaps however. I really don't understand why it does this every time for this particular song when other songs don't have this issue.

-Also for Drummer Boy, I taught the chord guesser what a V7/IV chord is since this definitely appears (the Eb in the melody).

-I also had it do a harmonization of Silent Night. All the songs it already harmonized were in 4/4 or 2/2 so I wanted to make sure it could handle a 3/4 song. This required a few changes but nothing major. Sounds pretty solid imo. It had the middle phrases go IV-V7-I when "normal" harmonizations just go IV-I. This is another one of those instances where the program is selecting a chord we are not used to hearing but still fits.

silentnight.mp3 | silentnight.pdf

drummerhalf.mp3 | drummerhalf.pdf

01/04/20

-I changed my mind on what to work on next. Instead of doing several easy songs, I decided to do one challenging song instead. I figured expanding the capabilities of the song harmonizer was more important. The next song is Little Drummer Boy.

-This song had a whole bunch of new things that the program had never seen before. The biggest issue was the phrase guesser and chord guesser could not do tied notes over barlines. Little Drummer Boy has tied notes, 5-beat duration notes, staccato eighths, phrases that start and end with rests, and long sections where the harmony does not change at all. Basically this song was a great choice because it had tons of stuff that made the program trip up. So now all of those things can be dealt with by the MIDI reader, phrase and chord guessers, MIDI writer, etc. I won't get into the details but revisions had to be made every step of the way. Having a rest at the end of the piece instead of note turned out to cause a whole bunch of problems.

-Surprisingly, I didn't have to make any changes at all for the chord guesser to stay on the I chord for all of the first two phrases. Earlier I made it so that it wants to change harmony at the midpoint of phrases, but it avoided that and just stayed on I which is great. There were some other bad harmony choices though.

-It took a while to get the program to make MIDI files that were even readable due to all the weird new stuff in this song. As you can hear in the examples, there are still some issues. The whole note harmonization sounds bad since there are supposed to be chord changes halfway through bars. There were also huge leaps in the tenor and alto voices. This is penalized like crazy in the GA rules, so I have no idea why it let so many get through for this piece.

-The half note harmonization also has issues. It stays on the dominant way too much in the final phrase where there are supposed to be more changes. Some notes are also wrong like that tenor D in an F chord. This is probably an error from turning the GA results into the correct rhythm. So there's still more work to be done on this song, but at least it's making playable MIDI files now.

drummerhalf.mp3 | drummerhalf.pdf

drummerwhole.mp3 | drummerwhole.pdf

12/17/19

-Added a phrase guesser before the chord guesser runs. Next song I wanted to harmonize was Good King Wenceslas since it has an unusual structure of 4-4-4-5. The phrase guesser is pretty basic right now. It looks for long notes then if that doesn't work, then it guesses just based on length (like 16 bars almost always will end up being 4 bars of 4). This will need to be revised in the future but it works for now.

-The chord guesser now cares about where the halfway point is for each phrase, since this almost always has a change in harmony. So now for example if it sees a I at the halfway point in a phrase and the next chord could be either I or V, it will tie break to V rather than staying on I.

-Now that I've heard the Wenceslas examples, I think it would be smart to break the phrases down into quarters as well, since these can usually indicate a chord change in 4/4 pieces too. Gonna do this next.

-I made the decision to give IV chords priority over ii in a tie rather than using RNG to decide. I'm sure this will backfire when it harmonizes a different song later, so I really need to figure out a better method.

-GoodKing is the harmonization with the chords I entered in.

goodking.mp3 | goodking.pdf

-GoodKingWhole is with the chords the program guessed when analyzing one bar at a time.

goodkingwhole.mp3 | goodkingwhole.pdf

-GoodKingHalf is with the chords the program guessed when analyzing two beats at a time. Like Jingle Bells, there are some chord choices that sound weird to me but this is probably just the result of me being brainwashed into thinking there is such a thing as "right" chords for Good King Wenceslas.

goodkinghalf.mp3 | goodkinghalf.pdf

-Overall the method to harmonize is working really well. It never chooses wrong notes anymore. Since everything seems to be going smoothly now, I'm gonna try to bang out harmonizations for a bunch of more songs

12/03/19

-I fixed why it was giving me bad notes in those other examples. It was an error in the process of turning the results from the GA into the correct rhythm. (JingleGood)

jinglegood.mp3 | jinglegood.pdf

-Now it can add in passing tones. I thought this would make it sound better, but now I think it sounds worse. It sounds too cheesy and the inner voices moving around so much distracts from the melody. (JinglePT)

jinglept.mp3 | jinglept.pdf

-I made the routine to guess chords based on the melody alone. It does this when it analyzes the melody from the original MIDI file. It saves the chord guesses into the analysis text file along with the melody itself. First it slices up the melody into measure-long chunks and looks at which chord or chords fit the notes best. Then it does this process again using half-note chunks. So a 4/4 melody gets harmonized with two different chord progressions based on how fast you want the changes to come. It averages the scores for all of the chord choices so you can see whether half notes or whole notes was a better fit. In JingleWhole you can hear how using whole notes really clashes with the melody in places where there is obviously supposed to be a mid-bar chord change.

jinglewhole.mp3 | jinglewhole.pdf | jinglehalf.pdf

-Figuring out which chords to choose is actually a really tricky process. There are lots of times when there is a tie between several chords that are equally good. At first I wrote a process that would look at strong beats and treat them as more important to break the ties. But this meant that any time there is a non-chord tone on a strong beat it completely messes up the choice. So I got rid of this method completely.

-I settled on breaking ties just by looking at the previous and following chords. Like if it sees a V7 before and the choices to break the tie for are a I and IV chord, it is more likely to resolve to I. This is an imperfect method though. I think I need to make it look at the chord progression on a more macro level and treat where the chord falls in the progression as being important rather than just looking at the preceding and following chords.

-Another example of a difficult choice: whether to use IV or ii on subdominant parts. These two chords often fit equally well. The only way to choose is whether you want the harmonization to have more of a major or minor flavor. I have no idea for how to make a computer decide this so it just flips a coin.

-A lot of times choosing the "right" chords for a melody is a really philosophical question. Sometimes chords that technically fit the melody of Jingle Bells just as well as the "right" chords sound really weird. Why does it sound weird? Is it just because I have gotten so used to hearing Jingle Bells a certain way? Or is there a logical reason chords that technically should sound good don't? Listen to the chorus of JingleHalf in the first "one horse open" part which is B-A-A-B. Technically speaking, both a I chord and a V chord fit. In fact, you could even make the argument that I is a stronger choice because B is on the downbeat. However, this sounds mega weird harmonized as I.

11/22/19

-I took a break from working on the algorithmic hymn maker to the harmonization stuff. Now that I have the basic framework for the harmonizer working, I'll start working on both.

-First thing to do for the harmonizer was to read in a MIDI file of Jingle Bells and save a text file. Unlike the detailed harmonic analysis text files for the Bach chorales, this only saves basic stuff (key and time signature) as well as the melody. I threw out all the accompaniment.

-The harmonizer then grabs the melody line from the text file. For now, I have it so that the user enters the chords they want to harmonize the melody with by hand. In the future, I think it would be badass if the program itself made chord choices based on the melody. The user could just input how chordy they want the song to be. For example, level 1 could be I, IV, V, and V7 chords only. Level 2 all diatonic chords. Level 3 all 7th chords too. Etc. So you could go from most simple harmony possible to jazz arrangements.

-Since I tell it the chords and how long the chords lasts (half notes in Jingle Bells), it figures out the bass without having to run the GA.

-The alto and tenor GAs work by quarter note. Since most melodies have varied rhythm, I needed to make a list of the notes falling on strong beats.

-The first attempts (running the exact same rules as Bach chorales) sounded bad. JingleBad is an example. It only has an alto line for phrase 1, but that was enough for me to know it was failing miserably. After I got rid of the rules for parallel fifths, the rule to avoid making the same pitch over and over in a row, and the rules to be more specific at cadence points, it suddenly got way better. The overly specific voice leading rules blocked obvious choices and it would start picking bad notes. JingleThree is SAB and JingleFour is SATB and both sound better but still a lot of mistakes.

jinglethree.mp3 | jinglebad.mp3 | jinglebad.pdf | jinglefour.pdf

-Actual Christmas carol arrangements need to have the voices moving in unison so the other parts can sing the words of the song. This is now an option, where the results from the GA just get altered so they follow the rhythm of the melody. JingleUnison is the first example it made. There are quite a few wrong notes, especially at ends of phrases.

jingleunison.mp3

-Next I need to figure out why it's still giving me garbage note choices. In a short song with nothing but I, IV, and V7 chords, any wrong notes is completely unacceptable. It is also voice crossing even though the rules against that are still in place. Also the fact that the bass, tenor, and alto voices are just quarter notes chopped up into eighths makes them a lot less dynamic. After the GAs run, they should be nudged around to make some passing tones and stuff.

11/10/19

-I've been working on getting the rhythm of the harmony for modern hymn pieces figured out. The rhythm of harmony was super easy for Bach chorales; every beat is a new chord. But in a hymn, each chord could be a beat, or hang around for 2 or 4 beats etc. So when I analyze the piece "Pass Me Not O Gentle Savior" it records the rhythm of just the harmonic changes and saves that in my analysis text file. I also have it record the common small rhythmic elements. Obviously a hymn with nothing but whole and half notes would sound boring, so I need to spice up the parts where the harmony doesn't change for a while with better rhythms later.

PDF | MP3

-When I make a random hymn, it grabs that rhythmic info from the text file. This tells it how many chords I need per phrase and the durations of each. I also made it figure out where the halfway point in the phrase is, since this can now differ. For example, a phrase that goes "half, half, whole" would have the halfway point be on the second chord whereas a phrase that goes "whole, half, half" has the halfway point at the first chord.

-Since the hymn I loaded in has an A'ABA structure, I needed to make sure the chords get copied for the A phrases that are repeats. It also needs to make sure it has a little ending variation for the A' phrase.

-The example shows only the rhythm of the chord changes. You can see it has 4 phrases of 4 bars each and the chords are changing at the right pace (it is structurally a total ripoff of "Pass Me Not" since that's the only hymn scanned in so far).

-The next step will be to add the common small rhythmic elements which will replace most of the long notes by adding non-chord tones and stuff. Also it needs to copy the soprano, alto, and tenor voices for the repeating phrases. Right now it just worked like it does for the Bach chorales where it runs the GA for every phrase. So in the example the chord progression and the bassline reflect the AABA structure, but not the three higher voices. Also it was supposed to make a V chord at the end of phrase 3 but it gave me a iii chord instead. Gotta figure out why it did that.

10/20/19

-I got another Bach chorale scanned in and fully analyzed. This one had a few things that were new to the OCR. I put off natural signs until I had to scan in a piece that had them and this was the first. Naturals are slightly more complicated than flats and sharps since I have to look back to see whether the natural is affecting a sharp or flat in the key signature or one that appears earlier in the bar.

-This Bach chorale was in F minor and ended with a picardy third. The other minor/dorian chorale that was scanned in ended on a half cadence. So now that this piece is in the analysis database, the random Bach chorale maker can do F minor pieces and end on a picardy third in a minor chorale. Very exciting. (Example of this)

-I added a lot more into how a MIDI file gets displayed. When it shows all the bytes in the file, it can now show it in int, hex, or binary depending on what I want to look at. No matter which of the three of those it is showing, it now also tells me what the hell I'm looking at next to important bytes. So now instead of having to hunt around for where my pitches and durations are, I can just look for labelled events to find stuff quickly.

-Now that I have tried to scan in MIDI files from a bunch of different sources, it's now very clear just how much redundant information can be included and how different the redundant info can look. Ideally, I wanted the program to be able to scan in any MIDI file and not be thrown off if there is redundant info (like channel messages or lyrics or copywrite notice etc), so I spent a lot of time working on this. Eventually I decided it was a fools errand after seeing so many wildly different-looking MIDI files. So now every time I want to scan in a MIDI file, I import it to MuseScore first and then save another MIDI file. This eliminates all the weird shit since MuseScore saves a MIDI file just with the basics (note on, pitch, duration). It would be nice if I could handle everything just within my program, but it looks like sometimes I just have to take this extra step in MuseScore.

-I got "Pass Me Not O Gentle Savior" by Fanny Crosby scanned in and analyzed as my first modern hymn example. The method I have in place for analyzing Bach chorales does most of the stuff perfectly fine for hymns, but there are a few important differences. In Bach chorales, I had to jump over to the OCR to look for fermatas in order to determine where phrases end. In a modern hymn, it's super easy: find the whole notes. What's harder is identifying the structure. The Crosby hymn is AA'BA so it needed to deal with figuring that out.

-The last step that I need to do to get some random hymns is working out the different rhythmic elements. Bach chorales are just a series of chords with some non-chord tones snuck in there to make it sound more fluid. For hymns, I need it to look at which rhythmic elements are the most common. Then the random hymn maker needs to worry about the rhythm when choosing which chords to use. Just rolling the dice for a string of chords will sound bad in this context. It's a bit more complicated than rhythm for the Bach pieces but I don't think it'll take too long.

09/10/19

I got the Bach chorales sounding better and started working on getting new pieces analyzed. Here's the details:

-Right after the chords get chosen, I now double check to make sure that a combination of chords isn't something that Bach never would have used. Like going from a major V to a minor v is not ok. Every time I hear weird combos, I can now add them to the prohibited combo list.

-After it chooses the inversions for those chords, I do the same thing to make sure the inversion choices make sense in combination. For example V65/V must lead to V in root position, not V6 or V64. This fixed all the voice leading issues in the bass.

-I added more rules in the GA's that select the pitches to deal with unresolved leading tones and 7ths.

-Sometimes the GA that chooses chord functions would choose the same thing in a row. Like it could give me three dominant chords in a row since it was choosing at random what should follow a dominant. Bach never did three of the same function in a row. Now I have a rule that keeps this from happening.

-When giving a range to the GA, I made a really dumb mistake that I can't believe I didn't notice earlier. If I give it a range of 0-12, the numbers it actually gives are 0-11. I always need to bump up the high end by 1 in order to have it actually give me the highest possible note I want.

-Since voice crossing sometimes keeps happening even though I have rules against it in the GA routines, I now double check after all the voices have been chosen. If it sees voice crossing, it simply swaps the pitches for those two voices. Of course this can sometimes lead to parallel 5ths or octaves.

-My rules for prohibiting parallel 5ths and octaves in the GA's used to be too harsh. I would just look at the intervals between voices and if I saw two fifths or octaves in a row, I said it was bad. However, it only counts as a bad parallel if the voices move. If alto goes A A and the soprano goes E E, that is ok because the voices don't move even though it is two perfect fifths in a row.

-Example "afterfixes" is what it sounded like after these changes.

MP3 Example afterfixes

-Then I got to work on trying to get the tenor to move around less. I had the idea that it needs to analyze the fundamental intervals (distance btw soprano and bass voices) in the actual Bach examples. If it picks soprano notes with this in mind, it will get forced into picking a bad tenor note less often.

-I re-entered the Bach examples after changing it to also keep track of the fundamental intervals.

-Then I added penalties to the soprano GA for the fundamental intervals that are less common.

-Example "ints" shows what it was like afterwards. You can see in the score that the tenor is better, but there are still some big jumps.

MP3 Example ints

-Then I added a rule to the tenor GA that already existed for soprano and alto: just penalize for large intervals. The combination of these two things made the tenor a lot better.

-Example "tenrule" is what it sounds like now. Still not perfect, but the tenor voice looks much more melodic than how it used to be.

MP3 Example tenrule

-I wanted to load in the next Bach chorale and a 19th century hymn by Fanny Crosby, but neither of them worked. This is because there are two different ways of keeping track of time in MIDI files (metrical time and time-code). All the other examples I had loaded in used only one method, but these new MIDI files used the other one, so all of the delta times looked totally wrong to the program.

-Now I got the next Bach chorale loaded in after getting the MIDI reader to have an entirely different set of values for how long each note is. Pretty annoying. I wish there was just one standard. Hopefully this will work with all of the other MIDI files that use this method for timing, but so far I just have the one new Bach piece leaded in.

08/21/19

Algorithmic Bach chorales are (almost) done! The program can really just crank them out now. None of them have been flawless so far, but overall I'm definitely feeling the Holy Spirit when I listen to them. Here's what I did:

-Added the rest of the non-chord tones (NCT) and fixed a bunch of little but important errors that were still happening.

-I fixed an error where sometimes chorales were being made with wildly different phrase lengths. One had a phrase as short as four beats while another phrase had 12 beats. None of the Bach chorales in the database had this extreme range of phrase lengths.

-A lot of chorales were being made where the soprano voice would end on scale degree 5 instead of 1. This was an error. Every chorale that ends on a I chord had the soprano ending on the root note. Now this is fixed.

-Similarly, almost all phrases that end on I need to have the soprano covering scale degree 1. A lot of my authentic cadences sounded weird and I realized it was because Bach wouldn't have 5 on top. However he would have 5 on top for half cadences and cadences that tonicize other chords.

-Some of the major key chorales would end on a half cadence but Bach never did that in the loaded examples. Now they all end on I.

-Very rarely, some cadences would end up with all four voices choosing the root note. Now it is forced to at least have a third present too.

-My first method for selecting whether to do a NCT or not actually did it way too often. The example "toomanyncts" shows this. (I only put the soprano and alto through the NCT routine here). It sounds good, but it's way too embellished for a Bach chorale.

-A big issue that comes up frequently still is that the tenor voice, since it is selected last, gets railroaded into a note that creates other issues, like a parallel fifth or crossing above the alto voice. I gave it a little bit more flexibility. Now I say "you really want to choose scale degree 1, but 3 is cool too." So now if picking 1 creates a parallel, it has the better option of choosing the backup. This hasn't solved everything though.

-Sometimes the tenor choices the GA makes are inexplicably bad. In example "badnote" listen to the 3rd cadence. It knew it was supposed to go from a D7 to Gm. It knew the tenor voice needed to get the 3rd in D7 (a F#). But for some reason it rolled with Eb, which gave itself a penalty of 100,000 in the GA. Weird and annoying.

What next:

-I still want to fix the tenor. It jumps around way too much and in general it just gets forced into making bad choices. I need to have something that will go back and shuffle around the soprano and alto voices if the only option for the tenor is dubious, is a big awkward leap, etc. This seems like a difficult problem to solve but I'm pretty sure I can come up with something.

-Accidentals are often not resolving. In example "unresolvedsharp" there is a cadence on Bm, but the soprano goes from A# (the leading tone) to F# instead of B. This doesn't sound terrible but it is absolutely not okay according to the rules of voice leading. This type of thing would be a dead giveaway to musicians that the piece was not by Bach.

-In phrases that end on a ii or IV chord for example, I should hear lots of those chords earlier in that phrase. If it is a bunch of I and V chords and then it suddenly ends V7/ii ii, the cadence on the subdominant sounds like it comes out of nowhere.

MP3 Example 'Bad Note'
MP3 Example 'Long but Good'
MP3 Example 'Mostly Good'
MP3 Example 'Pretty Good'
MP3 Example 'Too Many NCTs'
MP3 Example 'Unresolved Sharp'

08/07/19

-There was a problem where once a MIDI file got over a certain length, it would not write correctly. It took me a while to figure out that the issue was with how the MIDI file saves large numbers with variable length form. There is a simple routine that would take in the large integer that indicated the length of a given MIDI chunk, and then figure out what that would be in variable length. The problem was that this routine would not reset its values if it got used several times in succession. Since the Bach chorales write a 4-chunk file, it would give me incorrect chunk length numbers once the chorale got too big.

-Once I got that solved, I could make a full-length Bach chorale in quarter note texture. Example 1 is the first decent complete chorale progression it made. The score doesn't have fermatas indicated where they should be, but the audio file has them. This is the chord progression:

Phrase 1: I,VII,vi6,V,I,V,V,I,
Phrase 2: I,V,I,V,I,I,V/V,V,
Phrase 3: V65/V,V6,V,V/V,V6,V,vii*7,Ip,
Phrase 4: ii6,VII,IV7,I,I,vi,V7/V,V,
Phrase 5: I,I6,vi42,ii,I6,IV,I,V,
Phrase 6: vi,V7/V,V,V,ii42,IV6,V,Ip,
Phrase 7: vi,I,I64,vi65,IV,IV6,V7/IV,IV,
Phrase 8: V6/V,V6,vi,iii,ii,V/V,V,I,
Phrase 9: vi,IV,vii**65/V,V,V,IV,I,V,
Phrase 10: I,IV64,IV6,I,V,vi,V7,Ip,

MP3 Example 1
PDF of Sheet Music 1

MP3 Example PT
PDF of Sheet Music PT

-Overall I'm pretty happy with how it sounds. It would probably fool people who don't listen to classical music. The problems: too many D-sharps that don't go to E, tenor voice jumps around too much, alto and tenor distance exceeded an octave, and the use of a VII chord that early is really questionable.

-Next I started working on how to add non-chord tones (NCT). Since there are lists of all the NCT's that Bach used in the examples I loaded in, I can figure out how often in every chorale each voice would have a NCT instead of just being a quarter note. Some Bach chorales have lots of embellishments and other ones are very quarter note heavy, so I didn't want to have an average of all the pieces. Instead it picks the NCT frequency from only one.

-The stats also break down by which type of NCT. Now that I know I have a soprano NCT for example, what are the odds it will be a passing tone? What are the odds it will be a neighbor tone? Will it go up or down? Etc.

-Example "pt" shows descending passing tones. When it saw a descending major third in the quarter note texture, it knows that is a great place to throw in a descending passing tone. In this example it put one in every time it could (for the soprano voice). Next I need to make it apply the stats to limit how often it adds an NCT so it doesn't add too many or have a non-Bachian balance of which NCT types appear.

-Next week I'll teach it the rest of the NCT types and then hopefully it will be making some tasty chorales.

Back to top

6/20/19

The Bach chorales now have tenor lines. Tenor is pretty much the easiest line to make because I make it after the other three so I just look at which scale degrees are already present and grab the one I need. In a triad, if all other scale degrees are already accounted for, it is customary to double up the root note.

The problem I think this makes is since the tenor is so picky, it can end up moving by a lot. Bach's tenor lines also have big leaps sometimes, but in EXAMPLE 1 you can see the tenor going up by a seventh and then back down a seventh the very next beat. I'm pretty sure this would be a no no. I think I need to make a routine that will look at the 4-part arrangement and decide if there can be a change in which voice is on which scale degree to make there be less jumping around.

Before I was only making first phrases just for simplicity, but now that I have all four voices I want to now have it make an entire chorale. I keep getting an index out of bounds error on the final phrase and I'm not exactly sure why that's happening, so this is the very next thing that I need to fix. Example 2 has the first two phrases and it sounds good, so my process of going phrase by phrase is working, it just crashes on the last one for some reason.

In these examples, you can see the parts are mislabelled a lot. It also always wants to put the tenor as a treble clef when it normally would be in the bass clef with the bass voice. These are issues with MuseScore. It doesn't know that I am feeding it a four-part choral piece, so it just makes its best guess based on the fact that the instrument I chose was midi choir. I guess I should be impressed that MuseScore can even take a look at the range and say "this looks like a soprano part" instead of just labeling everything "voice," but it does choose the wrong label more often than not.

MP3 Example

Why would the ga always put the big leap at the beginning? The reason is because I have to compare two adjacent numbers at once which I do with a for loop. The for loop runs from the beginning of each generation to the end looking at two notes at a time. Is there a way that I can penalize based on two notes without using a loop? If I can penalize pairs of notes without having to go in a particular order, it will solve this. Right now it looks like: if(Math.abs(g.a[i] - g.a[i+1]) > 4){add penalty}

4/29/19

I finished the GA routine that gives me chord functions for the Bach chorales (tonic, dominant, subdominant, or secondary dominant). It is making different choices based on which phrase it is within the chorale since a first phrase or a final phrase will have certain characteristics that are different from phrases in the middle. It is choosing chord functions based on their overall prevalence in the chorales that I have scanned in as well as knowing the likelihood of which one will come next in the sequence. It pays special attention to the starting chord of each phrase and the cadence point to make sure those are something Bach would've chosen. This is what an example looks like (1 is tonic, 2 dominant, 3 subdom):

(1, 3, 2, 3, 3, 2, 1, 3, 1, 1, 2, 2)

I also finished the next step with a few things that still need to be done. This next step is taking the chord functions and assigning which chord (without inversion) shows up. For now, I'm not doing this in the GA, I just roll random numbers and pick based on how often they appear in the loaded chorales. Like, given a chord is a subdominant, what is the likelihood it will be a IV chord? This is what it chose for the same example:

I v7 V ii7 IV V I iii I I V V

So pretty good job overall. It is ending on a half cadence. This is a first phrase, so that is a pretty common thing to see. The reason there are two V chords at the end is because this phrase ends on a fermata'd half note and each chord represents one beat. It looks like it's picking 7th chords less often than triads at the same rate as Bach did.

There is some bad stuff going on here. One thing I want to modify is have it break down the stats further by major or minor. Would Bach have a minor v chord appear in a major key chorale as the second chord? I'm guessing the answer is no. Too much mode mixing too early- too romantic, very not baroque.

There is also the potential for picking chords in a non-Bachian order. Like going from IV to vi in the Baroque era was considered weird, but going from vi to IV would be totally cool. These combos would both be equally likely to be chosen. I think I am going to move these rules over into a new GA routine so it can penalize order choices as well.

The way subdominants work is they look to where they resolve to. A subdominant that resolves to IV must be a V/IV, V7/IV, vii*/IV, etc. The problem is secondary dominants often happen in sequence (ie, this phrase ends with a cadence on ii, so I need to see a lot of V/ii chords in this phrase), so picking the subdoms they resolve to at random would not sound right. I need to make sure they correlate with each other. I think if I move this stuff over to the GA, it would be able to handle that as well.

4/2/19

I am making the GA rules that will give me a Bach chord progression but only the chord functions (tonic, dominant, etc) for now.

I tell it what type of cadence to make and it will make the final selection (or two selections, if the last note is a half note or whatever) match whatever it needs to be.

Before I run the GA, I sweep through ALL of the input chord progs from actual Bach chorales and figure out how often Bach used any given chord function in a phrase. This data is broken down further by position of the phrase in the song, because this is important. For example, it now can tell me the ratio of tonic chords in phrase one (using data from all of Bach's first phrases). Alternatively it can tell me the ratio of dominant chords in all of Bach's second-to-last phrases. It's very specific info.

So now, depending on what phrase I am making functions for (let's say it's a phrase 1), I just feed the GA the ratios for the first phrases. Then my GA penalizes the results if they stray too far from how often Bach would've used them in a first chorale phrase. This way we make sure we are not getting non-Bachian distributions of chord functions. Like subdominant chords are rare in a first phrase, but for a phrase in the middle of the chorale, they appear more frequently.

Next I need to figure out the ratios for how the chords lead into each other so the chord functions are in a good order in addition to having a good distribution. Like, given that I'm in a phrase 1 and I'm sitting on a tonic chord, what are odds the next chord will be tonic? What are the odds it will dominant? Once I do this and get the odds for the first chord in the phrase nailed down, the chord functions will be good to go. Then those will go on to be assigned specific chords (is my dom a V or a V7?)

3/15/19

So this week I have been working mostly on the OCR side of things still. I made a new method for detecting barlines, and I think this one actually works. When figuring out the pitch of a note, I would need to look backwards until the previous barline for a sharp, flat, or natural. If the line was too jenky to be detected, it just keeps looking. This was bad.

My old method for finding barlines was like anything else: I had an image and I would look for the hamming distance. Now I have a multi-step method. First I look for long barlines just to find out if the staves are grouped together in pairs. This is unreliable for finding every single barline, but it is guaranteed to hit at least a few. Before I was making educated guesses on how staves are grouped based on the distance of the staves from each other. Now I know for sure by IDing a big barline.

Once I have that info, I can slice up the image into just each system. So the upper and lower margins and space between systems is cut out. Then I simply look for what percentage of each column in that sub-image is black. Lots of black = barline. So far this method has been accurate.

If I see two lines a certain space apart, I know if it is either a repeat sign or an ending. For now, I am assuming if I see it in the lower right, it's an ending, but I know this will give me a wrong result eventually. Looking for the two dots to confirm repeat sign-hood seems like a fool's errand since so many other things to the left of an ending could throw it off. I've been pondering this dilemma.

After doing all this, I definitively know the x-coordinates of the barlines for each system on the page. When I figure out the pitch of a note, instead of doing a barline search backwards, I already know which two barlines it is in between based on the x,y coords of the note.

3/5/19

Getting back to the image recognition in a big way. I had some issues because I was keeping track of what row the staff lines are on in two different lists: where I saw them on the right versus on the left. So if there is an indented first system, those lines would be missing from the left hand side list.

This system was shit for a lot of reasons that I wont go into. The main reason is if there is basically any curvature in the lines, the value for that line would be wrong since I only knew how far down it is on one side of the page but not the other.

So now I look until I find staff lines once, save those away, and then keep looking in either the top or bottom half of the page until I find the missing lines. Then I make those lists into one list, which would be where I see all the lines from the left side, regardless of how close to the margin they were.

I do the same thing with the right side, since hitting an ending (thick line) hides the individual staff lines.

Then I find the mean Y-value of all the lines based on their leftmost and rightmost ends. So now instead of having to deal with two different staff line lists that may be incomplete, I have one master list. And if there is a 3 pixel bend in the line over the course of the page, it remembers a number in between. Now the program has much more accurate info on where it thinks staff lines are.

Another thing I was doing as a quick and dirty workaround was figuring out how many staves are in a system. Now I look for really long barlines, like the kind that couple a treble and bass clef together in a standard 2-staff system. Instead of guessing how big a system is, I now have a routine that actually checks.

I worked on the barlines finder to make it less strict. Now it is guessing how many bars are in each system way better.

I scanned in the data for two more chorales.

I'm adding in more info into how the algorithmic Bach chorale generator chooses basic structure stuff. It was making some bad choices with regards to how long phrases would be and which cadences happen together.

2/8/19

Here's what I have been working on. I'm finally ready to start turning the Bach chorale data into chord progs. I hit a few more major weak points with my system and I was fixed those:

Black noteheads in spaces kept having big issues with getting ID'd correctly. I added way more definitions into my deal-breaker list of pixels that I absolutely do not want to see. I had the same issue with sharp signs in spaces.

The way my pitch guessing system worked was overly complicated. It looked at the note and figured out the letter name and the actual pitch (if it has a sharp or flat in the key sig) at the same time. It is much more more efficient to figure out the letter name first, then do a simple keysig check to figure out if that is one of the affected notes.

Also my pitch routine didn't know about dummy accidentals. Before, if it hit a dummy accidental, it would change the pitch just like it was an actual one. This resulted in double-sharps which I don't want. Now it knows and skips the second pitch change.

I added a whole bunch of stuff to my non-chord-tones finder. Basically it needs to keep track of everything going on that doesn't fit the chord progression. So it wasn't enough to know that a note was just a suspension- I also needed to know if the first note is longer or not. If there is a passing tone, is it going up or down? Little extra info like that.

Also, if there is a voice change that is not an NCT (like if a voice goes from scale degree 3 to 5), then I also need to know about that. When I create a random Bach chorale, I will start from the chord prog. Each chord hits one beat. Once I have that, you basically just have to scoot around notes to add everything that make a 4-voice piece sound interesting instead of just being a series of chords. I need my program to remember every instance that Bach has something not lockstep in the chord prog.

1/17/19

Got another Bach chorale's harmonic analysis saved.

Made classes to group data for voices and phrases together.

Redid all the functions that related to that info. Most notably, the non-chord-tone function is a lot more efficient now that it can break stuff up into smaller problems thanks to the info being sorted smarter.

1/8/19

I finally got chorale 3 analyzed correctly and got the data saved away. That was the dorian mode chorale, so my harmonic analysis now knows how to deal with that. (Dorian pieces form this era usually have tons on mode-mixing from the parallel minor key).

I started doing the harmonic analysis for chorale 4, which I hoped would go through without any issues. Of course I hit some.

What I discovered was the way I analyzed sixteenth notes in a SATB setting was totally dumb. It made no distinction between which sixteenth fell on the upbeat. Now it checks to see if the rhythm was "sse" or "ess" or "ses" so it can tell which sixteenth note is harmonically important and which one was just the passing tone or whatever.

Also, I realized that when suspensions happened, the list of notes that happened in the preceding beat would not be cleared since it was looking out for notes that start on downbeats only. Now I keep a tally of how long the SATB notes are in a beat, and once it goes over the length of the beat, it knows that that is the time to clear the lists.

1/1/19

Development updates from Ellery Wolfe:

I worked on these as two different NetBeans projects: one for composition and one for OCR. Now I need to have them talking to each other a lot, and this has resulted in some issues. The most nightmarish of these was a problem I explained in an earlier email. I needed to jump over into the OCR to ID fermatas in order to find phrases in Bach chorales. My first method was to figure out which bar the fermatas are in. This is bad because spacing of notes changes based on page layout.

Basically, even knowing the bar number, I could not figure out "fermata is over beat 3" without having to ID all the preceding note durations in the bar. So then I completely redid the function to have the OCR figure out the four notes under the fermata and then I can sweep through the MIDI data looking for where that combo of notes happens. I spent like three weeks on just this.

I was getting a lot of false positives in the OCR. Especially it thinking a line note is a space or vice versa. Just going off of a hamming number wasn't enough info. I came up with the idea for "deal-breaker pixels." Like, if it sees a bunch of white pixels in an uninterrupted column on the left edge, it knows it must be a note in a space. Now I'm getting much more accurate identifications of objects on a page.

I also redid how the OCR scans for something like a note. Before it would go left to right along staff lines. However, getting staff lines perfectly flat across just one or two rows of pixels is almost impossible, even after rotation. Now it looks everywhere within a designated rectangle. This takes longer, but it doesn't slow down the run time too bad. Now notes that would be missed from bending staff lines (like from scanning too close to the book binding) are now being detected. I started working on a function that would auto-straighten staff lines a while ago, but I think this fix was much simpler.

The OCR would often count the same note (or sharp sign or whatever) twice if it managed to clear the hamming distance at different coordinates. Now I have a function that throws out any double-counted items by figuring out if it is the same type and if it's too close.

I was getting a lot of wrong info from a function that told me the pitch of a note based on its page location. Now it distinguishes between space and line notes, which makes it much more accurate. It can now also change the returned pitch based on the key signature OR an accidental!

I made image rotation more accurate for songs where the first system is indented.

In general, my output box was spewing out way too much info and it was slowing down the program and making it hard for me to troubleshoot. Now basically everything I may need to see are all attached to local "show" variables that I can turn on and off.



I can now make a beat by beat harmonic analysis from a MIDI file (great for choral music). It analyses the chords multiple ways: with inversions, without inversions, and simply the harmonic functions. It also figures out all the non-chord-tones, their types, their locations, and which SATB voice they are happening in (passing tones, suspensions, etc). For now the analysis assumes every beat is a new chord, but this same idea will work for other genres of music later. Instead it would just have to look out for the frequency of notes (lots of C, E, and G in this bar, must be a C major), rather than assuming a complete chord will exist on every beat as in choral music.

The harmonic analysis can also figure out the ends of phrases based on where the fermatas occur on a page like I described earlier. This should be much SIMPLER for non-choral music where the phrase endings are really obvious from the rhythm and chord progression.

It can figure out the cadence types for the phrase endings.

It can now recognize repeat signs and treat repeated phrases as such, instead of keeping track of arbitrary repeated info.

I made a method to save all of this harmonic analysis data for later and to read it back in various ways. It saves by song and composer. These files provide the needed info for a random song of that type (just Bach chorales for now). For example, it can look through all of the files for the keys of the chorales, put them all into one list, then grab one at random. This way the pool of available Bach chorale keys accurately reflects the keys he actually wrote chorales in. This method has the potential for some insanely meticulous analysis that is beyond anything a human has ever done. Like, I could ask "what is the probability Bach uses a subdominant chord in the second beat of phrase 2?" or some shit like that.

I made a function that tells me the pitches of all active notes at a given point in time, regardless of whether or not the note started at that moment.

It can figure out if a song has a pick-up note, provided the MIDI data accurately keeps track of time signatures.

7/24/18

Once a score is scanned in, all of the data that was contained on that page is stored for us to use however we want. Some of the tasks that this program can then perform with our page data are incredibly useful for performing musicians.

It can isolate one staff from a page and reprint it, getting rid of the clutter of having to look at other musicians' parts. Most duets that I've played do not have separate scores; it is quite annoying having to ignore every other staff when reading a page. The OCR remedies this common problem easily.
Jana Top Staff (PDF)

Transposing a part in your head can be frustrating, and this commonly comes up for wind players and accompanying musicians. With this program, transposing parts also becomes a very quick and simple task. You simply tell it how many half-steps you want to transpose, and it does the rest, including changing the key signature and accidentals... all from your scan! No more need to hunt around for a digital copy of the music! Here's the same right hand part transposed up a whole-step.
Jana Top Staff Transposed (PDF)

Whenever an output score is made, it always comes from a playable MIDI file. You can always use this program to listen to your score or part and can even change the tempo.
Jana Top Staff (MIDI)
Jana Top Staff Transposed (MIDI)

Back to top

07/17/2018

Optical character recognition for music is a really great tool. Once a paper score is scanned in, it can then have a very thorough harmonic analysis performed on it very quickly. This will provide lots of useful information to be used in algorithmic composition. If we want to imitate a certain composer or musical style, we need lots and lots of information that would be too tedious for a human to gather.



Additionally, this will allow for fast transposition of musical parts. Just scan in the original, tell it how to transpose, and it will create a new page for you! Also it can be used to isolate a single part from a score with many instruments. There are many practical applications for sheet music OCR besides feeding data to an algorithm.

When a paper score is first scanned in, it is resized, rotated, and converted into black and white. This is what a score looks like after that process: (use my example, maybe. I just got it off google images, so I'm not sure if under copyright.)

The OCR looks for elements on the page using the staff lines and bar lines as guides. It will show me both what it thinks an element will look like as well as whatever it saw that looked close. Here is an example for a treble clef:

This is what the treble clef finder is looking for:


------------------------------
ham number: 65



TREBLE CLEF FOUND AT 143, 406
------------------------------

It will then give me a read-out of basic layout information:

• the first system is indented and there are 2 staves per system
• the end of the piece is on this page
• there are 8 staves on this page
• there are 4 systems on this page
• the systems on this page contain 2 staves
• all of the staves on this page are treble clefs

It will also give me all the info I need for the notes:
-------------------------
ALL DOTTED HALF NOTES ON PAGE:
x: 512, y: 583, pitch: A4, measure: 8
x: 1093, y: 593, pitch: F4, measure: 12
x: 360, y: 878, pitch: B3, measure: 14
x: 359, y: 998, pitch: D4, measure: 14
x: 722, y: 1121, pitch: E5, measure: 24
x: 722, y: 1302, pitch: G3, measure: 24
x: 512, y: 729, pitch: C4, measure: 8
x: 1094, y: 739, pitch: A3, measure: 12
x: 642, y: 884, pitch: A3, measure: 16
x: 911, y: 894, pitch: F3, measure: 18
x: 642, y: 1004, pitch: C4, measure: 16
x: 910, y: 1014, pitch: A3, measure: 18
x: 618, y: 1117, pitch: F5, measure: 23
x: 618, y: 1298, pitch: A3, measure: 23
-------------------------

Back to top

06/06/2018

• This week I got my images program detecting staff lines going in from the right and left margins like you recommended. If the lines are flat, then it figures out the average distance between staff lines, resizes the image so that they are 10 pixels apart (18 in my first example is unusually big. It was printed landscape style. Most scores that I've looked at have about 8 or 9), and then saves the resized image.
• It works really well for image files that are aligned.
• I also thought about what happens when you scan from a book with a spine instead of a loose page. There is always a downwards curve that appears as you near the spine even though the rest is level. If this happened in a sheet music scan, we would have to throw it out entirely, right? This is no longer a simple rotation but some Photoshop type fix.

Back to top

05/22/2018

• Lots and lots of housekeeping. The code has become large and difficult to navigate. It's hard to figure out where along the chain of command something is going wrong. I've been refactoring, adding in comment lines, fixing indents, and adding in universal constants to make stuff easier to read. Still lots of work to be done, but lots of places already look much nicer.
• Redoing how rests are done. I've been having troubles with rests for a long time and it all goes back to how I originally chose to do rests. I had them as notes with a velocity of 0. Now they are long delta times. Changing over is fixing the big issues but this is time consuming. A 1/4 note rest following a 1/4 note is different than a 1/4 note rest following a 1/8 note. Now I have to figure them all out. And since it's all in variable length format, it's not just a matter of adding two ints together.
• I don't like how melody-like the bass parts are. I'm working on something will create a simple bass groove.
• Making a Texas Hold'em game. Not music related, but I wanted to make some poker AIs.

Back to top

04/18/2018

This week I continued working on drums.

• Drums can now make fills using the same beat by beat method as the general drum beats.
• Each song chooses a base fill. At other times during the song, it will either add the base fill or create a new fill that is a variation on the base fill. This way all of the drum fills in a given song will sound somewhat similar.
• Expanded the library of possible drum elements.
• Refactored the library of drum elements into separate classes since it was getting way too long.

Back to top

04/11/2018

• This week I made a whole new system for creating drum beats. Before, I was making beats one or two bars at a time. I realized this was going to take way too long and would take up a huge amount of code since a lot of different beats are actually only slight variations on other beats. So the new drum builder works only one 1/4 note beat at a time. It takes four individual beats and puts them together to get the main drum beat. I like this system more b/c I can use it to make both main beats and drum fills, though I haven't given it instructions for how to build a fill yet. This system is also much better coz it makes it easy to switch to a closely related beat for a new section of the song. For example, tons of rock songs keep the same kick/snare pattern for the chorus, but switch from the hi-hat to the ride for the general time-keeping, then go back to hi-hat for the verse. My program now does this.
• Right now this is only happening on 4/4 rock songs. Next week, I am going to get it to make custom drums for any genre and all of the usual time signatures. I also need to get it building drum fills. I will do that using the same method. It will grab 4 or 8 beat-long drum fill type elements to establish its basic fill. Then that will be the basis for other fills, which will just introduce little variations.
• Another method that I thought about but decided against was making more than one track chunk to deal with rhythm. This would be nice since you could deal with only the hi-hat pattern on one track. Why this sucks is if you open the MIDI file in another program, you def want to see all the drums in one track. You don't want to be clicking back forth between the track that contains only the kick, only the snare etc.

Back to top

04/14/2018

• Lots and lots of fixing little mistakes in all the classes. Basically, I wrote everything not thinking about what would happen if the program gets run several times back to back. So then when I made the music 'game' start making more than one song at once, everything started going wrong. This is now fixed. I have a function that resets the many ArrayLists, booleans, and other important variables after making a song.
• When creating many songs, the program will back up and try again if it his an error.
• The path for saving MIDI files gets saved on the first run and remembered for later.
• Fixed an issue with determining strong beats on 6/8 songs.
• Fixed an issue with MIDI files having the correct tempo.
• Got the MIDI files to start writing in the key signature. Not having a key sig defined didn't make the songs sound bad, since pitch is always given in integer form. However, now when you open up a MIDI song in MuseScore, there aren't a bunch of accidentals everywhere and incorrect sharps/flats. It looks a lot more like how music is supposed to be written.

Back to top

03/21/2018

• Instead of mostly adding new beats/phrases/basslines, work was started on an interface for the user to input the info needed to generate a song. First, I made it so that the input came in from Scanner, but then a day later I changed my mind and decided that dialogue boxes would be better. Here's my reasoning: if a user misspells something (like an instrument name) in scanner or makes some other simple typo, it would make the program not work, but if the user is forced to pick things from a dropdown menu, there is no room for user input error.
• I made some code that will randomize all of the needed inputs to make a song. I will simply copy this randomizer but narrow down the options for making random songs within a particular genre.
• I tried to get the GA to look at 3-note chunks, but this turned out to be really hard. Like you said to me earlier, the GA's weakness is getting it to stumble onto specific combos. It has no problem looking at note-to-note and giving it a grade, but getting it to hit specific desired note-note-note combos was just simply not happening. So this was a dead end.

Back to top

03/14/2018

• Java stuff on the website: parts 6 and 7
• Made some drum fills and a few more beats.
• Made changes to how the melody GA works to have it choose leading tones and out-of-key notes less often.
• Messing around with MIDI files in a DAW for better instrument selection.
• Started working on bassline maker.
• Can now make 4 chunk songs.

Back to top