In which I reflect upon which Solstice songs have proven rigorously to be the best, as determined by my not-especially-scientific feedback form.
(NOTE: While I won’t be updating this blogpost again, survey responses on the form are still welcome)
Each year I send out a feedback form for the NY Secular Solstice. This year’s is available here (while the feedback is stable enough for me to write up my thoughts, any additional responses are still appreciated). I’m grateful for a pretty strong feedback culture among people who attend. As I write this I’ve gotten 45 responses, out of roughly 180 attendees. This doesn’t reliably attract the sort of person who comes to Solstice and then doesn’t like it or care enough to provide feedback, but still is a pretty decent sample.
25 responses were from people who hadn’t attended a Solstice before, 20 from people who had.
There’s some nuance to interpreting the open-ended questions like “How did you feel about Solstice?”, but when it comes to rating songs it’s simple-ish: each person rates each song “Hated”, “Disliked”, “Eh”, “Liked”, “Loved.”
Here were the raw results of that:
| Name | Hate | Dislike | Eh | Like | Love |
| 01 – Bright Side of Life | 0 | 2 | 8 | 26 | 12 |
| 02 – X days of X-Risk | 0 | 2 | 10 | 16 | 19 |
| 03 – Bring the Light | 2 | 3 | 15 | 22 | 5 |
| 04 – Gather Round | 1 | 5 | 18 | 15 | 6 |
| 05 – When I Die | 0 | 1 | 9 | 17 | 20 |
| 06 – Bitter Wind Lullaby | 1 | 2 | 6 | 14 | 22 |
| 07 – Chasing Patterns | 0 | 2 | 9 | 21 | 12 |
| 08 – Time Wrote the Rocks | 2 | 1 | 3 | 22 | 21 |
| 09 – Bring the Light (reprise) | 2 | 2 | 14 | 24 | 5 |
| 10 – Do You Realize | 0 | 4 | 10 | 17 | 15 |
| 11 – Blowin in the Wind | 2 | 5 | 11 | 18 | 7 |
| 12 – Stopping in the Woods | 1 | 7 | 12 | 16 | 9 |
| 13 – Bitter Wind March | 1 | 2 | 8 | 14 | 21 |
| 14 – Endless Lights | 0 | 2 | 7 | 18 | 18 |
| 15 – Brighter Than Today | 1 | 1 | 4 | 15 | 24 |
| 16 – Seasons of Love | 2 | 3 | 12 | 17 | 11 |
| 17 – Here and Now | 1 | 1 | 5 | 24 | 12 |
| 18 – Here Comes the Sun | 0 | 0 | 7 | 27 | 10 |
| 19 – Uplift | 0 | 1 | 6 | 16 | 19 |
| 20 – Five Thousand Years | 0 | 3 | 4 | 27 | 11 |
The tricky thing is deciding how to weigh those answers, when deciding questions like “should a song be cut for next year and replaced with something else?” or “Does this song deserve top billing when I’m advertising the Solstice?”. I ended up sorting them three different ways, with slightly different results:
Overall Best Liked
The first method is just “which songs were people at least pretty happy with?”. Which songs had the most Likes and Loves, weighted equally, minus Dislikes and Hates, also weighted equally?
| Name | Hate | Dislike | Eh | Like | Love | Best Liked |
| 04 – Gather Round | 1 | 5 | 18 | 15 | 6 | 15 |
| 12 – Stopping in the Woods | 1 | 7 | 12 | 16 | 9 | 17 |
| 11 – Blowin in the Wind | 2 | 5 | 11 | 18 | 7 | 18 |
| 03 – Bring the Light | 2 | 3 | 15 | 22 | 5 | 22 |
| 16 – Seasons of Love | 2 | 3 | 12 | 17 | 11 | 23 |
| 09 – Bring the Light (reprise) | 2 | 2 | 14 | 24 | 5 | 25 |
| 10 – Do You Realize | 0 | 4 | 10 | 17 | 15 | 28 |
| 07 – Chasing Patterns | 0 | 2 | 9 | 21 | 12 | 31 |
| 13 – Bitter Wind March | 1 | 2 | 8 | 14 | 21 | 32 |
| 02 – X days of X-Risk | 0 | 2 | 10 | 16 | 19 | 33 |
| 06 – Bitter Wind Lullaby | 1 | 2 | 6 | 14 | 22 | 33 |
| 14 – Endless Lights | 0 | 2 | 7 | 18 | 18 | 34 |
| 17 – Here and Now | 1 | 1 | 5 | 24 | 12 | 34 |
| 19 – Uplift | 0 | 1 | 6 | 16 | 19 | 34 |
| 20 – Five Thousand Years | 0 | 3 | 4 | 27 | 11 | 35 |
| 01 – Bright Side of Life | 0 | 2 | 8 | 26 | 12 | 36 |
| 05 – When I Die | 0 | 1 | 9 | 17 | 20 | 36 |
| 15 – Brighter Than Today | 1 | 1 | 4 | 15 | 24 | 37 |
| 18 – Here Comes the Sun | 0 | 0 | 7 | 27 | 10 | 37 |
| 08 – Time Wrote the Rocks | 2 | 1 | 3 | 22 | 21 | 40 |
Weighted Score
It seems like “Love” should count for more than a “Like.” How much more? I asked Jacob Falkovich, who runs a blog on putting numbers on arbitrary things for his advice, and he suggested:
“If you’re looking for “the best”, I would make the weighting convex, i.e. Love+Meh > Like+Like. A couple reasons:
1. The scale is bounded, so someone who feels “this is my favorite song ever” can’t go higher than 5.
2. Songs with high variance are more memorable and exciting, like profiles that get high variance on OKCupid. I think the “best” song should be one that stands out, not a milquetoast compromise.”
So I decided to weigh Loves at 2.25, Likes at 1, “Ehs” at 0, Dislikes at -1, Hateds at -2.25. When doing that, the outcome is:
| Name | Hate | Dislike | Eh | Like | Love | Weighted |
| 04 – Gather Round | 1 | 5 | 18 | 15 | 6 | 21.25 |
| 11 – Blowin in the Wind | 2 | 5 | 11 | 18 | 7 | 24.25 |
| 03 – Bring the Light | 2 | 3 | 15 | 22 | 5 | 25.75 |
| 12 – Stopping in the Woods | 1 | 7 | 12 | 16 | 9 | 27 |
| 09 – Bring the Light (reprise) | 2 | 2 | 14 | 24 | 5 | 28.75 |
| 16 – Seasons of Love | 2 | 3 | 12 | 17 | 11 | 34.25 |
| 07 – Chasing Patterns | 0 | 2 | 9 | 21 | 12 | 46 |
| 10 – Do You Realize | 0 | 4 | 10 | 17 | 15 | 46.75 |
| 17 – Here and Now | 1 | 1 | 5 | 24 | 12 | 47.75 |
| 20 – Five Thousand Years | 0 | 3 | 4 | 27 | 11 | 48.75 |
| 18 – Here Comes the Sun | 0 | 0 | 7 | 27 | 10 | 49.5 |
| 01 – Bright Side of Life | 0 | 2 | 8 | 26 | 12 | 51 |
| 14 – Endless Lights | 0 | 2 | 7 | 18 | 18 | 56.5 |
| 02 – X days of X-Risk | 0 | 2 | 10 | 16 | 19 | 56.75 |
| 13 – Bitter Wind March | 1 | 2 | 8 | 14 | 21 | 57 |
| 19 – Uplift | 0 | 1 | 6 | 16 | 19 | 57.75 |
| 06 – Bitter Wind Lullaby | 1 | 2 | 6 | 14 | 22 | 59.25 |
| 05 – When I Die | 0 | 1 | 9 | 17 | 20 | 61 |
| 08 – Time Wrote the Rocks | 2 | 1 | 3 | 22 | 21 | 63.75 |
| 15 – Brighter Than Today | 1 | 1 | 4 | 15 | 24 | 65.75 |
Meanwhile, Daniel Speyer, another person who I trust to have solid opinions on both numbers and ritual, made a slightly less orthodox argument (edited very slightly by me).
Five hundred twenty five mixed up survey responses
Five hundred twenty five survey answers unclear
Five hundred twenty five mixed up survey responses
How do you measure the best song of the year?
In “hated”s? In “disliked”s?
In “meh”s, “like”s and people voting?
By giving words numbers? Taking the mean?
Five hundred twenty five mixed up survey responses
How do you measure, the songs we should sing?
How about “love”s?
How about “love”s?
How about “love”s?
Just count the “love”s…
Songs that we love!
What happens if we just look at the songs people Loved the most, with no other considerations?
| Name | Hate | Dislike | Eh | Like | Love |
| 03 – Bring the Light | 2 | 3 | 15 | 22 | 5 |
| 09 – Bring the Light (reprise) | 2 | 2 | 14 | 24 | 5 |
| 04 – Gather Round | 1 | 5 | 18 | 15 | 6 |
| 11 – Blowin in the Wind | 2 | 5 | 11 | 18 | 7 |
| 12 – Stopping in the Woods | 1 | 7 | 12 | 16 | 9 |
| 18 – Here Comes the Sun | 0 | 0 | 7 | 27 | 10 |
| 16 – Seasons of Love | 2 | 3 | 12 | 17 | 11 |
| 20 – Five Thousand Years | 0 | 3 | 4 | 27 | 11 |
| 07 – Chasing Patterns | 0 | 2 | 9 | 21 | 12 |
| 17 – Here and Now | 1 | 1 | 5 | 24 | 12 |
| 01 – Bright Side of Life | 0 | 2 | 8 | 26 | 12 |
| 10 – Do You Realize | 0 | 4 | 10 | 17 | 15 |
| 14 – Endless Lights | 0 | 2 | 7 | 18 | 18 |
| 02 – X days of X-Risk | 0 | 2 | 10 | 16 | 19 |
| 19 – Uplift | 0 | 1 | 6 | 16 | 19 |
| 05 – When I Die | 0 | 1 | 9 | 17 | 20 |
| 13 – Bitter Wind March | 1 | 2 | 8 | 14 | 21 |
| 08 – Time Wrote the Rocks | 2 | 1 | 3 | 22 | 21 |
| 06 – Bitter Wind Lullaby | 1 | 2 | 6 | 14 | 22 |
| 15 – Brighter Than Today | 1 | 1 | 4 | 15 | 24 |
And the winner is…?
What does it all mean? Parsing this well enough to make good decisions for next year requires some delving into the more subjective comments – did a song fair poorly because it’s intrinsically bad? Or did it have a particular flaw that could be fixed? And which of these metrics do we care about most anyway?
Discussing this turned out to be an entire separate article – stay tuned!