ZeroLSTN Planned Database Redesign
This whole weekend I've been working on trying to figure out how to implement wiki-style song metadata updating, aka Community Song Management. Going forward, as more and more music is added to the site, it's going to become a necessary feature in ensuring the the overall library is kept clean and organized.
This weekend, after going through nearly 10 different iterations of implementations, I've come to the conclusion that we're going to need to recreate the merger database with some slight tweaks, specifically the way songs are stored.
Due to how songs are stored currently, it's not feasible to be able to implement a wiki-style system on top. The current system assigns an ID to each song consisting of the current unix timestamp of when it was uploaded. Initially, I wanted to create a new "update" object which would reference a song's ID and provide new information, which the site would then show instead.
Unfortunately, upon spending time implementing this and then trying to query it, the queries quickly got out of hand. For each query for song information, the site had to account for songs that did have updates, and those that didn't, requiring multiple queries for each. This resulted in inefficient table lookups. And, this was of course is all made worse by the fact that the way existing song data is stored cannot be changed by design, as it's all signed with each respective user's private key, on merger sites which are controlled by the community that no one entity holds the keys to.
Several further implementations were tried around until I finally decided the simplest method would be to store the songs in an array in each user's data.json, give each one an ID, and if two song's IDs match, just show the one with the most recent
The idea is that an update is just a new 'song', with the same ID attribute as the original. These can then be grouped together and queried very easily.
Although, one attack against this is someone could simply change their
date_added value as the maximum value allowed for that data type, therefore always being the latest edit. I'm not quite sure how to resolve that one just yet. Am open to thoughts there.
Additionally, if there are multiple copies of the same song with different IDs, then they'll both show up, and while we can edit one or the other, we can't delete one of them. Perhaps we could have an update which simply marks the song as deleted. Thoughts on how duplicate songs would be worked around are appreciated.
So that's the songs update, and while we're redesigning things, I've decided to change the way the merger sites are grouped too. Genres was a good initial idea, but in practice it's often difficult to assign a single genre to a song that may be a mix of many different ones, or a song that changes course halfway through.
I originally planned on being able to assign multiple genres to a song, with the "main" genre being where the song file is actually stored, but if one is uploading a mix of different songs, thinking about which genre is the "main" genre is quite confusing...
Instead, I propose merger sites by grouped by Year instead. Years are unambiguous, even for mixes, where their year would be the time at which the mix is produced. We could either have one merger site per year, or one per a range of years, say 5. One per year may be better as we break the data up as much as possible, which will be especially important as more and more songs are uploaded.
All of the above changes come with a small cost, however, and that is deprecating the old, current databases and starting the entire library from scratch. I'd like to thank everyone who has uploaded music thus far, and assure you that it has been very useful in getting the initial foundation of the site off the ground. For the new databases, I plan to have a native batch uploading tool so that people can simply 'dump' their libraries and get the new databases populated very quickly.
Once it is implemented, ZeroLSTN will switch over to v2.0.0. I'm very excited to see how it will all look once released, and for the future of the site in general. Thoughts and feedback in the comments on the new design are very welcome, and I look forward to hearing everyone's thoughts. Thanks for following along!
TL;DR: ZeroLSTN's merger sites are being redesigned, meaning all music will need to be reuploaded. Should be quick with the ability to batch upload. Songs will no longer be grouped by Genre, but by Year which should be a lot less confusing. We'll also have Community Song Management, allowing anyone to edit any song. I have an initial design for how to do that, but it still has some problems and I'd appreciate any feedback or ideas you might have :)