We have a guest article today, from @batsonjay on our Discord. He went through much effort to migrate his extensive Beatport Pro library to Lexicon and preserve as much information as possible.

---

Beatport Pro to Lexicon Migration

I’ve been a DJ since 1998, and over that time moved from mostly vinyl to mostly digital. For the past 10 years I’ve been playing a weekly 2-hour set online at one of two streaming stations (currently Fridays at balearic-fm.com). This constant drumbeat of a set with ~24 songs / week requires structure & workflows to keep my DJ life sane. In preparing for every set I fill a crate with ~50 songs from which I expect to play 24-25. I try to not replay songs I’ve played recently in order to keep my sets fresh. I try to ruthlessly throw away tracks when I know I’m not liking them anymore, though I may not be very good at it - I still have ~10,000 songs in my collection.

Doing this for a decade has required software that goes beyond what iTunes, or Traktor (my DJ software of choice) can do. In (roughly) 2012 Beatport released an application called “Beatport Pro” (BPP). It provided a basic, but powerful (for its time) form of library management (as well as a way to preview / buy music). Sadly, ~5 years ago Beatport sunsetted the application. Happily, Beatport didn’t disable it, so I’ve kept using it.

The key features that kept me on it were its track tagging, simplified track auditioning, and its basic library management. I have invested tons of time using its Mood & Set Time tags for the obvious. But I also overloaded the Mood field by tagging each track with a yymmdd tag for the date I played it in a set. (This latter has been incredibly useful over the years.) After I used BPP to create one or more Crates for a set, exiting BPP would automatically export an iTunes Library.xml file which I then could IMPORT into Traktor for my live set.

But, BPP is a deprecated application, and getting long in the tooth. I became worried that somehow it’s going to fail me. And there are things it doesn’t do that I would like. So I need to move to a replacement. But without an automated way to move all of my investment in tagging & organization, I was “stranded” with it. For a loooong time. I’ve tried other paths out. There have been other management software apps (e.g. Rekord Buddy), but none could fully incorporate my BPP data. (And sometimes they would strand me even further.)

And as time passed without a way to migrate out, I developed “management debt.” Big time. Mostly around file organization, and a creeping issue with BPP not knowing everything. A few years ago I started putting all the new media I purchased on Dropbox to permit me to use Traktor from multiple laptops. However, when I began I did NOT move all my existing media that had been accumulated during early iTunes days, and was only moving those tracks into BPP on-demand as I used them. I had also distributed my media on Dropbox across a set of folders, one each for the source where I obtained the media - Beatport, Traxsource, Bandcamp, Soundcloud, my DJ pool, etc. But this had become a source of problems - I occasionally purchased the same track from multiple sources because discovering if I already owned it was buried across too-many folders, and BPP was somehow not displaying everything I owned. Then I had a moment in 2016 when I accidentally re-imported a ton of tracks into BPP after a laptop crash / recovery, and I’ve had to spend nuisance-time since then merging the tags on duplicate BPP library entries as I go along. 

Lexicon showed up for me as providing real potential to resolve all these issues and more, and replace BPP as my long-term solution. But I saw no way to get my tagging out of BPP. Happily, the Lexicon community is thriving and helpful. I was discussing my dilemma with @G-Force in the Lexicon Discord. With his help writing some custom (Node.js) code, we found a path that enabled the automated migration of my BPP data. 

Even after we got the software & process ready, I put the task off for a while. Finally, the event that triggered me to finally do the work to move from BPP to Lexicon was my purchase of a new Apple M2 MacBook Pro to replace my 5-year old Intel-based MacBook Pro. I wasn’t sure if a migrated BPP would run at all on the M2 Mac. (And it’s not available for download any more.) Even if it did, I was doing a ton of housekeeping in the move to the new computer anyway, so it felt time to do the move. Even better, Lexicon’s sync capabilities would enable me to use 2 computers: my new one for day to day library management & set preparation, and my old one - now dedicated to DJing - to take to live venues.

The process involved the following steps:

1. Consolidate media files in a stable hierarchy. I had major file management debt. So I took the time to rationalize it all, and in the process put everything in Dropbox).

2. Beatport Pro exit. Export EVERYTHING from BPP to Traktor one last time.

3. Import BPP library to Traktor, & repair. Import the BPP contents into Traktor & have it relocate as many moved media files as it can.

4. Import Traktor collection to Lexicon. Overwrite everything to create a fresh Lexicon library identical to Traktor, which is by extension now identical to BPP.

5. Find missing files. Using Lexicon’s awesome tool for same.

6. Import BPP tags to Lexicon. This is the custom software discussed at length below.

7. Convert to Custom Tags in Lexicon. This is so you can use the BPP tags as actual Custom Tags in Lexicon (vs. text strings).

8. Do final housekeeping in Lexicon. Make things nice before you push out the library elsewhere.

9. Sync to Traktor. Use the usual Lexicon capability for this.

10. Sync to iTunes for good measure.

For those who are reading this with general interest, you can probably stop reading here after I conclude with a couple of final things. First, I’m a big fan of thriving digital communities, and the Lexicon community (on Discord) is a great value to Lexicon users. And a special shout-out for the help @G-force has given me along the way. If you’re still considering whether Lexicon is for you, know that you’re not just buying software, you’re buying into a helpful & vital community, which is worth a lot by itself. Second, Lexicon has more than proven its mettle for me. It’s a powerful platform that is already paying off after moving to it. Finally, @Christiaan (the coder behind Lexicon) is fast, good, and dedicated to Lexicon’s success. I’m loving what he’s done. If you’re considering a move to Lexicon, I commend it highly.

——

For those that might want to do this same migration, or are just interested in exploring the process taken, below is ALL the details you need to know. I have several preliminary items.

• First, this process requires you to be comfortable running commands from a macOS command line in Terminal. If you aren’t comfortable with this, you may need some help at one step on the way.

• Second, this involved me paying for custom software to be built. I’m happy to provide this to you on request. Contact @batsonjay in the Lexicon Discord for more info.

• Third, plan on this taking most of a full day - or two, or three. Not because it takes your computer that long; it will take YOU that long to clean up cruft that keeps this from going smoothly. You have library debt. You may just not want to admit it to yourself. Yet.

• Third, this (sadly) ONLY copies the tags for .mp3 files that you have, and not other file types, for reasons explained below.

Consolidate Media

To address the media management deb I described above, I took the following actions:

• Consolidate & reduce the number of folders on Dropbox. One key question was whether to have one big folder, or a handful. The major downside to one big folder is various inefficiencies in having so many files in one folder. I also opted not to have a zillion folders - one per artist, or album, or ….  Those options just felt like they would perpetuate some of the problems I’m trying to solve. In the end, I was swayed to have 6 folders under a single top-level one - retaining the ones for Beatport, Traxsource, Bandcamp, my DJ pool, plus on new one where I consolidated all the media that lived in folders with just a few tracks in them - plus, one for….

• Move all “local” iTunes media files onto Dropbox. I moved all that into a folder called iTunesMedia. In the end, I’ll tell Apple Music app (er, iTunes) this is where everything lives.

• Wait. Because I wanted to have ALL the cloud sync to finish before I proceeded, I left this to sit overnight. Ready to go the next morning.

Beatport exit

I made sure that I cleaned up any old cruft in BPP playlists that I cared about, organized, moved, or deleted any playlists I wanted to keep, plus other miscellaneous housekeeping.

Next, I created a single playlist called EVERYTHING, and added ALL the audio files from BPP into that playlist. (Select “Audio” in the Collection top-left, click the first track in the list, click Command-A (Mac) to select all tracks, and drag the whole bunch into EVERYTHING.)

Then, I simply exited BPP. This (as you know if you use BPP) creates an updated iTunes Library.xml file with the state of BPP at that moment.

Bye bye, Beatport Pro. It’s been nice. I’ll miss ya. But I’m headed to a better place.

Import BPP library to Traktor & repair

If you’re a BPP user, you are already likely using Traktor’s ability to read the iTunes Library.xml file that BPP created. So, open Traktor, and open that (using the iTunes Music icon in the left bar of the track pane).

Then, select the “EVERYTHING” playlist from that section, and “Import to Playlists”. Note: I chose this instead of “Import to Collection” because it both added the tracks to Traktor, but did NOT trigger Traktor to immediately Analyze all those files - which I think the latter does, and which would have taken quite a long time. I’ll analyze tracks as-needed.

Next, if - like me - you did any media file reorganization, use Traktor’s feature to find them. Select the entire library, and right-click -> Check Consistency. Among other things, this causes Traktor to list all the tracks for which it can’t find a file.

When it’s done scanning, click the Missing Tracks tab on the Consistency Check Report dialog, then click the Relocate button at the bottom and select the top-level folder of the new media library, and let Traktor do its thing.

This was 95% successful the first time. It left behind a few tracks that I needed to handle manually (that had been missed in the work to-date). Then I clicked Remove Missing for those which I knew were probably gone. Lexicon can help me later fill in any gaps.

Finally, be sure to import (from BPP via Apple Music icon in Traktor) all the playlists you wish to migrate from BPP to Lexicon (via Traktor). This is important - it’s the only way you’ll get your playlists from BPP to Lexicon. This took me a little while, but I’d already kept BPP & Traktor fairly well in sync, so I got it done in 5-10 minutes.

Exit Traktor. We have a rational Traktor collection now, and it’s time to move to Lexicon.

Import Traktor collection to Lexicon

Lexicon already has great help for this. So I don’t need to say much here. I made sure to overwrite the existing Lexicon stuff I had (which I’d done to play around with Lexicon) and make this be my complete, fresh library in Lexicon. To overwrite your Lexicon library, simply disable the “Merge with current library” option.

Find missing files using Lexicon

It turned out that when I ran Utility -> Find Missing Files in Lexicon, there were a ton of files that were present in my newly-rearranged Media folder hierarchy that I wanted to be sure to have processed. (This is what I spoke of above that I thought I didn’t care about - but needed to) I added them here.

I also took this opportunity to pull OUT of my library those fully-mixed sets of files that had occasionally arrived with a full album / release - which I’d left hanging around and NOT added to BPP, but which were taking up space on my disk and living in my media hierarchy. (I moved them to the separate Library I use at home with my Sonos system.)

Import BPP tags to Lexicon

This required custom code. I hired (and paid) @G-Force to write it. This code moves the Mood & Set Time tags from BPP to the Extra 2 and Extra 1 fields (respectively) in Lexicon.

The key here is that BPP added the Mood & Set Time (and other tags) for a track into each track’s .mp3 file in an identifiable way. So this custom code reads each track’s .mp3 file to obtain the BPP tags, and writes those back to Lexicon. It gets lists of tracks for which to perform this action by asking Lexicon for a track list to work on. This is all enabled by Lexicon’s (awesome) API interface.

Several key notes.

1. The reason this entire process only works for .mp3 files is that it’s the only file format that BPP could put its tags into. There’s no definable fields in .wav, .aiff or other media container formats. (BPP stores those internally, and we can’t get at them.) So, you’ll lose tags BPP had on those types of files. Sad, but we couldn’t get this info out of BPP directly.

2. This code must be run from the command line (in a console / terminal window). You’ll need to be at least minimally familiar with a command line interface to do this. There isn’t MUCH to know, but I don’t want to teach you here how to do it.

3. This code is written to use Node.js. You’ll need to set your laptop up to run Node code. Instructions for this are available online, and omitted here.

4. I did this conversion in small batches, just to be safe. I created the small batches by building Smartlists that incrementally processed all tracks in my library. We did run into some bugs in G-Force’s code along the way, and this allowed me to not have to back-out massive amounts of incorrect results as we discovered them.

The steps you’ll take are:

1. Enable the API on the Lexicon Settings -> Integrations menu.

2. Create a Smartlist in Lexicon that contains some fraction of tracks with tags. I recommend starting with one Smartlist with the following rules. Let’s call this Smartlist “ConvertThese”.

Field: Title Operator: Starts With Value: A

3. LEAVE LEXICON RUNNING. Now, at a shell command line in a terminal window run the custom script to determine Lexicon’s internal playlist number for this (note: Your prompt might be %, or $, depending on what prompt / shell you’re using):

% node ./getplaylists.mjs

In the output, get the “ID” from the line that lists “ConvertThese”

4. Run the custom script to convert the tracks in that list:

% node ./processBPPTracks.mjs –playlist xxx

Replace xxx with the ID of the playlist you obtained.

The converter will tell you when it succeeded - or could not pull any tags because the file format didn’t support them (e.g. .wav, …). (There’s still a few tracks that throw some errors for reasons we haven’t debugged, but it continues to proceed when those show up. Because the count of these was fairly low, I just rolled with the errors and didn’t worry about them. Out of 10,000 tracks, there were probably 3-4 dozen tracks with errors. I can live with it.)

This will copy the BPP-created tags from the .mp3 files in that Smartlist into the Extra 1 and Extra 2 fields in Lexicon.

5. Repeat once per alpha letter.

1. Return to Lexicon and edit the ConvertThese Smartlist and change the “Value” from A to B. Re-run the SECOND of those two command lines above (./processBPPTracks). Note that by re-using the Smartlist called “ConvertThese”, the playlist number you obtained in the first command does not change. If you make a new Smartlist, you’ll need to re-run the ./getplaylists command. Continue repeating this rule change / re-run until you’ve reached Z.

6. Repeat for non-alpha characters.

1. You may have remaining tracks whose names begin with a number or a special character (e.g. “, or ‘, or etc.) If so, repeat the Smartlist rule change once for each of those, until you’re sure you’ve processed all your tracks.

At this point, you will have pulled all your BPP tags (from your .mp3 files) into Lexicon. All the Set Time tags will live in the Lexicon field named Extra 1 and the “Mood” tags in Extra 2.

Conceptually, you could simply have the converter do all the thousands of tracks in your library in one step. I was overly-cautious, and did it in the 25 A-Z, 0-9, … steps here. It took me about 2 hours. There WERE occasional things I wanted to see / fix along the way. I’d do it the same way again - there’s too much possibility for things to go unexpectedly awry.

If you would like to do this conversion, please contact me (@batsonjay) in the Lexicon Discord. I’ll get you set up with this code.

Convert to Custom Tags in Lexicon

In this step, you’ll use Lexicon to read the contents of Extra 2 to find the Mood tags, and convert them to actual Custom Tags as used by Lexicon. This is pretty simple:

1. Select tracks (in some way; see below) that have fields in Extra 2 that you want to migrate to the Tags system. Right-click, and go to Edit -> Recipes -> Import Tags from Text.

2. Set the Field to Extra 2; set the separator to “,”.

3. Run the recipe. This will turn your old BPP tags into Lexicon Custom tags.

Note that I - again - took the conservative approach. I did these again by re-using that same iterative Smartlist approach in the prior section, but with two rule changes:

Field: Extra 2 Operator: Exists

While not entirely necessary, this limited Lexicon to only add tracks to the Smartlist that originally had tags from BPP, and excluded tracks that didn’t.

Field: Extra 2 Operator: Starts With Value: A

I keyed off of the Extra 2 field, vs. the Title field, because of the above exclusion.

One bit I haven’t done: Handle the Extra 1 tags to convert them to equivalent data in Lexicon.

Final housekeeping in Lexicon

At this point, I used Lexicon’s ability to find duplicate tracks. I found a ton. (Man, I paid for all that twice? Grrr.) This reinforced for me that Lexicon is an awesome tool for my use going forward.

I also re-created some of the “Filters” I’d used in BPP using Smartlists in Lexicon. I love the ability to nest them - super-useful to have combinations of AND and OR filters in the lists.

For the record, I left in-place the contents in Extra 1 and Extra 2 that were pulled in from my conversion process. Even though I don’t use them (and instead use the Custom Tags they were converted to), I left them there - for now - in case I need to re-run this again to correct something I did.

This left my library in super-fine shape!

Sync to Traktor

At this point, it’s time to make Traktor know all this.

Following Lexicon’s Sync instructions, I’ll admit that the first time I tried this, I didn’t export the entire library to Traktor; AND I made sure I had a copy of my most-recent Collection (from after Step 3 above) before I started. I first did JUST an sync of a single Playlist. That seemed to work just fine, though, so I took the risk and did a full sync.

Phew - all good. And again, I’m a happy man.

Sync to iTunes

… er, Apple Music App. (WHY, WHY did Apple decide to maintain the same name for its subscription service AND its app? Dumbest move ever. I’m still calling it iTunes here.)

Again, I was slightly reluctant to sync the entire thing first - lest I make a massive mistake. And, in fact I did make that initial mistake! Fortunately, I canceled the process and was able to recover.

Lexicon’s instructions essentially involve asking Lexicon to generate an iTunes Library.xml file, and importing that file into a “fresh” library in iTunes.

So I whacked my old iTunes Library (per Lexicon instructions), and created a new one on launch of iTunes, and started to import the .xml file Lexicon had created. ARGH - EMERGENCY! Cancel, cancel, cancel.

WARNING: Until / unless Lexicon changes the in-app instructions, you should IMMEDIATELY do this within iTunes in the application Settings -> Files dialog:

1. UNCHECK the “Copy FIles to Music Media Folder” option. NOT doing so will cause iTunes to make a full copy of all the music files in your Lexicon library when you import the file Lexicon generated. Bad news; turn this off.

2. UNCHECK the “Keep Music Media Folder Organized” option. NOT doing so will cause iTunes to rename all your media files as it imports them into iTunes. Bad news; turn this off, even though it’s only doing that to the copies it’s making because of #1

3. Just for safety, I set the Music Media folder to the root folder where you’ve got all your media files. This might not actually matter once you turn off the above two things; but better to do this than not (I suspect). (@Christiaan said he’s never had to do / done this.)

Once that’s done, import the .xml file Lexicon generated, and voila - you’ll have a fully-synced iTunes. Very useful if you want to sync that with your phone and use it in the usual iTunes way.

Final result - joy

That’s it. Yes, this is a long process. BUT, you end up in a wonderful, powerful place. Feel free to hit me (@batsonjay) up in the Lexicon Discord if you want to talk about any of this, or get your hands on the custom code required for this process.