Proposal for new Retro Music Container

Discussion about Amiga / C64 / Demoscene music and the ExoticA music archive. Favourite music, new rips, musicians, demo sounds, audio software. It's all welcome here.

Moderators: XtC, BuZz

Post Reply
User avatar
Posts: 628
Joined: Wed Jun 12, 2002 6:26 pm
Location: Rossendale, England

Proposal for new Retro Music Container

Post by XtC » Tue Mar 08, 2011 10:54 am

Ey up!

shd, author of UADE, is currently working on a new container for Amiga music.

In essesence, the idea of this container is to make it easier to play Amiga music using the UADE engine by ensuring that the correct format is detected by UADE and that the music file is played as it was in the actual game or demo that it came from.

I'm deliberately keeping this brief to allow shd to provide a deeper exaplanation and method...

Posts: 99
Joined: Thu Jan 16, 2003 12:52 pm
Location: Tampere, Finland

Re: Proposal for new Retro Music Container

Post by shd » Tue Mar 08, 2011 11:49 am

Hello everyone!

I've been writing uade for a rather long time. Primary problems with the project are these:

1. Support for different Amiga music formats
2. Plugins for different music players (e.g. audacious, rhythmbox, deadbeef, foobar2k, ...)

This post concerns the problem two, the support for music players. Detecting different music formats is a difficult problem
due to following factors:

1. Large number of different formats
2. Non-existent or bad magic markers in those formats
3. Over 200 file extensions used with Amiga formats -> the windows based file extension approach for detection is not usable. This is made worse by the fact that some music player APIs do not support multiple file extensions for a player plugin, and it would not make sense to duplicate 200 instances of uade plugin into a player ;)
4. Detection with emulation is slow: adding a directory of songs should not take a long time. Also, the most common case for music player users is to add files that are _not_ in any Amiga format. Amiga player plugin should therefore not slow down the most common file formats (mp3, ogg, ...)
5. Subsong detection may require emulation -> slow
6. Song length detection requires emulation and learning (uade actually creates a database of playtimes on-the-fly as it learns playtimes for songs)

The solution for these problems is a new file container, like psid container for C64 music. The new container tries to solve the compatibility with existing players with following properties:

1. All playable files are named ".rmc" (short for retro music container)
2. There is a unique and efficient magic marker inside the file that determines a file is an rmc file: 12 bytes in the beginning of each file: 'l9:rmc\x00\xfb\x13\xf6\x1f\xa2' (Python syntax)

It will be easy and fast to determine playability from a file or a stream.

3. All playable files are visible as a single file, unlike multi-file Amiga songs. Multiple Amiga files may be wrapped inside the rmc file.

4. The rmc file may, optionally, provide the name of the actual Eagleplayer that is used to play the songs. Or, it may provide the actual Eagleplayer plugin to play the song. I would hope that as little information as possible is put into rmc container to avoid complexities involved with maintaining song collections.

5. The format is two-directional. One tool converts Amiga files to rmc file, and vice versa. No information is lost in the transformation. This also tinkerers to maintain and tweak their collections.

6. The container is made indexable for sites like exotica. Each file consists of "file marker", "meta" and "files". Meta is a dictionary of attributes that related to the file. It may contain arbitrary data. Here's an example in Python dictionary format:

{'platform': 'amiga',
'subsongs': {0: 120000, 1: 60000}, # two subsongs, 0 plays for 120 secs, 1 plays for 60 seconds
'title': 'Foo',
'author': 'Bar',

The only two keys that are required are 'platform' and 'subsongs'.

Subsongs are listed in order to be able to rapidly expand different subsongs into the play list with their proper playtimes.

'platform' is 'amiga' for now, but we may addiotionally export the whole STIL database automatically into rmc format with 'plaftorm': 'c64'.

7. The format uses bencode standard from BitTorrent. It is a light-weight, well-supported, binary-safe and typed serialization format:

Right now I'm writing an rmc wrapper on top of uade and bencode. I have a first demo/evaluation version coming that you can evaluate and give me feedback.

The idea is partly that we abandon the idea that songs are ripped and then never modified again unless new rips come. The new model is to maintain a collection of songs, each of which is modifiable by anyone. The community and/or individuals should maintain these collections.

Note, I've resisted the idea of a new container for years, but it has come to this because I have not found an efficient and usable way to port uade to different music players.

There is one big open question I would like you to think about. We should have a namespace for different Amiga formats. Currently the namespace that uade uses is a set of filenames of eagleplayers that come with uade. But it is not necessary to continue this tradition. Despite this, we need both file format and subtype. Like this:

'type': 'ProTracker',
'subtype': 'protracker-1.0c'
'timing': 'vblank'

In boundary cases we need to include the actual eagleplayer with the song file because some songs require special tweaks in players..

So, there are tweaking options with formats, and there is the actual name of the format.

I would be delighted to hear your comments on this.

User avatar
Posts: 628
Joined: Wed Jun 12, 2002 6:26 pm
Location: Rossendale, England

Re: Proposal for new Retro Music Container

Post by XtC » Tue Mar 08, 2011 4:28 pm

Ey up!

Firstly, thank you for taking the time to actually work on UADE and this container project. :bow: I rely on the UADE engine for XMPlay - and DroidSound whilst I'm mobile! :)

Based on your description above, I read it that the container will have minimal details focusing on the format of the module, song length, sub-song and sub-song length, and that it will also contain the composer and song title. I also read that one container can host a multi-file soundtrack. Hopefully, I've read that correctly?

How would the container cope with, for example, the Agony soundtrack? An evil example as it contains several formats and several composers.

Will the container contents be compressed? I would suggest that it should be, as putting a .rmc inside a .zip is additional unnecessary parsing for a player to perform. Or do you think that in this day and age that we don't need to compress Amiga or other retro music?

As you may know, we at ExoticA (mainly myself) have been working to put together a STIL equivalent for Amiga music. Since you mentioned that the container would have meta data, would it also contain an MD5 checksum so the player doesn't have to perform this? Our STIL would be indexed on an adaptation of the MD5.

This STIL could also contain information regarding the replay for the particular module and also offer a download facility for the player client to obtain the correct and updated replay.

I'm quite sure ExoticA will be happy to host files in this new container, along with non-container versions for legacy players. We have to keep all audiencies happy! :wink:

Post Reply