DIY Multiroom Audio Part 2: Software

In my last post, I described how I figured out how to extract analog audio output from our A/V-receiver and feed it back into a computer. The ultimate goal is to stream that audio output to other devices on our home network, in particular our smart speaker. This second part is about how that came to fruition through software.

The requirements

The basic idea is pretty simple. The analog-digital converter appears as an input device (like a microphone) on the computer, so what I need is a way to

1. turn the data from that input device into a network stream encoded in some common audio format
2. figure out how to tell my smart speaker to play that stream

I already mentioned in the first part that my speaker is a Denon Home 150 which uses Denon's proprietary "HEOS" technology. You can control the speaker(s) through an app that lets you choose from a bunch of audio sources. Apart from a handful of streaming services and a curated selection of internet radio stations, the only two "generic" sources for this speaker are USB (you can directly connect a storage device to the speaker) and "Server".

"Server" actually means a DLNA/UPnP media server, as I found out through the speaker's manual. At this point I had never done anything with this technology, and trying to learn about it wasn't as easy as I'd hoped. For starters, the UPnP specifications are (apparently) hosted on https://upnp.org, which now redirects to https://www.openconnectivity.org/, which in turn is completely defunct at the time of writing. The spec PDFs are still accessible on upnp.org, though you can only find them through external search engines. Weird.

What doesn't help is that UPnP is a really complicated set of protocols, and technically you need to read and understand like 5 different specs to implement a media server. And don't get me started on DLNA. Apparently it's a subset of UPnP with tighter implementation requirements but I can't tell you exactly what it is because that standard is not freely accessible.

So, my preliminary conclusion was that it'd probably be easiest to find some existing UPnP/DLNA server and to somehow hook the audio stream up to that.

The search for existing solutions

There are plenty of options out there for streaming audio from a computer to a UPnP/DLNA device. However, all of these solutions require that the device can be controlled via UPnP, i.e. this kind of software is like a remote. Unfortunately for us, Denon speakers (or perhaps all HEOS-enabled speakers) can only be controlled through the HEOS app. Their UPnP integration works differently: they query a UPnP server for available audio tracks and then let you select through the app what to play.

Thus, what I needed was a server, not a remote. The problem was, as far as I could tell, there was no existing server out there built for my use case, i.e. relaying an infinite audio stream. The primary purpose of most implementations is to serve files from a directory on your computer. The closest thing I could find was the internet radio support in Universal Media Server, which lets you add an audio stream URL that is then served (and presumably can be selected by the speaker) like a regular file to play. However, this still didn't answer my question how to convert audio from an input device to a stream (in which format even?) at a web URL.

Perhaps I could have tried to MacGyver my way out of this with some bash scripts and a handful of existing softwares, but ultimately I opted for writing one piece of software myself that does exactly what I want, i.e. stream audio from an input device via HTTP, and serve the stream in a way that UPnP clients can access it.

Figuring out UPnP

As I said, UPnP is complicated. Instead of trying to understand it all from base principles, I decided to take a shortcut: grab an existing server implementation that works with my speaker and look at the network communication that happens between them. Luckily, there is minidlna which is spec-compliant and happens to be packaged for NixOS. So all I had to do for my testing was this:

echo 'media_dir=/home/johnny/Music' > minidlna.conf
nix-shell -p minidlna --run "sudo minidlnad -d -f minidlna.conf"

My music directory contains a bunch of audio files, and the -d flag starts minidlna in debug mode, showing you the HTTP requests and responses.

The only thing still missing was opening my firewall for incoming connections on port 8200 (minidlna default). The nice thing about UPnP is that you don't need to use a specific port for other devices to find you.

With all that done, it was time to check the HEOS app to see if my server appeared in "sources" section. And indeed, it showed up and I was able to browse it and play the tracks in my directory.

This gave me some assurance that if I simply copied minidlna's behaviour, I could probably make my own UPnP server work with HEOS.

I saved the logs to a file and looked at the requests that were sent back and forth. The sequence, as far as I could understand, works as follows:

1. Server tries to discover all the UPnP devices on the network
2. Speaker sends a request to the server's "root description" (GET /rootDesc.xml), which lists all the services the server supports (among them: ContentDirectory for listing and browsing available files)
3. Speaker requests descriptions for all the services listed before (e.g. for the ContentDirectory service, GET /ContentDir.xml).
4. Speaker subscribes to events from the ContentDirectory service
5. Speaker requests the ContentDirectory service's SortCapabilities and SearchCapabilities (list of attributes by which files can be sorted/searched)
6. When clicking through the files on the server in the HEOS app, the speaker sends SOAP Browse requests detailing which directory to show or what to search for
7. When playing a specific song, the speaker sends a GET /MediaItems/<id>.flac request to get the file (this URL comes from the Browse response from before)

From this sequence it seemed to me that I need 3 components: UPnP device discovery, a ContentDirectory service implementation, and a URL that serves the audio stream (e.g. as FLAC). I didn't want to do this from scratch, so I looked at UPnP server libraries I could use. I found a UPnP library for Java which looked promising enough. I ran the Getting Started example and saw that device discovery already worked without issues – my speaker was found, according to the logs. So that was one requirement down, two to go.

Implementing ContentDirectory

I was kind of dreading implementing a working ContentDirectory service myself – even with the amenities of the library at my disposal – because the spec is surprisingly long and complicated. Then, sort of by accident, I found this package called org.jupnp.support that goes completely unmentioned in the library's documentation but which happens to implement an almost-complete ContentDirectory service skeleton.

The only method you have to override from this class is browse which, judging from the minidlna logs, is the only feature the speaker needs anyway. The idea is simple then: just have browse always return a single result that points to the audio live stream URL (see next section).

This is what I ended up with:

public class ContentDirectoryService extends AbstractContentDirectoryService {

    private final String browseResponse;

    public ContentDirectoryService(DlnaarConfig config) {
        var streamUrl = String.format("http://%s:%d%s", config.getHost(), config.getStreamPort(), AudioStreamServer.STREAM_PATH);
        var streamTrack = new MusicTrack("0$0", "0",
                "Live Stream", null, null, (String) null,
                new Res(null,
                        new ProtocolInfo(MimeType.valueOf("audio/flac")),
                        null, null, null, null,
                        null, null, null, null,
                        null, streamUrl));
        var result = new DIDLContent();
        result.addItem(streamTrack);
        try {
            browseResponse = new DIDLParser().generate(result);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    @Override
    public BrowseResult browse(
        String objectID,
        BrowseFlag browseFlag,
        String filter,
        long firstResult,
        long maxResults,
        SortCriterion[] orderby
    ) throws ContentDirectoryException {
        // ignore browse settings, always return the stream file
        return new BrowseResult(browseResponse, 1, 1);
    }
}

All the nulls in the constructors there are optional metadata that I don't need or want to supply (like a track artist or the duration of the track). The streamUrl is served at a fixed URL by another web server running in the same program.

Responses to Browse requests contain an "inner" response that have the actual result. That is the browseResponse string which looks like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<DIDL-Lite xmlns="urn:schemas-upnp-org:metadata-1-0/DIDL-Lite/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:sec="http://www.sec.co.kr/" xmlns:upnp="urn:schemas-upnp-org:metadata-1-0/upnp/">
    <item id="0$0" parentID="0" restricted="0">
        <dc:title>Live Stream</dc:title>
        <upnp:class>object.item.audioItem.musicTrack</upnp:class>
        <res protocolInfo="http-get:*:audio/flac:*">http://192.168.0.139:1901/stream.flac</res>
    </item>
</DIDL-Lite>

DLNA stuff

The only thing missing for the speaker to find this service was to create a device with that ContentDirectory service enabled. That should have been enough for the server to show up in the HEOS app but... it didn't. I saw in the logs that the speaker discovered my custom server and queried its root description, so that meant something in there had to be missing.

I compared the /rootDesc.xml response from minidlna and the equivalent from my server and found one obvious difference:

<root>
  ...
  <device>
    ...
    <dlna:X_DLNADOC xmlns:dlna="urn:schemas-dlna-org:device-1-0">DMS-1.50</dlna:X_DLNADOC>
  </device>
</root>

This dlna:X_DLNADOC tag was missing from my server. I don't know what it does, I don't know why it's there or what it signifies, but I saw that you can add a DLNADoc object when creating the device details in jUPnP, so I did just that and copied the string DMS-1.50 into it.

And would you believe it, suddenly my server showed up in the HEOS app and I could even click on it to see the "Live Stream" track!

Streaming audio input as FLAC

The last part of the equation is to implement an endpoint where the plain audio stream is served (the streamUrl from earlier). Originally I wanted to stream it as WAV because Java has a native implementation, but then I realised that WAV isn't streamable because it has a "file size" header. Or at least, the javax.sound.sampled API wouldn't let me create a WAV stream of unknown size.

But I'm getting ahead of myself. Let's talk about javax.sound.sampled.

That API is the established way to work with audio in- and output from Java. You can do a surprising number of things with it, like building synthesisers! But I really just want two things: access audio input devices and get audio from them in a format that I can process further.

Finding audio devices

First question: how do we get a list of all available input devices? The answer to this is surprisingly complicated because of some unintuitive API design decisions. In the end, I took inspiration from a gist I found and ended up with this method:

public static List<AudioInputDevice> scanSystem() {
    List<AudioInputDevice> mics = new ArrayList<>();

    var mixerInfos = AudioSystem.getMixerInfo();
    for (var mixerInfo : mixerInfos) {
        var mixer = AudioSystem.getMixer(mixerInfo);
        var targetLineInfos = mixer.getTargetLineInfo();
        for (var lineInfo : targetLineInfos) {
            try {
                var targetLine = AudioSystem.getLine(lineInfo);
                if (targetLine instanceof TargetDataLine) {
                    mics.add(new AudioInputDevice(mixer, (TargetDataLine) targetLine));
                }
            } catch (LineUnavailableException e) {
                System.err.println(e.getMessage());
            }
        }
    }
    return mics;
}

The confusing part here is that all input devices, represented by the TargetDataLine interface, are attached to a Mixer. However, mixers can also hold outputs and non-physical audio interfaces. And, for some reason, when you use getTargetLineInfo() on a mixer, you don't always get target line (input) descriptors, so you have to do another manual instanceof check. The AudioInputDevice thing is just a data class that I wrote to collect these devices.

And, with an extra method based on this data:

public void printInfo() {
    var info = mixer.getMixerInfo();
    var format = dataLine.getFormat();
    System.err.printf("%s%n\t%s (%d ch. %dbit %.1fkHz)%n", info.getName(), info.getDescription(),
            format.getChannels(), format.getSampleSizeInBits(), format.getSampleRate() / 1000d);
}

I was able to make a command to print all available audio inputs to the console. These are the two I get on my NixOS laptop without any extra peripherals:

0: alsa_playback.java [default]
        Direct Audio Device: alsa_playback.java, alsa_playback.java, alsa_playback.java (2 ch. 16bit 44.1kHz)

1: PCH [plughw:0,0]
        Direct Audio Device: HDA Intel PCH, 92HD95 Analog, 92HD95 Analog (2 ch. 16bit 44.1kHz)

alsa_playback is the desktop audio (as far as I understand), and the second input is my laptop's built-in microphone.

Capturing and encoding audio

Getting an input stream with data from the microphon/input device is fairly straight-forward:

TargetDataLine dataLine = inputDevice.getDataLine();
dataLine.open();
dataLine.start();
var inputStream = new AudioInputStream(dataLine);

(Don't ask me what the difference between open and start is – but both must be called before reading any data from the stream)

Ok, but now what? This stream provides us with PCM-encoded samples (at least in all the cases I saw), so basically the kind of data you would find as the frames in a WAV file. This stream can't be sent straight to my speaker because that expects one of the popular file formats (not just a stream of samples). And WAV doesn't cut it for reasons explained earlier.

FLAC seemed like the obvious second choice. I soon realised though that Java implementations are... a bit lackluster. Here's what I found:

• flac-library-java looked very rigorous and complete – however I couldn't find utilities to work with the javax.sound.sampled API, meaning I would have likely had to figure out the transcoding logic myself (which I wanted to avoid if possible). Also, it wasn't officially published on any Maven repository.
• jflac sounded promising: a port of the official FLAC C library to Java. It even has compatibility with javax.sound! But one look at the source code and you realise the encoder is just 2000 lines of commented out C code. They either forgot to port it or never got around to it. The whole thing is over two decades old by this point though, so I wouldn't get my hopes up.
• Finally, java-flac-encoder. As the name says, this is only an encoder, not a decoder, but that's fine by me. And yay – there is a utility method for encoding a continuous AudioInputStream.

In theory, all I had to do now was to pipe the AudioInputStream obtained from the data line to the OutputStream of a HTTP response on request, via this utility method. This would require to create a FLACEncoder with a StreamConfiguration:

var format = inputDevice.getDataLine().getFormat();
var flacStreamConfig = new StreamConfiguration(format.getChannels(),
    StreamConfiguration.DEFAULT_MIN_BLOCK_SIZE,
    StreamConfiguration.DEFAULT_MAX_BLOCK_SIZE,
    (int) format.getSampleRate(),
    format.getSampleSizeInBits());

Then create a FLACEncoder with that configuration:

var encoder = new FLACEncoder();
encoder.setStreamConfiguration(flacStreamConfig);

Then, add an OutputStream that the encoder should write to. For example, assume that I have access to an OutputStream responseBody from the HTTP handler (what's written to that stream gets sent as the HTTP response body to the client who made a request for the stream).

encoder.setOutputStream(new FLACStreamOutputStream(responseBody));

And finally, initialise the FLAC stream by sending the necessary headers:

encoder.openFLACStream();

This sequence is well-documented in the Javadocs of the library, which was very helpful.

Ok, now all that's left to do is to call the aforementioned utility method:

AudioStreamEncoder.encodeAudioInputStream(inputStream, 4096, encoder, false);

The 4096 is just an internal buffer size (bytes), the false makes this thing run on a single thread instead of all processors (for now, let's not introduce parallelism headaches).

Doing all of this, I was actually able to open the stream URL in a browser and listen back to the audio from my microphone right there!

Streaming to multiple consumers

One glaring omission: what I've outlined above doesn't work for multiple consumers at the same time. If one encoding process is consuming the AudioInputStream, there can't be another. There were multiple ways to get around this:

1. Clone the AudioInputStream somehow (probably complicated)
2. Clone the OutputStream somehow (probably easier)
3. Modify encodeAudioInputStream to call multiple encoders instead of just one

On the surface, 2. seems like the most reasonable option. The idea would be to have a single encoder that outputs to a piped stream which my code would read and copy to multiple output streams. The advantage would be that I don't have to touch the encoding logic and that encoding only happens once no matter how many consumers there are.

A nice idea in theory, but a little more complicated in practice. First, I would have to remember to send the FLAC file headers manually (e.g. by creating a throwaway FLACEncoder) for every new stream. Plus I wasn't sure how well this would work given that FLAC frames can contain bits of data from previous or following frames. I played around with this but ultimately opted for the simpler 3. solution (multiple encoders). My use case won't ever have many consumers at the same time so this should be fine.

I wrote a class called AudioInputMultiplexFlacStream that allows me to add "sinks" (OutputStreams) while a background thread would continually read and encode the audio stream for all current sinks. The sinks are automatically removed when their stream is closed. You can look at the result here.

FLAC trouble

So now I could open multiple browser tabs and listen to the audio from an input device as FLAC. Great! But somehow, the speaker still didn't want to play the stream when clicking on it in the HEOS app. At first, I feared that this might be impossible after all because the speaker can only play finite audio streams or something like that. But then I noticed that VLC (it has a built-in UPnP client) couldn't play it either. So maybe something else was wrong.

First, I suspected that my mechanism for sending the stream (or metadata) was at fault. So, instead of piping mic input via AudioInputMultiplexFlacStream, I tried to send a FLAC file from disk instead. And that worked! Both the speaker and VLC could play it. So the UPnP and streaming part of my code was fine and the problem was with the FLAC encoding (which was confusing, because Firefox had no trouble playing the stream).

To troubleshoot, I decided to download a few seconds of my mic FLAC stream and compare it with the FLAC file that worked. To do this properly, I used the okteta hex editor and the FLAC spec as reference.

I went through the two files byte-by-byte, comparing headers, and everything looked fine until it came to this part of the streaminfo metadata block:

Data	Description
...	...
u(24)	The minimum frame size (in bytes) used in the stream. A value of 0 signifies that the value is not known.
u(24)	The maximum frame size (in bytes) used in the stream. A value of 0 signifies that the value is not known.
...	...

The bytes I found at this position in my stream FLAC were FF FF FF 00 00 00. In other words... minimum frame size more than 16 megabytes, maximum frame size 0 bytes. This obviously didn't make sense. Either they should both be 0 or the minimum should be smaller than the maximum.

Out of curiosity, I swapped those values (00 00 00 FF FF FF) and tried playing the resulting file in VLC. Suddenly, it worked!

I dug into the encoder implementation of the library and found this:

    /* minimum frame size seen so far. Used in the stream header */
  volatile int minFrameSize = 0x7FFFFFFF;

  /* maximum frame size seen so far. Used in stream header */
  volatile int maxFrameSize = 0;

The intent behind this was to determine the minimum and maximum frame sizes during encoding, and then to put the correct values into the headers retroactively, after the entire audio stream has been encoded. The problem, of course: that last part never happens in my code because the audio stream is infinite.

Unfortunately, these initial values are what the encoder library sends when first opening the stream. What it should do instead is send a bunch of zeroes ("unknown"). I forked the library and made this tiny change myself.

Suddenly, everything worked! I was finally able to stream the live audio from my microphone to my smart speaker.

Wrapping up

I published my code on Codeberg. It has a CLI for listing all available devices and running the application with custom configuration options. It may still feel like it's all held together by duct tape, but it works. I'm still working on packaging it as a Nix flake so it can be included in NixOS configs. This might also be useful for people who want to build a Docker image.

With the hardware setup from last post and the custom software in this post, we're now able to listen to the stereo audio in our living room from anywhere!