Is Real-Time Online Music Making Easy Yet? A First Look at FarPlay and SoundJack LITE

by Ian Howell

23 January 2022

Abstract

The Covid-19 pandemic forced many segments of the music performance and education communities online. This happened at a time that most participants were ill equipped to do so, in terms of both technology and knowledge of best practices. It quickly became clear that simple solutions were not professional, and professional solutions were not simple. Challenges to a seamless user experience remain surmountable, but this means they must be surmounted. This article explores two apps in their public beta phase, FarPlay and SoundJack Lite. While challenges remain to using high quality, peer to peer connections between musicians, these two apps represent a significant move toward ease of use and quality of experience.

License

This work is licensed by Ian Howell under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

Table of contents

• Introduction, or a brief history of (this incredibly challenging) time

• Falling short of the ease of FaceTime

• Intrinsic challenges, or things for which there is no (and may never be a) big green GO button

  • Routers, or things your teenager can do that seem impossible to you

  • Windows OS is generally bad at handling audio

  • You Must Choose: Highest possible audio quality or lowest possible latency

• FarPlay and SoundJack Lite

Introduction, or a brief history of (this incredibly challenging) time

From the start of the Covid-19 pandemic, it was clear that those who needed high-quality, low latency telecommunications solutions the most were the least likely to have access to them. In terms of hardware and software, yes, but also in terms of knowledge. Musicians and music teachers, even those in contemporary or commercial music programs, are rarely taught the basics of audio signal chains, how audio is converted from an analog to digital signal, or how to manage that digital audio within a computer. And this is dwarfed by how little we understand about how networks and the Internet work. 

During the pandemic I helped to support my immediate community at NEC to use technology as effectively as possible to teach and collaborate. However, the single most common tech support call was from a user who had forgotten to turn on the phantom power to their microphone. Not some complex adjustment of a hidden buffer; they forgot to turn on the mic. At most schools, the technology departments are in the service business. They record recitals, keep the Wi-Fi running, and make sure that projectors and speakers are set up for classrooms, but at many schools, their main goal is not to educate. For the most part we do not teach people how to succeed with digital audio.

Falling short of the ease of FaceTime

The desire for a simple solution in a box with a big green GO button was strong from the start of the pandemic. Perhaps it is a testament to how well some consumer technologies work that we expected everything to function without much effort on our part. FaceTime on an iPhone is remarkably human proof, but it also bakes in multiple compromises. Apple’s most elegant and user-friendly products embrace those tradeoffs. Their professional products do not. They are as complex and user-unfriendly as anything else out there. But this is because professionals take the time to learn their tools. 

Those teaching, studying, and practicing music during the pandemic experienced a strange mental split. Their violin may have cost thousands of dollars and they may consider the value of putting in years to understand it obvious, but no one wanted to learnhow to make online collaborations work well. People who had spent a lifetime working at the mind-bogglingly complex task of playing a piano felt that they would never be able to learn how to use technology more complex than their phone. 

What is the next phase of low-latency, real-time online collaboration tools? Are we to the point that a range of options exist to serve both power users interested in professional features, and also novices who just want to make a connection and play? Perhaps. At minimum, with the public beta phases of two new options, we are closer. This multimedia essay will explore FarPlay (farplay.io) and SoundJack LITE (soundjack.eu/sjl). They offer different features and may be appealing to different populations (even different students of the same teacher, depending on individual limitations). Both are still in development, and I suspect this article will be out of date by the time I publish it. But both work now, are free to use, and everyone should know about them. 

Intrinsic challenges, or things for which there is no (and may never be a) big green GO button

Before walking through what these two options can do, it is important to point out a few absolute limitations associated with real-time collaborative apps. Anyone who suggests that they have completely solved these problems is most likely trying to sell you something. I will briefly unpack each issue below, but they are: (1) no matter what, some routers will try to block your data, (2) Windows OS does a terrible job at handling audio at a system level, and the best (fastest/highest quality) experiences require external hardware. (3) the speed of your connection (latency) and the quality of the audio tend to vary inversely. 

Routers, or things your teenager can do that seem impossible to you

Your home router, sometimes the same as your modem and sometimes not, serves two functions for your home network. (1) It takes a single stream of data between your house and Internet provider and splits it so that all of your internet enabled devices can connect to the outside world, and (2) it protects all those devices from the outside world. Unfortunately, the way in which it protects you is problematic for the fastest data transfer protocol used by low latency apps (UDP packets sent directly between computers). In the majority of cases, neither FarPlay nor SoundJack require changing settings in your router. This has actually always been true, even for the fully featured version of SoundJack. However, please do keep in mind that you will likely always encounter someone you want to work with whose router will be a problem. This could be because their home router’s security setting is set too high or because they are trying to connect from a more complex network (common on college campuses). Without purchasing a custom router with an open firewall by default, or manually creating what is known as a port forwarding entry in your router (which sounds harder than it is), no one can guarantee that everyone will always have access to the fastest possible connection using a peer to peer app. Most people most of the time, yes.

Windows OS is generally bad at handling audio

I know that many people, especially those who have purchased Windows OS instead of macOS computers, argue that anything a Mac can do can be done with Windows for less money. Maybe this is true for some aspects of computing. For low latency audio applications this is absolutely not true. This is because Microsoft abdicated the responsibility to build professional audio drivers into the operating system. Yes, the default audio drivers in windows function, and in an app like zoom, a built-in microphone on a Windows laptop will transmit your voice. However, it does so with several limitations. (1) the default audio drivers do a worse job of digitizing the analog sound wave. This means a potential loss of fidelity. (2) the default audio driver may not make full use of the dynamic range available in digital audio. (3) the default audio driver does not work at low latencies.

The process of converting an analog sound wave into digital audio is a process of representing a continuous change in pressure with a series of data points. Digital, in this case, refers to these separate points, much like the separate fingers on your hand are called digits. In digital audio, these data points are called samples. The built-in drivers on windows require that you accumulate a large number of these samples in a buffer (think of a buffer as a waiting room for digital information) before that larger chunk of samples is sent off to your application. This adds time, as you are literally waiting for new samples to fill up the waiting room before they are sent off on their way. The default windows drivers require a large buffer, which adds a significant delay.

Industry has come up with a solution for this issue and most professional or prosumer audio interfaces designed to work with the Windows OS offer what is called an ASIO driver. ASIO drivers do function at extremely low latencies. However, two problems persist. First, not all devices, especially built-in microphones or USB microphones, offer the option of an ASIO driver. Second, one of the core features of this type of driver is something called exclusive control mode. That means that if you’re ASIO audio interface is being used by one app on your computer, it will not function at the same time with another app. Provided you have a nice enough audio interface, there are workarounds for these challenges. But they are absolutely workarounds and require troubleshooting and maintenance.

By contrast, the default Audio drivers on a macOS Computer are professional quality. This means that if you use the built-in mic or a USB microphone and the manufacturer did not write professional ASIO drivers for it, you will automatically be able to reach lower latencies on the macOS computer than you could ever get on the Windows OS computer. Can a Windows OS computer be set up to meet or exceed the audio latency and quality of a macOS computer for less money? Maybe. But not for a casual user.

You must Choose: Highest possible Audio quality or lowest possible latency

I mentioned above that low latency apps use something called a buffer, or a waiting room for digital audio samples. These apps all function along these lines. Typically, an app uses several buffers. Each could increase or decrease Latency. Additionally, we must account for the inherent latency of the network that connects the user’s computers. The smaller the size of the buffers, the faster the process that replenishes those buffers. More processes within your computer per second requires greater computing power. No matter what, the faster you attempt to run any low latency app, the more likely that the computer will slip. When it fails to keep up with the processing demands, you will hear clicks, pops, or garbled sounds. there is always a setting that is as fast as the connection can be made and still sound perfect. There is also more than one workaround that allows you to capture high fidelity audio at a higher Latency while simultaneously collaborating at low latency. More on that below. Some users will always choose to accept occasional pops and clicks in return for lower latencies. We should not be confused about the purpose of these apps. The fact that the settings allow one to create a slightly degraded sound is a feature, not a bug. Do not compare them directly with apps designed to only transmit perfect audio at higher latencies.

FarPlay and SoundJack Lite

Both FarPlay and SoundJack Lite offer exciting and relatively simple ways to connect online in real time. They are both feature rich, despite their simple user interfaces. They both remain under active development, and I would expect more features and potential bug fixes to roll out over the coming weeks. Which is right for you? Honestly this may depend on your circumstances. Although SoundJack does offer the option to port forward your router, both SoundJack Lite and FarPlay are intended for a user base who does not want to have to think about it. You may find that one app allows traffic while the other is blocked, and this may differ between collaborators. Assuming that you have a normal home router and either a macOS computer with or without an external audio interface, or a Windows OS computer with an external audio interface (with ASIO drivers), I would expect one or both of these apps to allow you to connect to a similarly equipped collaborator in a short amount of time. 

Please watch each of the following videos for a first look at each app and see the table below for a feature comparison matrix. These are not yet user-proof, and still fall short of the ease of an app like FaceTime. However, these are elegant steps toward that goal. Stay tuned, and check back for feature updates. 

Feature List