[mmichel]

I just found the following Jazz Night in America video in which Christian McBride discusses the open source software JackTrip, and how it makes synchronous remote performance possible (at distances under about 500 miles).

Has anyone tried using this?

 

[datsaxman]

Have not tried it, but would like to.

Great idea if they can keep the latent delay under control and all ...

 

[J-Moen]

I hadn't seen this yet either, but I had seen people doing this type of thing successfully with much longer than 500 miles. I know Daniel Chia has done this type of thing succesfully on his live youtube streams with many musicians.

If I knew any musicians who wanted to jam, I would definitely be trying it out

 

[mmichel]

I hadn't seen this yet either, but I had seen people doing this type of thing successfully with much longer than 500 miles. I know Daniel Chia has done this type of thing succesfully on his live youtube streams with many musicians.

I haven't seen Chia specifically, but what you've seen is almost certainly not synchronous performance, especially if it involves musicians more than 500 miles apart.

Countless groups have put out performance videos that were recorded piecemeal (asynchronously) and then assembled after the fact.

 

[J-Moen]

I haven't seen Chia specifically, but what you've seen is almost certainly not synchronous performance, especially if it involves musicians more than 500 miles apart.

Countless groups have put out performance videos that were recorded piecemeal (asynchronously) and then assembled after the fact.

We can ask him, as he is a member here (bombshell). I believe it is a synced performance though as the live stream wasn't cut to a previously recorded take, and done in part of Q&A where you could interact and ask questions as a viewer.

I was unsure how they could accomplish it though, so it is a viable question. I have seen a lot of advances in processing/handling of data though, even online video games are capable of being streamed to devices versus the old installed and only certain data being processed server side. And all of this with 10-15ms of delay. But 10-15 would have a bigger impact on music imo.

 

[bombshell]

Hey guys, happy to chip in the little that I know from experimenting!

This is the video where I played a live duet with Eric Marienthal on a Youtube live stream:

The time starts at 1:23:00

I was bringing him into my show live through Zoom. The first thing you have to make sure when using Zoom to make music is that you disable any audio processing on zoom. This can be done through the audio settings tab. This allows zoom to get out of the way and allow your audio equipment to pass through the system without any compression or distortion.

We were using a backing track that Eric was playing from his end. So when Eric and the backing track plays, it comes into my ears perfectly in sync. I hear it and play together with that accordingly. To the live audience watching, it sounds perfectly in sync too.

However the one downside is that MY saxophone playing, going back into Zoom and into Eric's ears, is slightly delayed. Now this delay can vary very widely. With good computer, audio equipment and very strong internet through an ethernet cable, this latency can be brought down to a minimum. Eric told me that it sounded like I was just behind the beat. So it didn't affect him that badly. Do bear in mind that he was in LA and I was in Singapore, 15 hour time difference away.

I would say if you were to make music as a duo, it is not impossible, but anymore than 2 instruments will probably be an issue.

I know many companies are trying to find a way to allow for remote live performances with zero latency. I'm not sure when that will happen, but at the moment this is probably the best method there is for now

 

[mmichel]

Hey guys, happy to chip in the little that I know from experimenting!

This is the video where I played a live duet with Eric Marienthal on a Youtube live stream:

The time starts at 1:23:00

I was bringing him into my show live through Zoom. The first thing you have to make sure when using Zoom to make music is that you disable any audio processing on zoom. This can be done through the audio settings tab. This allows zoom to get out of the way and allow your audio equipment to pass through the system without any compression or distortion.

We were using a backing track that Eric was playing from his end. So when Eric and the backing track plays, it comes into my ears perfectly in sync. I hear it and play together with that accordingly. To the live audience watching, it sounds perfectly in sync too.

However the one downside is that MY saxophone playing, going back into Zoom and into Eric's ears, is slightly delayed. Now this delay can vary very widely. With good computer, audio equipment and very strong internet through an ethernet cable, this latency can be brought down to a minimum. Eric told me that it sounded like I was just behind the beat. So it didn't affect him that badly. Do bear in mind that he was in LA and I was in Singapore, 15 hour time difference away.

I would say if you were to make music as a duo, it is not impossible, but anymore than 2 instruments will probably be an issue.

I know many companies are trying to find a way to allow for remote live performances with zero latency. I'm not sure when that will happen, but at the moment this is probably the best method there is for now

Thanks for explaining what you did and how.

I just wanted to clarify that the "synchronous" playing that McBride is talking about in the video involves back-and-forth improvisation (e.g., he shows examples of a pianist improvising with a bassist, with both of them quoting and building off of each others' lines). I didn't watch the live stream, but it sounds as if the reason what you did worked, was that you were effectively playing with Eric (+ backing track) in a way that was more similar to using a backing track than to a "live" musical interaction.

The critical thing for real interaction is the round-trip latency. This latency is fundamentally limited by the speed of light, which means that even if you had the world's best technical setup with zero added latency, the minimum round-trip latency you could have achieved would have been about 100ms: 2 x 14,112 km (LA-to-Singapore dist.)/ 300,000 km/s (speed of light). At a metronome marking of 120 bpm, that's already about a sixteenth note and, again, this is the theoretical minimum, which is far from what you can actually achieve with real-world networking infrastructure and software like Zoom.

That's why the video's authors cap the practical range of JackTrip at about 500 miles for synchronous performance.

 

[J-Moen]

The theory is sound, but it's never just a perfectly direct connection. The main thing to consider here is that these two are not connecting directly to each other, they are connecting to a zoom server, and then data is transmitted from there back to the connectors (and viewers). Doesn't change the physics of it all, but does allow the server to do some latency correction.

There are tons of different methods to handling data - the new primary is server data handling and processing. And there are minds far brighter than mine and probably most of ours coming up with newer methods to create near simultaneous solutions, but all currently consist of some delay.

Even the 500 mile solution with multiple instrumentalists would suffer from delays. Theoretical speeds be damned, the average general latency in the US alone is 28 milliseconds just to connect to local ISP servers (unless you pay for direct fiber). This can add up quick - the best you can hope for is great data handling that's uses some of the best latency correction and post processing.

Daniel's video was impressive how they played together, especially considering the delay, as they definitely played off one another, it was more than you would expect given the circumstances. And this is my normal day job - one of the few areas I feel I can speak as an expert on.

 

[lesacks]

...And there are minds far brighter than mine and probably most of ours coming up with newer methods to create near simultaneous solutions, but all currently consist of some delay.
....

Musing on this, of course, Alexander Graham Bell had a good solution. The telephone.
A real Heath Robinson solution, if old pstn is available, would be for the musicians to play together over a phone call - but to record them individually; and mix the individual recordings together with some synchronization.
That it wate for 5G.
... Or setup your own longwave radio system!

 

[J-Moen]

Musing on this, of course, Alexander Graham Bell had a good solution. The telephone.
A real Heath Robinson solution, if old pstn is available, would be for the musicians to play together over a phone call - but to record them individually; and mix the individual recordings together with some synchronization.
That it wate for 5G.
... Or setup your own longwave radio system!

POTS is a great solution - but many are digitizing these even. Taking the analog signal and putting it over digital line then sending it out via coaxial line versus the beautiful plain old telephone system is the new norm.

I did lots of satcom and radio work in the military, making custom antennas that could launch some pretty awesome signals, but the delay in bouncing these signals around the atmosphere can be rather large.

5G theoretically is near low end fiber speeds. Of course once everyone is connected and using it, the services will attain much slower speeds, probably closet to what LTE was in theory.

In my mind, the fastest would be something that is able to take the latency of each person connecting and provide a delay of that much to each person as they hear it, while mixing it server side - also accounting for delay. The feedback of each playing off each other may be limited in this regard though. This already happens by many services, but maybe not to the highest quality. Hopefully the solution in mind has something unique to let it function - although limiting to 500 miles would mean minimal latency correction.

 

[mmichel]

The theory is sound, but it's never just a perfectly direct connection. The main thing to consider here is that these two are not connecting directly to each other, they are connecting to a zoom server, and then data is transmitted from there back to the connectors (and viewers). Doesn't change the physics of it all, but does allow the server to do some latency correction.

There are tons of different methods to handling data - the new primary is server data handling and processing. And there are minds far brighter than mine and probably most of ours coming up with newer methods to create near simultaneous solutions, but all currently consist of some delay.

Even the 500 mile solution with multiple instrumentalists would suffer from delays. Theoretical speeds be damned, the average general latency in the US alone is 28 milliseconds just to connect to local ISP servers (unless you pay for direct fiber). This can add up quick - the best you can hope for is great data handling that's uses some of the best latency correction and post processing.

Daniel's video was impressive how they played together, especially considering the delay, as they definitely played off one another, it was more than you would expect given the circumstances. And this is my normal day job - one of the few areas I feel I can speak as an expert on.

Indeed, it's a testament to Eric's skill (given that he was the one dealing with the delay) that he could compensate so well for the delay, but it's clear that this did not feel to him like playing synchronously and that he had to work to compensate.

Regarding the JackTrip software, I didn't mean to imply that the 500 mile solution was feasible. In fact, my point was the opposite, that anything greater than 500 miles would certainly not be feasible.

I think that one of the features of the JackTrip software is that it in fact uses "direct" connections (i.e., in the same way that telnet or ssh use direct connections) and does not use some third party server to host and buffer data. Practically speaking, I think it would work well for something like the current pandemic situation: when you have several musicians in the same city/metro area who cannot meet face to face. If you're in a major city, you probably have access to decent internet infrastructure. E.g., I'm currently connected to my router via WIFI and the ping latency to my ISP (via Verizon FIOS) is still only about 9ms. I haven't yet tried JackTrip, but with a decent low-latency audio interfaces and ASIO drivers on a wired connection, I'd be surprised if it didn't work locally with ~30ms latencies for small group stuff.

But I don't know for sure. That's why my original post asked whether anyone had tried the software.

 

[mmichel]

A real Heath Robinson solution ...

Heh, a sure sign that you're British. In the US, the equivalent colloquialism is "A Rube Goldberg solution".

I had to look it up. It's interesting that Robinson, though Goldberg's predecessor, is virtually unknown here in the US.

 

[lesacks]

Musing on this, of course, Alexander Graham Bell had a good solution. The telephone.
A real Heath Robinson solution, if old pstn is available, would be for the musicians to play together over a phone call - but to record them individually; and mix the individual recordings together with some synchronization.
That it wate for 5G.
... Or setup your own longwave radio system!

POTS is a great solution - but many are digitizing these even. Taking the analog signal and putting it over digital line then sending it out via coaxial line versus the beautiful plain old telephone system is the new norm.

I did lots of satcom and radio work in the military, making custom antennas that could launch some pretty awesome signals, but the delay in bouncing these signals around the atmosphere can be rather large.

5G theoretically is near low end fiber speeds. Of course once everyone is connected and using it, the services will attain much slower speeds, probably closet to what LTE was in theory.

In my mind, the fastest would be something that is able to take the latency of each person connecting and provide a delay of that much to each person as they hear it, while mixing it server side - also accounting for delay. The feedback of each playing off each other may be limited in this regard though. This already happens by many services, but maybe not to the highest quality. Hopefully the solution in mind has something unique to let it function - although limiting to 500 miles would mean minimal latency correction.

I appreciate the latency from bouncing signals of the ionosphere - but I kinda had this image of a couple walking past a house with a huge wire loop in the roof, tutting and muttering "musicians, eh!".
... and I'm old enough to remember all the promises of 5g being made for 4g...
Maybe your old enough to remember Asynchronous Transfer Mode and B-ISDN? It that had worked, all this would be a no-brainer. Damn the Internet!

 

[lesacks]

A real Heath Robinson solution ...

Heh, a sure sign that you're British. In the US, the equivalent colloquialism is "A Rube Goldberg solution".

I had to look it up. It's interesting that Robinson, though Goldberg's predecessor, is virtually unknown here in the US.

"Rube Goldberg" - cool! I'll use that next time...

 

[J-Moen]

Indeed, it's a testament to Eric's skill (given that he was the one dealing with the delay) that he could compensate so well for the delay, but it's clear that this did not feel to him like playing synchronously and that he had to work to compensate.

Regarding the JackTrip software, I didn't mean to imply that the 500 mile solution was feasible. In fact, my point was the opposite, that anything greater than 500 miles would certainly not be feasible.

I think that one of the features of the JackTrip software is that it in fact uses "direct" connections (i.e., in the same way that telnet or ssh use direct connections) and does not use some third party server to host and buffer data. Practically speaking, I think it would work well for something like the current pandemic situation: when you have several musicians in the same city/metro area who cannot meet face to face. If you're in a major city, you probably have access to decent internet infrastructure. E.g., I'm currently connected to my router via WIFI and the ping latency to my ISP (via Verizon FIOS) is still only about 9ms. I haven't yet tried JackTrip, but with a decent low-latency audio interfaces and ASIO drivers on a wired connection, I'd be surprised if it didn't work locally with ~30ms latencies for small group stuff.

But I don't know for sure. That's why my original post asked whether anyone had tried the software.

Agreed, and sorry to further the speculation here. Just wanted you to know the encountered physic issues in latency are attempted to be handled by the processing. A direct connection doesn't necessarily make things better. FiOS is indeed a fiber connection, and I loved when I had it in Virginia but am limited more in my current connection. I also have 15ms latency to my ISP.

I have done multiple direct connections via VPN. The protocols for handling a video connection are far more intensive than ssh or telnet but it may provide a decent solution to closeby patrons as suggested. I will see if I can get a musician nearby and will test it with all of my networking gear to see how it works.

 

[JohnInDenver]

I've done quite a bit of testing with JamKazam, and the latency is exactly the issue everyone has pointed out. With two musicians (in our case, within 30 miles of each other) it's often quite workable. But when we add a third musician (in our case, hundreds of miles away) the latency really demonstrates its effect. The sense is that the other musicians are laying back on the time, and the natural reaction is to lay back with them...until a few minutes later (in the song) the tempo has been reduced to a dirge-like crawl. It's comical. In response, I made a conscious effort to keep pushing forward, and encouraged the others to keep up with me. It helped a bit. In the end, we played songs that had big dramatic stops in them, and the guitarist would say "re-sync" to acknowledge that the little pause was enough to get us back to the same moment in time.

 

[AddictedToSax]

Forget latency, think quantum entanglement. :mrgreen:

https://singularityhub.com/2018/10/22/from-quantum-computing-to-a-quantum-internet-a-roadmap/

 

[lesacks]

Forget latency, think quantum entanglement.

https://singularityhub.com/2018/10/22/from-quantum-computing-to-a-quantum-internet-a-roadmap/

The Science article was good... But

Entanglement makes it possible to communicate instantly across arbitrarily large distances in principle

FFS!

 

[mmichel]

I have done multiple direct connections via VPN. The protocols for handling a video connection are far more intensive than ssh or telnet but it may provide a decent solution to closeby patrons as suggested.

That's another advantage of JackTrip (re: latency). It's audio only. I'm guessing that the video part of the collaborations shown in McBride's video were not synchronous but were added in post-production.

Not being able to see each other isn't ideal, but it's something that musicians are very much accustomed to (i.e., in studio settings, while reading music, etc.).

Moreover, our visual systems have substantially poorer temporal acuity than our auditory systems do, so we can perceptually tolerate much longer video latencies.

 

[mmichel]

Forget latency, think quantum entanglement. :mrgreen:

https://singularityhub.com/2018/10/22/from-quantum-computing-to-a-quantum-internet-a-roadmap/

Quantum entanglement is sort of a non-sequitur here. Even if there were currently a practical way to use it (there isn't), it can't actually transmit information any faster than the speed of light.