LinkLink service is unreliable

magenta6847193

    I've had a LineLink device for a few months now, and when it works, it seems to work well.  However, like many others, I experience persistent failures in the LineLink service from time to time.

     

    The failure mode is that inbound calls don't get delivered to the LineLink device, but instead are diverted to voicemail.  Outbound calls fail with a "fast busy" or reorder tone.  This is annoying because the device generates a dial tone when you go offhook.  

     

    This failure happens every so often.    Once it gets into this mode, power cycling the LineLink device will resolve the issue.  I also have a Vonage VoIP adapter in my home, connected to the same Ethernet switch and on the same network.  It continues to work just fine.  So probably not a problem with my router. 

     

    My guess is that there's a memory leak in the device.  Or perhaps the registration with the T-Mobile network fails at the far end, but the device doesn't notice some stale registration.  Not sure, there's no external visibility into the working of this box.

     

    I used the be Chief Technology Officer at Vonage some years ago, so I've had some experience building residential VoIP products, and having them operate on random residential networks, behind all manner of crappy routers, cable modems and the like.  I know it's possible to solve this problem, because I've personally done it.  I would really like to have this service work reliably. 

     

    I'm about to set up some automation to power cycle the box every morning at 4:00AM.  But that doesn't fix the problem, it only mostly avoids or recovers from it.  From what little investigation I've done, this seems to have been going on with lots of end-users and for some time.   I don't know why T-Mobile hasn't been able to fix this issue.  From my personal experience when I ran a platform with more than 2 million VoIP CPE devices, we took these types of failures very seriously because lives are at risk since 911 calls probably don't work either. 

     

    Is someone from T-Mobile going to step up and commit to solving this issue?  I opened a ticket and was told it was going to be escalated to engineering, but there was no clear response when I asked about what happens next, and how do I track this issue towards resolution?  I'd be happy to work with someone from engineering to characterize the nature of problem and do some debugging. 

     

    If that's not going to happen, I guess I start calling and asking for bill credits every month, costing you guys money for the service, as well as another customer contact and those associated costs.  And rebooting the thing every night, which is a totally crappy workaround to what should be solvable problem.

      All replies

      • tmo_amanda

        Re: LinkLink service is unreliable

        Hey, magenta6847193!

         

        Welcome to our Support Community and thank you for providing such an in-depth post about what's going on with your LineLink device. Awhile back I worked with an Engineer on a similar issue. The call would be three times then disconnect. It sounds pretty similar to what you're describing but power cycling the unit did not solve the issue. Do you know if there's a specific amount of beeps you hear or is it just a continuous busy signal? While I wait to hear back from you, I'm going to email the Engineer I was previously working with to see if I can get any additional info on a solution. Last question - is the email address associated with your Support Community account the same as your T-Mobile ID? If needed, I'll pass that info along to our Engineers to begin looking into this.

          • magenta6847193

            Re: LinkLink service is unreliable

            When the device is in the usable state:

            • inbound calls don't ring, and are delivered to voice mail
            • attempted outbound calls do not ring; I get a fast-busy/re-order call progress indication after dialing, which sounds similar to a busy, but has a different cadence.  I believe the fast-busy/reorder call progress indication is originated locally by the VoIP adapter, not from a remote media stream.

             

            I'm not sure how long that the fast-busy persists; when I hear it, I just go on-hook and abandon the attempt.  This happens at call origination time; I've not had it occur during a call that's in-progress.  But I don't use that line for very many outbound calls, so I can't really say with certainty if there are any failures during an active call.  Just for clarity; when in this failure state, I don't get a locally (or remotely) originated ring-back call progress indication; just the fast-busy/reorder when I attempt to originate an outbound call.

             

            So far, power-cycling the TA seems to resolve the issue in every occasion.  I just had another failure that I noticed today which required a power cycle to restore.

             

            My email address should be the same as the T-Mobile ID email address.

              • tmo_amanda

                Re: LinkLink service is unreliable

                Thank you so much for getting back to me with more info. I'm going to be 110% honest with you -- I understand the issues you're having, but I don't understand the back end system for LineLink at all (from your past tech experience it sounds like you have a pretty good grasp of it). I forwarded all of the info you've shared with me to the engineer I previously worked with along with your email address should they need to contact you. I will get back to you as soon as I receive a reply from them or within the next 48 hours, whichever is sooner.

                 

                I'm sorry that I don't have immediate answers but I also don't want to brush this off and have you call Tech Support when we may be able to get this handled more efficiently.

                  • magenta6847193

                    Re: LinkLink service is unreliable

                    tmo_amanda, thanks for  your follow-up.  I'd be pleased if you can get my issue in front of an engineer who might be able to work efficiently towards a resolution.  I look forward to hearing back with an update.   I would agree that trying to open a case via tech support might not be the most productive way forward, as I think there's a more fundamental problem at work that won't get resolved via the usual troubleshooting process.

                     

                    Thanks for your help!

              • pastiche

                Re: LinkLink service is unreliable

                I experience precisely the same problem as does magenta6847193, and I have since I was migrated from the old @Home service and equipment to the new LineLink service and equipment.

                 

                Like him/her, while the LEDs on the LineLink equipment indicate all is well, my outgoing calls will periodically all fail with a fast busy/error tone, and my incoming calls will fail and go directly to voicemail.  I can always fix this by power-cycling the device, typically somewhere between once every two weeks to twice a week.  My modem/Internet service seem not to be the cause (its uptime is currently 82 days with no downtime logged since September.)

                 

                I never experienced this issue with the previous @Home service; that device was only ever power-cycled by a power outage. 

                 

                @tmo_amanda, if you’re able to get assistance from engineering to help magenta6847193, I’d greatly appreciate if you could share the solution with me.  I’m happy to help in troubleshooting any way I can.

                  • magenta6847193

                    Re: LinkLink service is unreliable

                    I am hoping to get this resolved.  I'll probably end up inserting a Sonoff Basic Wi-Fi switch on the power supply so that I can more easily power cycle it if necessary.  Perhaps nightly at 4:00am when I'm pretty sure no one will be using it.  From the way it behaves, it feels like a memory leak or similar in the LineLInk device; alternatively, it could be some event that happens upstream that voids the device registrations.  Unfortunately, there's no way to detect the issue until you try to actually put an inbound or outbound call into service, at least that I can tell.

                     

                    When I was at large OTT VoIP operator, our devices would restart themselves upon loss of SIP registration.  Some of that was defense against weirdness in the local residential IP network and CPE router screwing up NAT mappings or other craziness that you wouldn't believe.   A restart, with new DHCP requests had the effect of working around those issues.   It also helped with firmware problems on occasion as well.

                     

                    We had one hard to diagnose problem that turned out to be a memory leak.  What made it difficult is that we couldn't reproduce it in our lab environment.  Eventually, somehow, our support people noticed that the fails seemed to happen for customers that had a printer!  WTF?  Turns out the mDNS service announcements (multicast UDP DNS packets on port 5353) used for service discovery of the printers ended up taking a code path in the VoIP CPE that did not free them.  After some period of time, all the buffers used for inbound network traffic got consumed and orphaned, and the device had to be rebooted.

                     

                    It would be funny if that was the case here, though TI did fix that particular bug in their software stack.  Only took 3 months of intensive effort to figure out, as those devices were in the hands of hundreds of thousands of customers.  I fear that the LineLink product is just an add-on service, and it will be difficult to get sufficient attention to the problem to get to the bottom of it.

                     

                    Maybe I should put the LineLink ethernet switch port on its own VLAN with no other devices to see if that helps.  The problem is trying to measure progress when the problems on show up many days or weeks later..

                    • tmo_amanda

                      Re: LinkLink service is unreliable

                      pastiche, I will most definitely keep you in the loop. I'm sorry this is happening to you as well, but I'm hoping we make some progress thanks to the vast knowledge magenta6847193 is sharing with us.

                       

                      magenta6847193, I haven't received a reply as of yet, but I wanted to give you a heads up that I will be out of the office until Friday. I will reply first thing Friday morning with an update.

                    • tmo_amanda

                      Re: LinkLink service is unreliable

                      Phew! I finally have an update for y'all -- I'm sorry it took so long. I received a reply after I already logged off for the evening yesterday so I'm passing it along first this this morning. The Engineer I've been working with stated that they've already identified the potential root cause of this issue and are testing a new patch from one of our network vendors to make sure it fully solves the issue. He'll update me on the testing and let me know when it'll be rolling out.

                      • magenta6847193

                        Re: LinkLink service is unreliable

                        Is there any movement on the resolution of this problem?  It's been 5 months since this thread started, and surely I'm not the first to be having this issue.

                         

                        I'm presently power-cycling my LineLink device every day early in the morning to workaround the issue.  As a result of this automated daily power-cycle/reboot, I seem to experience reliable service... this leads me to believe that in fact there's some sort of memory or resource leak in the software stack, that over time causes the malfunction.

                         

                        I can power cycle this thing every day since I have a wi-fi controlled switch do it all automatically, but this certainly isn't a great way to treat the as-cheap-as-possible power supply and electronic components in the device.  I worry about this because I used to build and sell very similar devices to customers.  It would really be great if this was a proper fix in the software so these workaround can be discontinued.

                         

                        Just to poke the bear - I sure hope no LineLink customer has to make a 911 call, only to discover their device needs to be rebooted.  It seems pretty clear there's a known issue here, and I sure hope there's adequate attention and prioritization of remediation efforts to assure fundamental, basic functioning of this product that you put into the hands of your customers.

                         

                        If there is some new software that's been rolled out, and there's a belief this problem has been resolved, I'll discontinue my daily automated reboot and test it.  Thanks for your attention to this matter.

                        • magenta6847193

                          Re: LinkLink service is unreliable

                          Have any software or service updates occurred in the last 6 months since I reported this issue?   While my strategy of rebooting the device late every night seems to result in more or less "reliable" abilities to make and receive calls, it's not optimal.  Doing that automated restart tends to lose pending voicemail notifications, I think, which isn't great.

                           

                          I'd be happy to work with someone on the engineering team with this product to resolve this issue once and for all.