CellMapper

The final leg of my triangular approach to improving the quality and reliability of our internet was moving off Verizon into Google Fi.

We were on Verizon because an important part of the extended family was on it and calling within the network was free.  I mean, like 10 years ago.  And 10 years ago Verizon probably worked a bit better in our neighborhood (2G?).  But over time it got worse and worse until it would take minutes to get a text message.  Luckily with Wi-Fi calling it was not so much of a problem - except then Comcast went bonkers and we couldn't depend on Wi-Fi calling either!

Ok.  So big deal - switch providers - how hard can it be?  Well, since I also had the T-Mobile hotspot (later replaced with Starlink) I knew that T-Mobile coverage worked, albeit poorly.  But more research was required!

Enter:  CellMapper.

T-Mobile (which is what Google Fi uses in our state) had way more cell towers than Verizon.  They are also 5G towers - that's why there are so many.  So even though we are in a cell-shadow even with T-Mobile, it's not as bad.

I couldn't really test T-Mobile without getting a T-Mobile phone (because had locked Verizon phones) so it was a risk to switch carriers, but when bad is bad (our Verizon coverage), how much worse could it get?

(My wife says:  "Never challenge worse.")

Google Fi is also 50% of the cost of Verizon!  That's nice.

Anyway, moral of the story is:  Check out CellMapper!  It's awesome.

Also, speaking of legs and triangles:

BTW - if you switch cell providers for you and your family, move your phone LAST.  I thought I would be a helpful guinea pig and move my line first - but my line is the billing line!  Once I moved it, it was much harder to login to the Verizon site!  So do yourself a favor and move your line last.

The Speed

My original hypothesis was slowing down the Xfinity modem would make less use of the higher frequencies (which are more subject to interference) and my problems would be solved.

As it is, I think, from something one of the techs told me, that slower modems are just configured to run more slowly, and still use all the frequencies.

But!  I wanted to get a speed reduction from Comcast because my modem used to work before we bumped it up to gigabit.  It wasn't a pretty process.  They kept saying they had me on a new, cheaper, and slower plan, but it didn't work, and there was no confirmation email.

Once I mentioned Starlink I got a real tech on the line who configured the modem (and my billing plan) to slower (and cheaper).

I bought the slowest plan they had - 300 megabits - and then a few months later they cranked it up to 400 megabits for the same price!  So I'm at 400 megabits, which usually will go up to 440.

I suspect since the bad connector was fixed that I could go back to gigabit ... just not feeling like taking the risk right now.

One benefit of getting the slower plan (because Comcast screwed up) was my premium channels went away.  So Comcast cut the cord for me.

Thanks Comcast!

[2023-11-14 - back up to a gigabit and things are working!  My previous poor performance was caused by a single bad connector that was out of site in a bush.]

The Modem

As I started to concentrate on my Comcast issue I felt like all of my problems started when I upgraded to Gigabit service.  Previously it was ... 500 megabits?  I don't remember.

One thing about faster modems is they use higher frequency bands to push more data down the wire.  I thought, hey, maybe, for whatever reason, the infrastructure, somewhere along the way, can't handle the increased frequencies.  (Turns out I was (almost) right - it was a connector that couldn't reliably handle the higher frequencies.)

So I thought, a relatively cheap experiment would be to get a cable modem that has limited speed - one that can't do gigabit speeds.

I bought a Netgear CM1150V for a hundred bucks.  It was only rated to 400 megabits by Comcast, but it turns out it could easily handle gigabit speeds.  Oh, and it had to handle VOIP, which the CM1150V did fine with.

Well, shit.  It's still fast.  For some reason I did a web search for CM1150V, probably looking for the manual, and discovered this awesomeness:  A script for scraping the error rates of every frequency band and plotting it!

I downloaded it and I was off to the races.  

Here's a graph of the data it collected over a few days:

That's a lot of errors!

So my hypothesis about the errors being worse at higher frequencies - wrong.  But boy that's a lot of errors.  And they were worse in the morning for unknown reasons.

Now somewhere I got the idea, I think from the author of the modem scraper, Harold Holm, that if the modem gets too many errors it reboots!  So that's what was happening.  Which explains why the big outages lasted as long as it typically would take me to reboot it manually.

I get the logic of defensive programming on the cable modem - hey, if there are too many errors, maybe rebooting will fix it! But even with all those errors it worked - until the modem itself decided there were too many errors and rebooted.

Oh my.

I'm back on the Comcast modem, because I wanted to have all their own parts online when the tech came out to analyze my system.  And it turns out that for not much money you rent the modem and also get unlimited internet.  So I need to keep the modem anyway.  

Ideally I would hook up the Netgear modem again, and scrape it for more data, and see how I'm doing on errors, but since it's all working, why bother?

Comcast - one tiny error

After two years of my family and I getting five-minute internet outages once or two (or three times) a day, I finally decided to put the energy into fixing this problem.

I decided to have Comcast come out until it was fixed.  What happened twice in the preceding two years was a technician would come out, replace a lot of connectors, maybe put in a filter that made things worse, put in a signal booster, measure things, declare it fixed, and leave.  Generally the internet service was great except for these outages.

And now, in July 2022, having made up my mind to just keep calling Comcast until it was fixed ... it was fixed on the first visit.

I believe I mentioned to the Comcast phone rep that I had Starlink as my backup.  Suddenly several people at Comcast were interested in my case and talked to me on the phone.  I talked to a more senior person, and he told me, don't hold back, show the tech everything, walk the neighborhood, whatever.  Normally I feel a little bad telling someone how to do their business, but now I was empowered!

The tech that came out did a terrific job of replacing connectors and simplifying the setup - my MOCA network worked again.  He measured things and showed the measurements to me.  Everything looked great just as it always had.  We walked over to the distribution point down the street a bit, and which doesn't fully close anymore, and is kind of filled with dirt, and talked about how rain affects it.  The tech said he had seen worse and it wasn't the problem.  We went across the street to the look at the new fiber drop, which is a disaster:

and

The tech was pretty sure that wasn't Comcast fiber.  That's another story.

He cleaned up the junction box at the house:

Before

After

That's much nicer than the mess that was in there before.

But I thought ... self, you are empowered.  Maybe ... poke around a bit.

So I traced the cable from that box to where it goes into a big, beautiful conduit that goes under the neighbor's yard and driveway.  The origin of that is another story.

And as I looked ... what to my wondering eyes did appear but another connector that had been there for years!

It had been hidden in these bushes.  I pointed it out to the tech and mentioned that it didn't look like the same waterproofing he had been doing to the other connectors.  He opened it up and said, "Eww."

It was full of ... green stuff.

He replaced it, waterproofed it, and now everything is great.  One frickin' connector.  Auggh!

As to the specific cause of the five-minute outages ...

That is also another story.

Backup?

Why do I have a backup internet?

When we moved into our house way back in 1995 I had a permanent dial up connection to WorldCom or some such.  It was not supposed to be a permanent connection but nobody was using the local connection point too much, so the vendor didn't care, and so I was basically dialed up 24/7, and using Windows connection sharing to get the internet through the rest of the house.

Then one day enough people were using their service so they suddenly cared and told me I couldn't be dialed up 24/7.  Luckily just then DSL service came to our neighborhood.  But just barely as we were at the outer reaches of how far those telephone lines could carry a signal. But it worked and we went from sort-of 56K on the modem to an actual 64K.  Woohoo!

Then our small local cable company laid fiber in our neighborhood - not "FTTH" (Fiber To The Home) but fiber to distribution points that then came into our home over the cable lines.

Then Comcast bought that company many moons ago and we've had Comcast every since.

Comcast gets a bad rap but they've been pretty great to us:  generally increasing our speeds for less money - what's better than that?

Shortly before the Covid-times I upgraded to Gigabit Comcast and was happy.

But then the Covid-times hit and I was working entirely from home and I became aware that my Comcast connection would go out to lunch for about five minutes once or twice a day.  

During the first two years of Covid I did try to get it fixed.  First, I switched out the modem in case it was the problem (it wasn't); and then I called Comcast, twice during those two years, and a technicians would come out and rotate all the connectors and try some signal boosters and such.  But none of that fixed it.  It would work really well - very fast - faster than promised - but go out to lunch for five minutes.

Those big thick red lines are five minute outages.  Spoiler alert: it's the same amount of time it takes ro reboot the router.

How it looks now.  Beautiful!

It was not good for me to disappear for five minutes in the middle of an online meeting, so I needed a backup.  I went with T-Mobile home internet which was LTE and about 20 megabits down and 2-3 megabits up - but reliable, except during power outages.

A couple of years went by and I finally decided I really needed to get my Comcast connection fixed - it felt like some distribution equipment was broken in the neighborhood which caused the five minute outages.  

And while I was at it, I wanted to make my backup internet systems more reliable too.

Oh, and we got virtually no signal from Verizon, our cell carrier, and most of our phone calls were carried over Comcast via Wi-Fi calling.  So I couldn't use our cell phones data as a backup.

Things were worst during a power outage - Comcast crapped out even if I had my modem on a UPS; and T-Mobile would drop it's connection although it was on a UPS; and Verizon never worked unless we physically drove out of our cell coverage shadow (never mind about a power outage).

My plan to get Comcast fixed was to call them and have them come out again and again until they figured it out.  If things went like they had in the past, a technician would come out, change the connectors, try a new amplifier or two, and one time they swapped out the modem.  Nothing got better., so I assume going forward, a tech would come out, do the same things, and it wouldn't get better, but eventually they would have to escalate the problem inside Comcast.  That was the plan at any rate.

My plan for T-Mobile internet was to switch to Starlink after my friend HB demonstrated how far along they were with satellite deployments.

And, ironically, my plan for Verizon was to switch to Google Fi, which is T-Mobile in our area.  We get a little data over T-Mobile, but I doubt it will work during a power outage.  We haven't had an outage yet long enough to test my hypothesis.  It's got a better chance than Verizon.

It took six months to do these things and now they are all done!  I have slower but more reliable Comcast (400 megabits down); T-Mobile cell service which works (mostly) at our house; and Starlink.

In succeeding posts I'll describe what went right and wrong with my plan.  It was an eventful transition.

Starlink

Starlink is amazing.  My results are highly variable, with 150 mb down at the higher end, and 20 mb down at the lower end.  I think it's more affected by general usage in the area (since it's a shared resource) as opposed to visibility of satellites but I don't really know.  It's my backup internet.

Think of all the engineering that went into this thing:  reusable rockets; mass production of satellites that can maneuver themselves into position and have highly accurate phased array antennas; and my little dish, also with a phased array antenna that shoots a signal 350 miles into space!  That's twice the distance between Seattle, WA and Portland, OR.

I climbed up on the roof to install Dishy (as it is known) temporarily.

Later, thank goodness, I had it professionally mounted by the same person that installed our roof.

So far it's come through all the interesting weather we've had with flying colors.

If this blog post triggered you because you're mad at Elon, take comfort that SpaceX is run day-to-day by the COO Gwynne Shotwell who appears to be a genius, given the scope of SpaceX operations.

A colleague at work HB put me onto this real-time map of the 3,000 (-ish, as I write this) satellites is what convinced me to take a chance on Starlink.  

We live in an electromagnetic shadow, so cell towers don't work so great, like, at all, well, barely, so my T-Mobile backup internet kind of worked (and got my son through eight weeks of a remote trial where he was a juror), but it never worked during a power outage.  I suspect that cell towers go into a low power mode during an outage so they can provide minimal service for longer.

For Starlink, I'm genuinely lucky.  I wasn't supposed to get a dish until sometime in 2023 and it arrived in 2022.  And my base station ("internet gateway") is SpaceX satellite headquarters nearby in Redmond!  The Redmond office has ten or twelve radomes on the roof which I believe is ... a lot.

5G - fast if you can find it for realz!

T-Mobile has a "test" installation of fast 5G in downtown Bellevue, Washington, on Bellevue Way.

Holy moly!  That's fast!

(Google Fi is T-Mobile in WA.)