Comment to the FCC re: Consumer Router Firmware

Update: The deadline for filing has been extended to 9 October 2015, so you now don’t have any excuse for not sending in your opinion.

I filed the following comment on the FCC page regarding the new Proposed Rule that would (ultimately) prohibit individuals from installing third-party firmware (such as OpenWrt, DD-WRT, Tomato, and the myriad others) on their home router.

If you have an opinion on this, you should go to the FCC Comment Page and click the green SUBMIT A FORMAL COMMENT button.

You’re free to plagiarize my note, or take a cue from the Save WiFi page at I wrote:

I recommend that the FCC RESCIND its Proposed Rule, Document number 2015-18402 regarding wireless devices. The Proposed Rule is overbroad, would harm many communities of Americans, and is not warranted by the facts on the ground.

Although the FCC has the power to regulate equipment that generates radio frequencies, this is a heavy-handed rule that could be addressed other ways. Specifically, I am concerned about the ability of third parties to modify and create new firmware for “consumer routers.”

The proposed rule would require that router manufacturers lock down the RF portion of the router to obtain FCC approval. This “lock down” would prevent modification to the radio’s power, frequencies, etc to prevent it from radiating outside the specified limits. This is a laudable goal, but the application of this rule as written would result in undesirable consequences.

In practice, most radio functions are very tightly wedded to all the other factors of the hardware/software. The most likely way manufacturers would likely lock down the RF operation would be to make it impossible to modify any of the code in the routers.

There would be a number of adverse consequences both for me personally, to consumers in the US, and the networking industry. These consequences can be ameliorated by allowing the owners of routers to install their own code.

1) Security of the router. It is well known that vendor-supplied firmware for consumer routers often contain flaws. Just last week, the CERT released knowledge of a vulnerability to Belkin routers. See The ability to install well-tested, secure firmware into a router benefits all consumers. The ability for a person to update their own router on a regular basis (as opposed to many manufacturer’s seemingly lackadaisical schedule) preserves security.

2) Research into the field of computer networking. Non-traditional research efforts (outside academia) lead to real improvements in the state of computer networking. An example is the CeroWrt project that developed the fq_codel algorithm. The result of this multi-year effort was a major advance in performance for all routers. The fq_codel code has been accepted into the Linux kernel and now runs in hundreds of millions of devices. As a member of the team that worked on this, I assert that without the ease of modification of a consumer router to prove out the ideas, this improvement would likely not have occurred.

3) Personal learning environments. Individuals, as well as network professionals, often use these consumer routers as test beds for increased understanding of network operation. Losing the ability to reprogram the router would make it more expensive, if not prohibitive, for Americans to improve their knowledge and become more competitive.

4) I would incorporate all the other “talking points” listed on the Save WiFI page at:

5) Finally, I want to address the FCC’s original concern – that these consumer routers are SDRs, and they must not be operated outside their original design parameters. While the goal of reducing radio frequency interference is important, the FCC has failed to demonstrate that the widespread practice of installing/updating firmware in consumer routers has caused actual problems. Furthermore, the FCC can use its current enforcement powers to monitor and shut down equipment that is interfering.

Creating a broad, wide-ranging rule to address a theoretical problem harms industry and individuals, and is an overreach of the rules necessary to preserve America’s airwaves.

Bufferbloat and the Ski Shop

Bufferbloat is undesirable latency caused by a router buffering too much data. It makes your kids say, “The Internet is slow today, Daddy”. It’s caused by routers and other network equipment buffering (accepting for delivery) more data than can be delivered in a timely way. Bufferbloat causes much of the poor performance and human pain experienced using today’s Internet.

First a story…

Imagine a ski shop with one employee. That employee handles everything: small purchases, renting skis, installing new bindings, making repairs, etc. He also handles customers in first-come, first-served order, and accepts all the jobs, even if there’s already a big backlog. Imagine, too, that he never stops working with a customer until their purchase is complete. He never goes out of order, never pauses a job in the middle, not even to sell a Chapstick.

That’s dumb, you say. No store would do that. Their customers – if they had any left – would get really terrible service, and would never know when they’re going to be served. And you would be right.

The Ski-Shop Router

Unfortunately, a lot of network routers (both home and commercial) work just like that fictitious ski shop. And they give terrible service. These routers have a single first-in, first-out queue for packets. All traffic is queued, regardless of whether it’s a big or small packet, whether there has been a lot of traffic from a particular source, or whether things are getting backed up.

Since the router has no global knowledge of what’s happening, it cannot inform a big sender to slow down, or throttle a particular stream of traffic by discarding some data (in the ski shop analogy, sending away a customer who has another long repair job). The dumb router simply accepts the data, buffers it up, and expects to send it sooner or later. To make matters worse, in networking (but not in ski shops), if delays get long enough, computers can resend the data, thinking that the original packet must have been lost. These retransmissions further increase bloat and delay, because there are now two copies of the same data buffered up, waiting to be sent…

This is the genesis of the name “bufferbloat” – the router’s memory gets bloated with buffers of packets. When the router doggedly determines to send that data, it blocks newer sessions from even starting, and the entire network gets slow.

What’s the solution?

The members of the CeroWrt team have been working for the last three years to solve the problem of bufferbloat. We’ve largely succeeded: the CeroWrt firmware works really well. CeroWrt users no longer see problems with “the internet being slow” even when uploading and downloading files, watching videos, etc. We have pushed those changes into the Linux Kernel, and also into OpenWrt mainline, and now its Barrier Breaker release contains the Smart Queue Management (SQM) fixes.

SQM introduces a new queueing discipline called fq_codel (see and for details) that can detect which flows (streams of data between two endpoints) are using more than their share of the bottleneck link (usually, the connection to the ISP).

SQM divides the traffic into multiple queues, one per flow, and sends packets from each queue in round-robin order. (The algorithm is somewhat more involved, so read the full description for details.) fq_codel also measures the time that each packet has been queued (its sojourn time). If packets for a flow have been in the queue for “too long”, then fq_codel either marks them for ECN (Explicit Congestion Notification), or discards a certain percentage of them, preventing the flow from using more than its fair share of bandwidth on the bottleneck.

Wait a minute – discarding packets? Doesn’t that make things worse?

It does slow the rate for the affected flow, but that is exactly what should happen. If a sender has sent so many packets that they’re building up in the router’s memory, then the router must offer back pressure for that flow by dropping/marking some of its packets.

In the meantime, all the other flows (in their own queues) have their packets sent promptly, since they’re not building up and their sojourn time stays low. This automatically keeps everything responsive: short packets, and those from low-volume flows automatically get sent first. The big senders, whose packets are dropped/marked, will re-send the data, but at a slower rate, bringing the entire system back into balance.

What about Quality of Service (QoS)? Doesn’t that help?

Yes, it helps a bit. If you configure your router for QoS, the router will use that information to prioritize certain packets. Good QoS settings can help certain traffic types by sending it first, ahead of the bulk traffic that’s buffered up. But there are several problems with QoS:

  • It doesn’t solve the problem of overbuffering. The QoS rules allow certain packets to go to the head of the queue. But the buffers from large flows are still there, and will have to be sent at some point. And those buffers will stand in the way of any newly arrived traffic that hasn’t been prioritized.
  • As a corollary, there’s no throttling of the big senders: they don’t get early feedback that they are using more than their fair share of the capacity. If the queue/delay gets large enough, the sender could even retransmit the data, making the overall situation worse.
  • It’s annoying to configure QoS. You have to understand how to configure the router and manually make the changes. This is something that only a network geek could come to enjoy. It’s also a maintenance hassle: if you hear about a new application, it may not work well until you adjust the QoS rules to take it into account.
  • Finally, QoS doesn’t help for the download direction. It can improve traffic being sent from your local (home) network toward the Internet. But if the equipment at the far end at your DSL or cable provider is bloated (and very often it is), then QoS in your router won’t make things any better.

The fq_codel and other algorithms in CeroWrt handle all this automatically. The only configuration parameters are what kind of link you have (DSL, Cable, etc.) and the speeds of those links. You don’t have to adjust QoS settings or make other adjustments.

Is my network affected by Bufferbloat?

Quite possibly – here’s one symptom: If the network works well when no one else is using it (early morning, or late at night after everyone else is asleep), but gets slow when others are on the net, then you are suffering from Bufferbloat. Another symptom is if your voice, video chat, or gaming degrade when others are using the network.

Here’s a more scientific test. The DSLReports Speed Test runs a latency measurement during the download and upload speed tests. (Most speed test sites only measure a few pings when the line is idle.) If the latency gets high on this test, then your router is probably bloated. You can also check out the Quick Test for Bufferbloat on the CeroWrt site for additional information.

What can I do about this?

Three years of network research have paid off: the networks work great at our houses. Our algorithms have been adopted and implemented in the Linux kernel, other operating systems, and an increasing set of commercial network equipment. Our changes have been pushed in to the OpenWRT project. We are making the code available at no charge, and are encouraging all vendors to embrace it.

Regrettably, nearly every piece of equipment with a network connection – home router, ISP headend and DSLAM, phone, personal computer, laptop, tablet, even big commercial routers and switches – needs to have some form of SQM installed. We now have the technology, and it’s simple, but it needs to be deployed.

TL;DR – What you can do about Bufferbloat at your home

  • Consider installing the stable OpenWrt Barrier Breaker firmware on your router. The luci-app-sqm and sqm-scripts packages include the enhancements that we’ve tested and then pushed into the OpenWrt mainline source code. Use the Supported Devices page to find your router, then read the SQM/fq_codel HOWTO. If you don’t want to do all that…
  • Call your router vendor’s support line. With the information from the DSLReports Speed Test in hand, you can mention that the ping times get really high when up/downloading files, and that it really hurts your network performance. Ask if they’re working on the problem, and when they’re going to release a firmware update that solves it. Leave a comment here with their response – I’d love to hear.

An earlier draft of this note appeared in the Bloat and Codel mailing lists. See Latest update 3May2015

More magma…

We have been planning a trip to Iceland (mostly to see the aurora) next year. I am now looking into tours that go near the volcano at Bárðarbunga and Holuhraun as well.

I think that land tours (I would love to stand next to the lava front) are out of the question, since the authorities have closed the area for ~18 miles around.

But there appear to be a couple air tours flying from Akureyri to Bárðarbunga, Holuhraun, and over the Askja crater for a little more than US$300 per person. The view from the planes can been seen in these Youtube Videos… I’m psyched.

A Memorization Tip

I have given out this tip in past years to people in Revels North who’re preparing for the Christmas Revels show…

I found this trick for memorization in a wonderful book, Don’t Shoot the Dog, The new art of teaching and training, by Karen Pryor. It talks about modifying behavior — yours and others. (One way of modifying behavior is shooting the dog. But she talks about lots of other—usually more appropriate—ways to change behavior.) She tells about coaching, teaching, and maybe even making things better in your life. And it’s a fun read…

But back to memorization: The trick is to learn the last verse first, then the next-to-last, and so on.

Why? Because the usual way of learning the verses sequentially is enormously dispiriting. You struggle to get the words of the first verse. After a lot of work, you begin to be comfortable with them, so you attack the second verse. But you’re back at sea. It feels like such hard work. After the second verse, you hit the wall of the third verse. You despair of learning the words at all.

Instead, learn the last verse first. Then start learning the previous verse. After you get the new verse down, you can keep on singing, but they’re words that you already know. You’re drawn along by the familiar material. You feel competent and confident. You feel good.

Gettin’ my magma jollies!

I’ve been following Rei, a blogger from Iceland, who is reporting on the Bárðarbunga/Holuhraun volcano as it continues to erupt. It’s way more impressive/scary than I originally thought. It’s huge, with fresh lava covering 37 square km, or about 2/3 the size of Manhattan, and the magma fountains are shooting 300 feet or more in the air.

Her recent posts have shown a lot of information about the volcano’s progress. The one from last Saturday contains the best view so far of the extent of the lava, and absolutely stunning views of the magma sloshing around. The video is second-to-last picture on this page.

See it here: Rei’s post with great volcano video

For a bit more context, you can read her earlier posts. This map shown on this link tells more of the story. The big volcano, called Bárðarbunga, is at the red dot in the big glacier. The magma has pushed about 70 km (45 miles!) beneath the surface to erupt at Holuhraun, labeled with “Eruption” below.

The good news is that the magma remained underground until it got to a flat sandy desolate location, instead of coming up under the ice and melting it (to make flood) or an ash volcano. The other good news is that the (poisonous) sulfur dioxide has mostly been blowing northeast, so it hasn’t seriously affected either Reykjavík or Akureyri.

CeroWrt 3.10.50-1 Field Report

Comments on the CeroWrt 3.10.50-1 build that I installed on my WNDR3700v2 primary router:

  • Seemed to install and configure properly (I retrieved the updated version with the lighthttpd fix)

  • I used a local copy of (from to configure the router to have my own DSL user/pw, SSID names, etc.

  • mDNS seems to work across subnets. (I don’t think that it worked properly for the last few CeroWrt releases)

  • The first time I looked, /usr/lib/CeroWrtScripts didn’t seem to have been updated with the latest from (see link above). But I looked again (after a reboot, and running the script, and another reboot (maybe something else)), and the script matches the github version… Not sure how that could have happened.

  • Debloating seems to be working as expected. netperf-wrapper with RRUL shows a jump from 60 to 80 msec (normal on my 7mbps/768kbps DSL link). Here’s the plot:

Doc Searls mentioned in passing that he uses a new speed test website. I checked it out, and it was very cool… is an all-HTML5 website that seems to make accurate measurements of the up and download speeds of your internet connection. It’s also very attractive, and the real-time plots of the speed show interesting info. (See below.)

Now if we could get them to include a latency measurement so people could point out their bufferbloated equipment. – Bringing sanity back to campaign financing

I recently joined Lessig’s citizen-funded MAYDAY.US campaign, an ambitious experiment to win a Congress committed to ending corruption in 2016, and we did something amazing: We raised $1 million dollars in 12 days. That’s a ton of money, but it’s not enough.

The plan is to raise $12 million in 2014, and use it to make fundamental reform of campaign financing the major issue in five congressional races. Then we’ll apply what we learn from those races to 2016.

We’re raising $5 million more by July 4, and I’m asking my readers if you can help us get the rest of the way there. So my question is: will you help?


New CeroWrt Router scripts

I posted a set of scripts that people can use to test, configure and debug their CeroWrt router installations. CeroWrt router firmware is a test bed for learning about and eliminating bufferbloat.

The scripts are available on Github at They include:

  • – a script that emulates the famous (but limited) This script is better because it measures ping latency during the download and upload, and tells you how bad (or not) the bloat is in your network/router.

  • – a script that emulates the RRUL test suite. It runs on the CeroWrt router (or your laptop) to measure the latency under heavy load.

  • and – two scripts for configuring a CeroWrt router to a consistent state after flashing new firmware.

  • – a script that collects a number of configuration and operational stats that is useful for debugging.

Net Neutrality & petition

There’s a petition posted about Net Neutrality. It’s a little bit of a rant, but I agree with its punchline: No bandwidth modifications of information based on content or its source.

This relates to a comment from Philip Greenspun’s blog (How many times do we have to pay for the same internet service?) If I’m paying my Internet Service Provider (ISP) to provide bits from the Internet to my home, and content providers pay an ISP for getting their bits into the Internet, what’s the problem? Why do the ISPs ask to be paid more for certain content, say, from Netflix?

The problem is that most ISPs (cable company, DSL/phone company, etc.) seriously underprovision their facilities. If every home subscriber ever attempted to use the service (at 3 megabits/second or 7, or 15, or 100, or whatever) that the ISPs advertise, there would be dramatic slowdowns. The ISPs simply don’t have sufficient capacity in place. The “easy out” for the ISP is to brand Netflix and other content providers as “bandwidth hogs” and using “more than their share” of bandwidth.

Two questions come from this:

  1. If, Netflix, say, did pay more, would my ISP promise to provide great service that didn’t slow down? (I wouldn’t bet on it. They don’t like to promise anything…)

  2. Why shouldn’t we consumers treat these bandwidth claims as false advertising? ISPs take our money while promoting a service that they know they have no ability or intention of providing. Wouldn’t that be a fun class-action suit? :-)

In any event, you can read and sign the petition at They got almost half the signatures they need in a week; they need another 60K signatures by 24 May to get an official response from the White House. The more publicity on this issue, the better.