nfsen-dockerized Netflow Collector

[Part of the series of blog postings on Netflow]

A lightweight Netflow collector and web display based on NFSEN/NFDUMP in a Docker container. NFSEN and NFDUMP are documented and hosted at SourceForge.net

This container listens on ports 2055, 4739, 6343, and 9666 for netflow, ipfix, and sFlow exports. It displays the collected data in a web interface.

Check the Github repo. The files from the /docs directory have more info. Major thanks go to https://github.com/nerdalert/nfsen-dockerized
for a start on this Dockerfile and all the supporting documentation.

Pros:

  • Pretty graphs, for varying time periods: daily, weekly, monthly, yearly.
  • View data from a specific time period by going to Details, then dragging the thumbs at the bottom of the graph.
  • Top-N talkers also available from Details page: scroll to the bottom of the page, configure “Options”, the click “process”
  • Automatically detects the netflow exporter(s).
  • Lightweight – Runs on a modest computer. Works great if you’re only handling a single home-router’s flow exports. I haven’t tested it with more exporters/more traffic.
  • The Docker container displays graphs “out of the box”.

Cons:

  • Home page shows 12 charts, one each for flows/sec, packets/sec, and bits/sec, for each of the four time periods. This makes it hard to know what to “focus on” when you’re just starting up, since none of the charts has very much data.
  • Also, showing charts with “Flows/sec” makes it seem as if that’s a critical statistic. But Bit/sec is more important (IMHO).
  • Selecting a time period to view is a little clunky. (It’d be fun simply to drag across the graph…)
  • Alerts and Stats seem a bit inscrutable.
  • I briefly tried to install the “FlowDoh” plugin that purports to display Top-N talkers. It was probably my error, but it just didn’t work after a “good try” to use it.
  • As-is, this Docker instance only handles a single exporter (my home router), since the Docker networking doesn’t distinguish external exporter addresses.

This article is a part of the Netflow Collector series.

Webview Netflow Reporter (wvnetflow)

[Part of the series of blog postings on Netflow]

Webview Netflow Reporter is a lightweight Netflow collector and web display tool based on wvnetflow and flow-tools in a Docker container. Webview Netflow Reporter was created by Craig Weinhold craig.weinhold@cdw.com. The original wvnetflow site is hosted at SourceForge.net.

The Dockerfile is available from Github.

wvnetflow screen shot

Pros

  • Pretty graphs – Displays stack charts of the kinds of traffic (web, email, network, telnet, ssh, dns, mysql, sip, p2p, file server, etc) flowing through the router at any time. See the graphic above.
  • Click on an interesting point of the graph, and see exactly what traffic was being sent during that period.
  • Automatically detects exporters – no configuration required.
  • Lightweight – Runs on a modest computer. Works great if you’re only handling a single home-router’s flow exports. I haven’t tested it with more exporters/more traffic.
  • The Docker container comes pre-configured to display charts “out of the box”.

Cons

  • No automatic way to see “top talkers” – who’s hogging the bandwidth. You have to click the graph, then scroll through a table of hosts that were transmitting at the time.
  • One-minute granularity – The lowest granularity is one minute, despite the fact that the flow data has millisecond accuracy.
  • Five-minute time chunks – Data only gets updated every five minutes. This probably could be configured to change the processing rate.
  • Text-based configuration – Configuration files are arcane.
  • As-is, this only handles one exporter. See Known Issues/Questions
  • There’s an outstanding issue where there are gaps in the displayed charts. Hopefully this will be resolved.

This article is a part of the Netflow Collector series.

Getting Started with React!

I’ve been playing with the spate of new Javascript technologies, and ran into a problem. As a beginner in this new ecosystem, I chose one of those big “starter kits” because I wanted to avoid the hassle of configuring all the tooling (see Javascript fatigue).

But there was so much boilerplate that it was hard to see what was happening, and hard to steer away from its preconceptions to customize it for my needs. Here’s what I wish someone had told me a few weeks ago:

  1. I found create-react-app – a Facebook-supported tool for creating a basic React application. Although it’s a simple app, create-react-app brings along all the modern Webpack facilities without farbling around with configuration. You can focus on the code that’s important. See: Create Apps with No Configuration and the Guide for create-react-app

  2. In addition to the static pages, my application also needed to get data from a separate (“api”) server. There’s a great post from fullstackreact.com that shows how to create and integrate a server (to handle the API) alongside the Webpack-provided development server that handles the GUI. See: How to get “create-react-app” to work with your API

  3. I found a helpful tutorial from Christophe Coenraets that shows how to integrate Socket.io so that React components can listen for updates from a socket.io connection. See: Real Time Trader Desktop with React, Node.js, and Socket.io

  4. Finally (and too late to save me lots of work…), I found Dan Abramov’s React Makes You Sad flow chart. Regrettably, I took most of the “wrong” turns he depicts. But I am now happily working in a vastly simpler environment using the tools above.

TL;DR Dan Abramov is right (he’s always right 🙂 ) When you’re just getting started with React development, start simple. Follow the steps on the React Makes You Sad page. Maybe use create-react-app for your first prototype. And only add new technologies/packages/etc. when you understand why you need them.

Update: VW TDI Diesel Goodwill

A couple news items on the VW Diesel Goodwill Package:

  • I did get my 2.0L Goodwill package. It took about four weeks to arrive, and then I promptly used more than half of it on transmission service at VW. Sigh.
  • They have announced a Goodwill Package for owners of 3.0L diesel engines. That program expires on 31 July 2016 – hurry up!
  • There’s a hearing on the settlement package scheduled for 26 July 2016. If the judge approves it (highly likely), then the buyback/cash payments/other benefits will be available to TDI owners “in fall 2016.” See the announcements at vwdieselinfo.com.

VW Diesel Goodwill Package Expires Soon!

I’m one of the “lucky ones”… A “proud” owner of a 2010 VW Jetta Diesel 2.0 Liter engine. I’m afraid to look at the loss of value in the Kelley Blue Book from a year ago.

I signed up today for the Goodwill Package. As partial compensation for the loss of value and hassle, VW provides a $500 prepaid card to spend anywhere, a $500 card to use at a VW dealership, plus three years of Roadside Assistance.

So I’m going to use the cards to pay for needed repairs to the Jetta: one at my local garage that I trust, and one for (still other) repairs at the local VW dealer. Regrettably, I don’t think either card will cover all the expense 🙁

But… I guess I am sort of lucky – I just squeaked by the deadline. The expiration for the Goodwill Package is the end of this month – 30 April 2016. All you Jetta owners – get on the stick! Go to https://www.vwdieselinfo.com/goodwill_package/ and type in your VIN.

Bufferbloat on Packet Pushers

Back in October, I had the pleasure of chatting with Ethan Banks on the Packet Pushers podcast.

In it, we talked about the definition of bufferbloat, and how it harms the performance of VoIP, gaming, and general internet use. (It’s the reason our children say, “The internet is slow today, Daddy.”) I also described how the CoDel algorithm, and its fq_codel implementation in Linux makes the problem go away.

The most fun was the CoDel demonstration, which occurs near the end of the podcast. The podcast was recorded over Skype. When I turned on many netperf upload and download sessions to fully load my 7mbps/768kbps DSL link, the quality degraded without fq_codel. But it bounced right back when I re-enabled it.

Listen to the podcast – PQ Show 57 – Improve Your Home Internet Performance Using CoDel

Debugging Webpack app’s with WebStorm 2016.1

[Update Sep 2016: I just posted a support request with Webstorm asking for an updated tutorial on this subject. It’s Ticket WEB-23528]

It recently became possible to debug both server-side and client webpack applications with JetBrains’ WebStorm. The instructions came from a ticket WEB-20781, but they’re not in one place, so I figured I would write them out.

My application was built using react-starter-kit 0.5.1 – which is the current build as of mid-March 2016. These steps worked for me with WebStorm 2016.1 on OSX 10.10.5.

Update: WebStorm 2016.1 can automatically create the React Starter Kit 0.5.1 when you choose New Project…

Debugging Server-side Code

  1. Clone the react-starter-kit repo, then npm install as usual.

  2. You’ll need to tell webpack to generate source maps when debugging. Look in webpack.config.js for a line containing devtool (near line 204), and change it to:
    devtool: DEBUG ? 'source-map' : false,

  3. You also have to remove references to “hash” from the output filenames. Look in the same file near line 156 for a line like this, and change it to:
    filename: DEBUG ? '[name].js' : '[name].[hash].js',

  4. Create a new Run/Debug Configuration. Choose Run -> Edit Configuration, then add a “Node.js” item. Give the configuration a name, and if it isn’t already there, enter the path to your Node.js binary.

  5. Enter the path to your react-starter-kit project folder in “Working Directory”.

  6. For the “Javascript file:” enter the server-side code in the “build” directory. For react-starter-kit, it’s “build/server.js”

  7. Save the configuration by clicking OK.

  8. Build all the files by double-clicking “build” in the npm panel, or typing npm build in the command line.

  9. Set some breakpoints in the server-side code.

  10. In WebStorm, choose Debug (Ctl-D) for the configuration you created in 4. above. WebStorm starts debugging your server-side code, and should stop at breakpoints as expected.

Debugging Client-side Code

I haven’t needed to try these steps yet, but I understand this is the procedure:

  1. Follow Steps 1-3 above.

  2. Create a new Run/Debug Configuration. Choose Run -> Edit Configuration, then add a “JavaScript Debug” item. Give the configuration a name.

  3. Enter the URL to your app in the “URL:” field. For react-starter-kit, it’s http://localhost:3000

  4. Save the configuration by clicking OK.

  5. Start the application by clicking start in the npm panel, or typing npm start from the command line.

  6. Set some breakpoints in the client-side code. WebStorm should be debugging your client-side code, and should stop at breakpoints as expected.

Do you have comments/suggestions/improvements to these procedures? Leave a comment…

Hacking with React – A great book

I wanted to learn React, and I have been wandering around the Internet, looking at a zillion “how to get started” tutorials. I’m not sure what made me stop and look hard at Hacking with React, but I’m so glad I did.

Hacking with React provides a great combination:

  • a progression of straightforward, short lessons toward a working example program,
  • an interesting application (not another To-Do app, thank FSM!),
  • a Howto for setting up some of the confusing Node.js tooling (webpack, ESLint),
  • introduction to testing with Jest,
  • advice about best Node/Javascript/coding practices,
  • a decent understanding of how all these tools work together, and of course,
  • a great walkthrough of the concepts behind React.

At the end of the forty-some tasks (which take a while to complete if you actually do the work of understanding each lesson) you have a working React application, and a pretty good understanding of the basics of starting and building a React/Node application

The book is free to read, but you should buy this book to reward this excellent work. For $10, you’ll learn a lot.

WiTi Router Board update

I have the Witi Router working now with SQM-QoS – the bufferbloat-fighting technology that has been implemented in OpenWrt. Read more about Bufferbloat at Bufferbloat and the Ski Shop.

Demetris has made a build of OpenWrt Chaos Calmer (15.05) that has SQM-QoS built-in, so it’s a simple download and flash of the firmware. You can get a copy of the firmware at the mqmaker forum.

I haven’t tried it myself, but it looks as if stas2z’s BB version of the firmware also has the SQM-QoS as well at Alternate openwrt at mqmaker.