Pablo Averbuj

Comcast IPv6 redux

TL;DR: IPv6 router configuration can impact cablemodem provisioning on Comcast. Don’t let programmers run networks.

Follow up to my previous post on getting IPv6 configured on a Cisco router on Comcast.

I recently realized I was an idiot (for many reason) but in this case because I was still paying Comcast $10 a month for a cablemodem. Previously we had our home telephone service with Comcast and my research had indicated that you couldn’t buy a retail modem in that case (but I’m not so sure now). But in either case we moved back to Vonage and so after it while it dawned on me that we could save the rental fee.

After truly minimal research I ordered an Arris SB6183 from Amazon. I received it and activated it but I had a lot of trouble getting IPv4 reliably. In fact, as long as it was plugged into the Cisco it would not get sync with Comcast. If I unplugged the ethernet it would get sync and afterwards I could plug it in and sometimes it would even work.

That wasn’t a sustainable solution so I then ordered a TP-Link TC7160. I got the modem, it would get sync but I still wouldn’t get IPv4 assignments and when I got IPv6 assignments they wouldn’t route.

This was fairly strange, and with both modems I could plug them into a laptop and have them work normally. I could even have the laptop spoof the MAC address of the router interface, plug it into the router. The router would get DHCP but then not be able to route traffic. But the same router/configuration had worked with the old Comcast-rented modem without issue. I pondered this for a good while and it most likely had to be something IPv6 related.

I started turning off IPv6 settings in the router until it would work. The key configuration was “ipv6 unicast-routing”. When that was disabled, it would get IPv4 and work great. Enabled (even after IPv4 was assigned and working) and it would instantly stop routing traffic.

I’ll spare you the details of the fiddling that ensued but this is what I found: I needed to suppress IPv6 route announcements on the WAN interface.

The fix:

conf t
 int GigabitEthernet8
  ipv6 nd ra suppress

Analysis/Speculation:

I believe what happens, and what changed, is that the newer cable modems are getting their provisioning over IPv6 while the old modem was provisioning over IPv4 (or something else that’s not IPv6). The route announcements were confusing the cablemodem and causing it to prefer sending its provisioning traffic to my router instead of the Comcast next-hop. When the route announcements stop it works as intended.

Bonus #1:

Even though I saw recommendations for the Arris SB6183, the TP-Link TC-7160 boots faster and runs cooler than the Arris. The amount of heat from the Arris was very surprising, the coax connector was too hot to touch, and it heats up very quick. The morning after I switched I felt the coax connector on the TP-Link and it was normal as I would have expected.

Bonus #2:

Comcast doesn’t care about your IPv6 DHCP hints.

! doesn't work.
ipv6 dhcp client pd hint ::/56

How I’m Using Amazon Dash Buttons for SmartThings

When I first heard about Amazon Dash buttons I knew they’d likely end up being a fun and (arguably¹) cheap way to control a few things on my SmartThings network. I’m certainly not the first to accomplish this but I have some insights that might be of interest.

By way of background, Amazon Dash are $5 wifi-enabled buttons that Amazon would like you to use to buy things easily. And surely there are many places that I think they are actually useful as intended. Like wherever you keep your trash bags, since you probably buy the same bags every time, you put a button, when you grab the last bag (or you’re getting low) you push the button and your standard trash bag order is placed. It’s very certainly a “first-world problem” type of thing. To set them up you use a smartphone to configure your wifi credentials² onto the button and set up your default order. Once configured when the button is pushed it connects to your wifi network and sends a notification to Amazon that the button was pushed.

If you’re going to abuse the button for another purpose, you’ll want to know when the button is pushed, and there are 5 obvious points in the communication lifecycle where you can do this.

1. Wifi authentication (layer 1?)
2. ARP requests (layer 2)
3. DHCP request (layer 3)
4. DNS request (layer 4)
5. HTTPS request (layer 4/7)

Each of these has their own advantages and disadvantages, all of which I won’t go into but I’ll try and summarize. ARP (where some previous attempts have been documented), DHCP and Authentication are annoying because they are very low layer, you’re likely going to need to detect the events using packet capture or something even more elaborate for the authentication part. DNS and HTTPS are easier to intercept and the challenge here is that the button will retry unsuccessful attempts so you have to suppress repeat events.

Let me cut to the chase and describe my implementation and after I’ll explore so of the approaches I abandoned. After some trial and error, for my solution I ended up intercepting the communication at the HTTPS point.

First I set up my dash buttons as others have done, configuring them onto my wifi network², letting them communicate with Amazon and then not selecting a product at Step 3 in the registration process.

I configured the wireless network to assign DHCP normally, and I allow the DNS requests to occur as normal. However all TCP connections from the Dash buttons are redirected (via a Destination NAT rule) to a local server on port 9999 (you can choose your own port). I set up xinetd with a service³ that listens on that port and when a connection is received, start a python script called dashproxy.py³ to handle the request.

xinetd passes in the remote IP via an environment variable REMOTE_HOST. I parse this value for the last octet of the IP and have a simple lookup table to map the IP to a SmartThings switch. The script then invokes the SmartThings API to toggle the switch.

The only wrinkle with this otherwise straightforward setup is the aforementioned problem of retries. The Amazon Dash uses HTTPS and (as far as others reported) seems to expect a certain certificate chain. I work around this by not dealing with HTTPS at all. I’m only handling accepting the connection and then closing the socket. However this causes the button to retry. To workaround this I abuse a feature of xinetd that can limit the connections per IP. I allow only one connection from an IP at a time. In my Python script after I perform the work I close the file descriptors for stdin/stdout/stderr which causes xinetd to close the TCP socket. At this point the dash button will retry to reconnect. However instead of exiting the Python script I issue a sleep for 10 seconds. Because the process hasn’t exited xinetd will reject the retry connections by accepting and closing the sockets without re-invoking the Python script. This effectively means the script is only invoked once per button push and the retry attempts are successfully suppressed by xinetd’s connection limiting.

On the SmartThings side I basically follow what other tutorials have described in setting up a RESTful SmartApp and giving it access to the switches it needs to control. My biggest issue here was figuring out the OAuth and API endpoints. I wasted an embarrassing amount of time flailing on this until I found the section of the docs named “Just the URLs” and that was the piece of information I needed. Here’s the way I ended up doing this.

As a one time process, in a browser I issue the with code request and then the access token request. This gives me the endpoint URL and access token needed to make API requests and then hard code this into the script.

The other API requests the are 1) to get the list of switches to map a switch name to a switch ID (optional) and 2) the API call to toggle the switch. For reference I used this SmartApp for an API.

Other Avenues I pursued and abandoned:

I tried doing this at the DHCP and DNS layers.

For DHCP it was difficult to effectively limit the devices that would have their DHCP redirected to a different port because the only thing you can key on is a MAC address and this isn’t normally exposed by a UDP socket.

I had a working implementation with DNS redirected but the issue was suppressing the retries. It would have been possible to avoid the retries altogether if I could have made a valid DNS response but after a bit of researching with Scapy (which I’m sure could do it) I couldn’t find working examples of how to parse data read from a socket and/or compose a DNS response (though I had something I think would have worked). Lacking that the dash button would retry to resolve but because I couldn’t do the rate limiting trick on UDP (in truth I hadn’t thought of it yet) I switched to the HTTPS interception.

Footnotes

1. JDRoberts makes an argument about the cost of using these buttons
2. I actually set up a separate wireless network for the Amazon Dash buttons. For me this made it easier to apply the traffic modifications. Also, be careful about giving Amazon your wifi password.
3. Here’s the code listings for my dashproxy project

Resources:

• SmartApp for Restful manipulation of switches
• Documentation on someone else’s approach to the same problem
• SmartThings documentation for OAuth – “Just the URLs, Please“

Adventures in IPv6 – Cisco, Comcast, and Debian

I recently got motivated to deploy IPv6 at home. I’ve been meaning to do this for a while but everytime I checked the Comcast documentation was vague as to whether it was available in my area. Finally, Comcast announced they had rolled out dual-stack IPv6 across the country and I couldn’t make excuses anymore. Unfortunately, in the meantime all of the Comcast IPv6 pages also happened to stop working. Nearly any Comcast page in the Google index relating to IPv6 details just redirected me to xfinity.comcast.net.

Let’s just dig into the details. Here’s how I got mine to work, your mileage may vary.

Cisco Configuration

I run a Cisco 800-series router as my internet gateway. The Internet comes in on interface FastEthernet8. Here’s the configuration I have, along with some commentary on what why and how.

WAN (Comcast) interface

interface FastEthernet8
  description Comcast Cable Modem
  ip dhcp client client-id ascii whiteoak
  ip address dhcp
  ip flow ingress
  ip nat outside
  ip virtual-reassembly in
  duplex auto
  speed auto

! everything above this line was configure prior to ipv6
  ipv6 enable
  ipv6 address dhcp
  ipv6 nd ra suppress
! "default" was needed to install the IPv6 default-route into the router
  ipv6 address autoconfig default

! This requests a prefix-delegation, so we get our own prefix from which to assign addresses
! This part is tricky, in my case I named that prefix "comcast-ipv6" this is important because we'll use 
! this named prefix in other sections of the code
  ipv6 dhcp client pd comcast-ipv6 rapid-commit
  ipv6 dhcp client request vendor

! This is supposed to prevent source spoofing from the WAN, verifies that the 
! source address is reachable via that interface
  ipv6 verify unicast source reachable-via rx allow-default 

! These two traffic filters are to prevent traffic from reaching directly into the inside hosts,
! since they will have globally routable IPs
  ipv6 traffic-filter wan-in in
  ipv6 traffic-filter wan-out out

I initially did this without the traffic-filter on the interface, and I suggest you do the same. First get it working, then lock it down. This is just the final configuration I ended up with. I’ll expand on the access-list’s further down.

I also set these couple of parameters in the global scope:

ipv6 unicast-routing
ipv6 cef
ipv6 cef accounting per-prefix

With those options set, I started getting IPs and prefix’s from Comcast. Here’s an example of what “sh ipv6 dhcp int” returns:

FastEthernet8 is in client mode
  Prefix State is OPEN
  Renew will be sent in 15:29:49
  Address State is OPEN
  Renew for address will be sent in 15:29:50
  List of known servers:
    Reachable via address: FE80::201:5CFF:FE33:401
    DUID: 0001000117C5C79C14FEB5D78385
    Preference: 255
    Configuration parameters:
      IA PD: IA ID 0x00030001, T1 129598, T2 207357
        Prefix: 2601:abcd:abcd:25B::/64
                preferred lifetime 259197, valid lifetime 259197
                expires at Sep 06 2014 04:07 PM (228585 seconds)
      IA NA: IA ID 0x00030001, T1 172800, T2 276480
        Address: 2001:1234:1234:31:486:112A:BEEC:5A6F/128
                preferred lifetime 345600, valid lifetime 345600
                expires at Sep 06 2014 04:07 PM (228586 seconds)
      DNS server: 2001:558:FEED::1
      DNS server: 2001:558:FEED::2
      Information refresh time: 0
  Prefix name: comcast-ipv6
  Prefix Rapid-Commit: enabled
  Address Rapid-Commit: disabled

You can see the “IA PD” (prefix delegation)” and “IA NA” (non-temporary address).

LAN interface

The next step was to figure out the LAN side of the equation. On the Cisco I use Vlan1 for the Ethernet ports, but I issue DHCP (IPv4) from a Debian server. I started doing the research on this (See the kylemanna.com link below) but  it turned out to be completely unnecessary. To my understanding, the router does a route announcement (RA) on the LAN (once configured on the Vlan1 interface) and everything auto-configured itself accordingly. Here’s what the Vlan1 interface looks like:

interface Vlan1
 description Main LAN
 ip address 10.0.0.5 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
# Again, everything above this line was already configure prior to my IPv6 attempts

# This assigns ourselves a static IP from the delegated prefix we get form Comcast
# This comcast-ipv6 matches the dhcp option I used on FastEthernet8
 ipv6 address comcast-ipv6 ::1:0:0:5/64
# nd stands for "neighbor discovery"
 ipv6 nd autoconfig prefix
 ipv6 nd autoconfig default-route
 ipv6 nd other-config-flag

At this point in the configuration the Mac’s on my network started working automatically, which is pretty cool. By default, the Mac’s configure a permanent global address using the MAC address of the interface as a template, but they also create a daily temporary IPv6 address that they use for outgoing connections. This address gets rotated daily and kept for about 7 days.

Debian server

The Debian server similarly auto-assigned itself an IPv6 address based on it’s MAC address (this is called an EUI-64, by the way). That left me with two challenges:

1. Like on the Cisco router, I wanted to assign the Debian box a “vanity” IPv6 address that would be more easily remembered
2. I wanted the Debian box to also use temporary IPv6 addresses by default.

The gotcha with #1 is that it has to inherit the delegated prefix from the router. It correctly inherits the prefix for the automatic addresses, but I can’t figure out how to do the same for a static address. I haven’t found a solution to this problem yet.

As for the temporary address, a friendly SuperUser site member answered the question.

# echo "2" > /proc/sys/net/ipv6/conf/eth0/use_tempaddr

Security considerations

At this point the machines on my network had globally routable IPv6 addresses, which meant you could access them directly from the internet without having to do port mapping or static DNATs inbound. But of course that was very risky. Here is the Cisco ACLs I ended up with (applied on FastEthernet8 interface above):

ipv6 access-list wan-in
 permit icmp any any
 evaluate reflectout
# These two ports are opened for DHCP to work 
 permit udp any any eq 546
 permit udp any any eq 547
 permit tcp any host 2601:abcd:abcd:25B:1::5 eq 22
# These allow inbound ssh and http traffic to some of the previously NAT'd hosts
 permit tcp any host 2601:abcd:abcd:25B:DEAD:BEFF:FEEF:CAFE eq 22
 sequence 70 permit tcp any host 2601:abcd:abcd:25B:3E07:54FF:FE04:84FC eq www
# Outbound access-list is not so interesting.
ipv6 access-list wan-out
 permit icmp any any
 permit tcp any any reflect reflectout
 permit udp any any reflect reflectout

Finally to test all of this I used a shell account on a Digital Ocean VM ($0.007 per hour) because they support IPv6 natively. From there I was able to externally probe the network and make sure things were working as intended.

Additional resources

Resources I used along the way:

1. Kyle Manna’s blog post was the first really useful link I found (more than Cisco deployment guide). It pointed me in the right direction for a lot of other research
2. Cisco IPv6 deployment guide
3. This DSLReports forum post gave a decent starting point for ACLs
4. Hurricane Electric’s IPv6 certification was useful in helping me verify my progress and kept going to the next step.

Update 2016/08/03:

I found an issue with my configuration that breaks modern cablemodems (at least on Comcast). As a result I’ve update my configuration above with the fix. I also update the configuration to use “!” comments instead of “#” comments since that’s syntactically correct for Cisco configuration.

Readable QR codes in Terminal

Recently I needed to generate some QR codes securely. So the usual solution of going to one of the multitude of online QR code generators wasn’t an option. I found the rQRCode ruby gem looked straight forward and required no external dependencies. It even has a handy to_s function that renders it in rough ASCII, even though it’s not readable by QR code readers. The to_s method lets your override true/false values with your own strings.

The next step was to render it on the console without having to get too fancy. My first thought was the high-ASCII ANSI block codes, but the way my terminal was configured that wasn’t working. Then I realized I was overthinking it. I just need to change the background color of a ” ” (space) character (technically two space characters made something that more resembled a square pixel). I found the colorize gem and it also had straight forward documentation and examples.

There was one final, minor problem. My terminal has a black background, so I added some white padding around the QR code to make it legible by my QR code reader. This was fairly trivial to do in code although probably faster to change the terminal background color. Here’s the code I ended up with:



# Pablo Averbuj - 2014-01-21 - Public Domain
# Gist available here: https://gist.github.com/pabloav/8553319
require 'rqrcode' # install this gem
require 'colorize' # install this gem
b = "  ".colorize(:background => :black); w =  "  ".colorize(:background => :white) # these should each contain two spaces
url = "http://www.google.com/"
# you may have to increase the :size parameter (and the w * 51 correspondingly) depending of the content you're encoding
qr = RQRCode::QRCode.new(url, :size => 8); puts w * 51; qr.to_s(:true => b, :false => w).split("\n").each{|l| puts w + l + w } ; puts w * 51


The final result looks something like this:

Update: 2016/08/03

I revisited this code and made it autosize the QR code automatically. The code is slightly longer but easier. All you need to do now is change the value of URL. Gist URL is the same: https://gist.github.com/pabloav/8553319

On closed ecosystems

I awoke to an email from Microsoft in telling me my Xbox gamertag could be used on my Windows 8 computer and through no fault of theirs this got me thinking about closed ecosystems (aka walled gardens).

The walled garden is an age old proposition by an organization: give up lots of control and they will make sure your life is better. In technology the proposed value to the user is security and a curated experience at the cost of choice and (many times) the ability to customize the environment and various other freedoms.

What I ended up considering this morning is the difference in the Xbox closed ecosystem and the iOS closed ecosystem and more specifically how it reflects on Microsoft.

I am not (I believe) an exuberant fan of Apple but I am a long time despiser of Microsoft. So those are my biases. But I’m on my 3rd iPhone and I’ve owned now about 4 Xboxes (6 if you count the previous generation), so I am a user of both.

Both pieces of hardware participate in a closed ecosystem and I suffer them differently. Now I will also admit up front that gaming consoles have distinct idiosyncrasies due to the content industry but I would also argue the carrier industry was also a cluster at least until the first iPhone came along if not still.

What I’m tying to get at (6 paragraphs in) is that these two companies have very different approaches to the ecosystem. Microsoft tries to extort maximum value. I pay $60 per year to them and in exchange I get increasing ads on my screens, I get updates that increasingly distract from the experience and I get no choice in the matter. Can I leave the ecosystem? Yes and no. I could hack the hardware. The games I own would continue I work but I wouldn’t be able to play online which is, to a large degree, why I own it (and pay $60 per year). And on top of that they limit by choice by choking the content producers at the other end. They force Bing down my throat. In short Microsoft does what it’s always done: it uses its position of power to first further it’s agenda and secondly benefit me. This is the totalitarian dictatorship of closed ecosystem. This is the cautionary tale.

On the other hand my iPhones have updates with clearly defined features. Each iteration of the hardware and the device move the experience forward and (in my opinion) put the user first. Do they make decisions contrary to my desires? Yes they do. CarrierIQ and lately the advertiser tracking have smacked them in the face. But I would argue that Apple has (mostly) successfully fought against carrier stupidity to the benefit of the user. Not always. FaceTime over 3G and mobile hotspot features were unfortunate concessions. They have kept apps out of the App Store capriciously and arguably maliciously. But at the end of the day I submit to their ecosystem because the benefits largely outweigh the costs. I’ve also jailbroken my devices as soon as the option is available, the only consequence of which is possible impacts to my product warranty (which is again unfortunate but less significant than in the Microsoft case).

In either case neither company fully delivers on the promise of a closed ecosystem but where Apple tries to put the user first Microsoft makes a compelling example of the dangers of a walled garden mismanaged by selfishness.

Multiple Hashing for Security

With all the talk of MD5 and SHA-1 collision attacks it got me wondering the other morning: would storing two different hashes of the same data result in a more secure environment rather than just moving on to the next new hash algorithm?It’s not an easy case to make, but suspend disbelief for me.

The case I’m addressing is one of hashes as signatures, such as software delivery. The collision risk here is that an attacker breaks into a popular source of software (let’s say gnu.org) and replaces the good files with compromised files that share the same size/hash. When a user downloads the file and compares the hashes they find a match and proceed to trust the software. But what if instead of just providing the result of a single hash (let’s say MD5) the provider supplied both SHA-1 and MD5 hashes of the file. Now an attacker has to compose a file that collides on both algorithms. As a general method (regardless of algorithms) this seems like it would mitigate attacks and give longer life to hash implementations.

Consider Debian for a moment. The Debian package lists include a MD5 hash of the package file. If/when they decide to switch to SHA-256 (a reasonable choice right now), it would require a lot of retooling. Since they have to keep the MD5 hashes anyway for backwards compatibility, why not retool to support both hashes. Now an attacker who compromises a package has (what I would imagine to be) an exponentially more difficult task.

My coworker, Will, brought up an interesting question. In the case where the data is secret such as storing the hash of a password, does computing multiple hashes of the secret data provide an opportunity for an attacker to glean information about the secret data? Storing two hashes of a password would slightly decrease the efficacy of brute-force attacks, but also increase the efficacy of attack via rainbow tables. If there’s a potential for information leakage the medicine may be worse than the disease.

Update: Doug found a paper that addresses the first part of the question here. It concludes that concatenation is a hedge, but doesn’t quantify how much value it adds.

Rock Band Choices

Channeling xkcd today:

Mi Iaia

My grandmother, Carolina, or Iaia as we called her passed away today of natural causes after 94 years of life. 

She lived and died with grace and dignity and was an example to me and so many other people for how to live a life of adversity and persevere without losing your sense of humor or humility. She played a role in every stage of my life and I am very thankful for the lessons she taught me with her words and with her actions. I am glad to have spent three days earlier this week by her side.

She will be missed by more people than I can list here, but is survived by her daughter Cristina, her son Rodolfo (Yiye), her grandchildren Gaby, Veronica, Roger, and Pablo, and six great-grandchildren including my two girls, Alexis and Anna (whose middle name is Caroline after Iaia).

Ciao, Iaia.
June 16, 1914 – March 12, 2009

Silly Unix Tricks #1: De-crufting Bind9 configuration files

I’d like to resume writing occasionally, so the first one will be brief. Trying to debug a DNS problem tonight, I realized my BIND9 configuration file (named.conf) had accumulated too much cruft over the years. Unfortunately, the C/C++ comment style made it difficult to grep the useful bits. cpp to the rescue:

janus:/var/cache/bind/master# wc -l master-zones.conf
100 master-zones.conf
janus:/var/cache/bind/master# cpp master-zones.conf  | wc -l
49

This cleans up the file nicely. Of course, you’ll have to add the useful comments back in.

Loudoun Sheriff’s Day Out

Here’s a note I just submitted to the Loudoun County Sheriff’s Office. I think a few of the people who pay attention might have something to say about this.

On Wednesday, August 6th, I was driving home on Countryside Blvd. when I encountered what I can only describe as a checkpoint at the intersection with Carrollton Rd. Both directions of Countryside were blocked and all vehicles were being detained, as far as I can tell, without cause. I was asked by an officer to show my driver’s license and upon quick inspection was waved along.

Given the media attention that was received seemingly similar activities in D.C., I was wondering if you could provide some information as to what occurred that day and why. I’m worried this may have been a violation of mine and other citizens’ civil liberties, but would like to reserve judgement until I have more information.

I appreciate your time and considerate attention to this inquiry.
Thank you.

It didn’t occur to me to question the basis for asking for ID and it didn’t really anger me until after I left the checkpoint. I went back out with a camera to document what was going on. To their credit, the police didn’t seem to mind me taking pictures. At 6:45pm, just as I was about to drive through the roadblock again and this time refuse to show ID, they all simultaneously and unceremoniously packed up and went home.

Not even sure what to make of it. Someone’s already suggested I’m overreacting, and that may be so, but I nevertheless was left with an uneasy feeling. Some pictures below.

Update: I received a quick and helpful response from the Sheriff’s Office with this information:

Mr. Averbuj, for several years the Loudoun Sheriff’s Office have conducted Driver’s License Checkpoints throughout the county as a continuing crackdown on unlicensed drivers. We have conducted three such checkpoints this Summer.  The location for the checkpoints are selected based on deputies who report a number of unlicensed drivers during their daily traffic stops.

During the checkpoint on Wednesday nine motorists were cited with driving without  a license and two for driving on a suspended license. Those were among the 42 citations issued during the 3 hour checkpoint.

I found a page on the very recent overruling of similar roadblocks in South Carolina, and the SCOTUS decisions that govern such activity.