My Original Theory In going from the Level-3 system to the wholly-owned Verizon network, Verizon implemented
a new MAC authentication policy, coupled with a 15-minute DHCP IP address lease. This seems to be causing connectivity problems
for lots of people. I have tried two different modems with built-in NAT routers, a Zoom ADSL-X4 and a Westell 2200. Both crash
or lock up every few hours because their DHCP client waits until the very end of the lease (13-15 minutes) before attempting
to renew. If the renewal fails, there isn't time for a re-try, and the resulting IP address change causes other problems.
I also tried a Gigafast EE400-R broadband router that renews at t/2 (7 or 8 minutes) and it never has a problem with this.
Computers that pull their DHCP IP address directly from the WAN seem to be OK.
While this isn't theoretically Verizon's problem, it's a bit of a hardship for everyone to run out and purchase new hardware,
especially since this is a problem with Verizon-provided equipment. I'm hoping Verizon will recognize this as a problem and
make the necessary policy changes to restore "always on" connectivity.
My Revised Theory My original theory was that a large number of routers were non-compliant with the
DHCP spec. This seemed unlikely to me, but it was my working hypothesis at the time. After studying the DHCP spec more closely, I picked
up on something I had missed before. The renewal time is 1/2 the lease time by default. DHCP servers frequently use the default,
so the L/2 behavior is familiar to many. As it turns out, this value is configurable and controlled by the servers. Verizon
is evidently setting this value much closer to the lease time, and sometimes it isn't soon enough to be effective, and the
lease expires. This seems to cause all kinds of connectivity problems.
Now I realize that this is not obvious, because the renewal time is not displayed by any DHCP clients I have ever used.
The only way to see it without specialized software or equipment is indirectly by observing logs and DHCP client behavior.
I have three data points: A Westell 2200, a Zoom ADSL-X4 and a GigaFast EE400-R. The Zoom and the Westell renew their 15 minute
lease every 13 to 15+ (!) minutes. The GigaFast renews every 7 to 8 minutes. As you might expect, the Zoom and Westell fail
to renew occasionally. The symptom at that point is complete and abject failure. The Westell displays an endless string of
DNS failures; the Zoom simply locks up and you can't even access the log. But the GigaFast gets a renewal every time at 7
to 8 minutes, and the IP address never changes even after weeks of operation. The irony of it is, the GigaFast seems to be
ignoring Verizon's renewal time in the DHCP options field. The routers that are failing are the ones that are correctly observing
the setting. If you want this boiled down to a simple sound bite, it is this: Verizon is, in effect, programming our routers
to fail!
I have been ridiculed for this hypothesis, but my observation, which I can reproduce at will, is that routers that start
renewing their DHCP lease sooner can survive other network problems and delays, making the entire system more robust. There
is a trade-off however: earlier renewals will increase the load on the DHCP servers. One would hope that the IT professionals
at Verizon have the expertise and the tools to analyze the options. Until they have it worked out, you can probably guess
which router I'm going to use!
A remaining question has to do with PPPoE failures that also seem to be related to the recent network changes and the attendant
15-minute DHCP lease time. I have been unable to determine how PPPoE fits into my hypothesis. Perhaps it doesn't, but that
seems unlikely. I would expect PPPoE servers to use DHCP on the back end to obtain IP addresses for their sessions. Behavior
would depend on the interactions between the PPPoE and the DHCP protocols combined with Verizon's policies -- much harder
to analyze from the outside and not using PPPoE myself.
Conclusions Based on Some Actual Data I just ran an Ethereal trace with my computer connected directly
to my Westell 2110 bridging modem, and traced the bootp.dhcp packets that were exchanged for 6 renewal periods (about 45 minutes).
Each packet sent by Verizon for normal renewals contained the lease duration (15 minutes), the renewal time, T1 (7.5 minutes)
and the rebind time (13.5 minutes). So Verizon is doing the standard thing, and telling the routers to renew at 7.5 minutes.
Windows networking properly renews its lease at 7.5 minutes. My GigaFast router properly renews its lease at 7.5 minutes.
My Westell 2200 modem renews at (ahem) 13.5 minutes. My Zoom ADSL-X4 renews at (ahem) 13.5 minutes. That is wrong, wrong,
wrong! SOHO router firmware is a disgrace.
The Ethereal Trace The smoking gun is in the DHCP ACK, frames 2, 4 & 6, bootstrap protocol, options 51, 58 & 59. If anyone can see anything in here that can explain the behavior we're seeing with the routers, please let me
know.
No. Time
Source Destination
Protocol Info 1 0.000000 71.112.194.122
71.112.194.1 DHCP DHCP Request - Transaction
ID 0x3a87c24b
Frame 1 (342 bytes on wire, 342 bytes captured)
Arrival Time: Jun 8, 2005 20:29:59.813169000 Time delta from previous packet: 0.000000000 seconds
Time since reference or first frame: 0.000000000 seconds Frame Number: 1 Packet
Length: 342 bytes Capture Length: 342 bytes Protocols in frame: eth:ip:udp:bootp Ethernet
II, Src: 00:08:74:e6:07:be, Dst: 00:90:1a:41:54:0a Destination: 00:90:1a:41:54:0a (Unispher_41:54:0a)
Source: 00:08:74:e6:07:be (DellComp_e6:07:be) Type: IP (0x0800) Internet Protocol, Src Addr: 71.112.194.122
(71.112.194.122), Dst Addr: 71.112.194.1 (71.112.194.1) Version: 4 Header length:
20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable
Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length:
328 Identification: 0xb345 (45893) Flags: 0x00
0... = Reserved bit: Not set .0.. = Don't fragment: Not set
..0. = More fragments: Not set Fragment offset: 0 Time to live: 128
Protocol: UDP (0x11) Header checksum: 0x7303 (correct) Source: 71.112.194.122
(71.112.194.122) Destination: 71.112.194.1 (71.112.194.1) User Datagram Protocol, Src Port: bootpc
(68), Dst Port: bootps (67) Source port: bootpc (68) Destination port: bootps
(67) Length: 308 Checksum: 0x3753 (correct) Bootstrap Protocol
Message type: Boot Request (1) Hardware type: Ethernet Hardware address length:
6 Hops: 0 Transaction ID: 0x3a87c24b Seconds elapsed: 0
Bootp flags: 0x0000 (Unicast) 0... .... .... .... = Broadcast flag: Unicast
.000 0000 0000 0000 = Reserved flags: 0x0000 Client IP address: 71.112.194.122 (71.112.194.122)
Your (client) IP address: 0.0.0.0 (0.0.0.0) Next server IP address: 0.0.0.0 (0.0.0.0)
Relay agent IP address: 0.0.0.0 (0.0.0.0) Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Server host name not given Boot file name not given Magic cookie: (OK)
Option 53: DHCP Message Type = DHCP Request Option 61: Client identifier
Hardware type: Ethernet Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Option 12: Host Name = "Inspiron" Option 81: FQDN Flags:
0x00 0000 .... = Reserved flags: 0x00
.... 0... = Server DDNS: Some server updates .... .0..
= Encoding: ASCII encoding .... ..0. = Server overrides:
No override .... ...0 = Server: Client
A-RR result: 0 PTR-RR result: 0
Client name: Inspiron. Option 60: Vendor class identifier = "MSFT 5.0" Option
55: Parameter Request List 1 = Subnet Mask
15 = Domain Name 3 = Router 6
= Domain Name Server 44 = NetBIOS over TCP/IP Name Server
46 = NetBIOS over TCP/IP Node Type 47 = NetBIOS over TCP/IP Scope
31 = Perform Router Discover 33 = Static Route
Unknown Option Code: 249 43 = Vendor-Specific Information
End Option
No. Time
Source Destination
Protocol Info 2 0.018428 71.112.194.1
71.112.194.122 DHCP DHCP ACK
- Transaction ID 0x3a87c24b
Frame 2 (336 bytes on wire, 336 bytes captured)
Arrival Time: Jun 8, 2005 20:29:59.831597000 Time delta from previous packet: 0.018428000 seconds
Time since reference or first frame: 0.018428000 seconds Frame Number: 2 Packet
Length: 336 bytes Capture Length: 336 bytes Protocols in frame: eth:ip:udp:bootp Ethernet
II, Src: 00:90:1a:41:54:0a, Dst: 00:08:74:e6:07:be Destination: 00:08:74:e6:07:be (DellComp_e6:07:be)
Source: 00:90:1a:41:54:0a (Unispher_41:54:0a) Type: IP (0x0800) Internet Protocol, Src Addr: 71.112.194.1
(71.112.194.1), Dst Addr: 71.112.194.122 (71.112.194.122) Version: 4 Header length:
20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable
Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length:
322 Identification: 0x0000 (0) Flags: 0x00
0... = Reserved bit: Not set .0.. = Don't fragment: Not set
..0. = More fragments: Not set Fragment offset: 0 Time to live: 64
Protocol: UDP (0x11) Header checksum: 0x664f (correct) Source: 71.112.194.1 (71.112.194.1)
Destination: 71.112.194.122 (71.112.194.122) User Datagram Protocol, Src Port: bootps (67), Dst Port: bootpc (68)
Source port: bootps (67) Destination port: bootpc (68) Length: 302
Checksum: 0x0000 (none) Bootstrap Protocol Message type: Boot Reply (2) Hardware
type: Ethernet Hardware address length: 6 Hops: 0 Transaction
ID: 0x3a87c24b Seconds elapsed: 0 Bootp flags: 0x0000 (Unicast)
0... .... .... .... = Broadcast flag: Unicast .000 0000 0000 0000 = Reserved
flags: 0x0000 Client IP address: 71.112.194.122 (71.112.194.122) Your (client)
IP address: 71.112.194.122 (71.112.194.122) Next server IP address: 0.0.0.0 (0.0.0.0)
Relay agent IP address: 0.0.0.0 (0.0.0.0) Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Server host name not given Boot file name not given Magic cookie: (OK)
Option 53: DHCP Message Type = DHCP ACK Option 54: Server Identifier = 71.112.194.1
Option 51: IP Address Lease Time = 15 minutes Option 58: Renewal Time Value = 7 minutes, 30 seconds
Option 59: Rebinding Time Value = 13 minutes, 7 seconds Option 1: Subnet Mask = 255.255.255.0
Option 3: Router = 71.112.194.1 Option 6: Domain Name Server
IP Address: 68.238.128.12 IP Address: 68.238.64.12 Option
46: NetBIOS over TCP/IP Node Type = Unknown (0) End Option Padding
No. Time
Source Destination
Protocol Info 3 450.037201 71.112.194.122
71.112.194.1 DHCP DHCP Request - Transaction
ID 0x7e117b76
Frame 3 (342 bytes on wire, 342 bytes captured)
Arrival Time: Jun 8, 2005 20:37:29.850370000 Time delta from previous packet: 450.018773000 seconds
Time since reference or first frame: 450.037201000 seconds Frame Number: 3 Packet
Length: 342 bytes Capture Length: 342 bytes Protocols in frame: eth:ip:udp:bootp Ethernet
II, Src: 00:08:74:e6:07:be, Dst: 00:90:1a:41:54:0a Destination: 00:90:1a:41:54:0a (Unispher_41:54:0a)
Source: 00:08:74:e6:07:be (DellComp_e6:07:be) Type: IP (0x0800) Internet Protocol, Src Addr: 71.112.194.122
(71.112.194.122), Dst Addr: 71.112.194.1 (71.112.194.1) Version: 4 Header length:
20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable
Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length:
328 Identification: 0xb441 (46145) Flags: 0x00
0... = Reserved bit: Not set .0.. = Don't fragment: Not set
..0. = More fragments: Not set Fragment offset: 0 Time to live: 128
Protocol: UDP (0x11) Header checksum: 0x7207 (correct) Source: 71.112.194.122
(71.112.194.122) Destination: 71.112.194.1 (71.112.194.1) User Datagram Protocol, Src Port: bootpc
(68), Dst Port: bootps (67) Source port: bootpc (68) Destination port: bootps
(67) Length: 308 Checksum: 0x3a9e (correct) Bootstrap Protocol
Message type: Boot Request (1) Hardware type: Ethernet Hardware address length:
6 Hops: 0 Transaction ID: 0x7e117b76 Seconds elapsed: 0
Bootp flags: 0x0000 (Unicast) 0... .... .... .... = Broadcast flag: Unicast
.000 0000 0000 0000 = Reserved flags: 0x0000 Client IP address: 71.112.194.122 (71.112.194.122)
Your (client) IP address: 0.0.0.0 (0.0.0.0) Next server IP address: 0.0.0.0 (0.0.0.0)
Relay agent IP address: 0.0.0.0 (0.0.0.0) Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Server host name not given Boot file name not given Magic cookie: (OK)
Option 53: DHCP Message Type = DHCP Request Option 61: Client identifier
Hardware type: Ethernet Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Option 12: Host Name = "Inspiron" Option 81: FQDN Flags:
0x00 0000 .... = Reserved flags: 0x00
.... 0... = Server DDNS: Some server updates .... .0..
= Encoding: ASCII encoding .... ..0. = Server overrides:
No override .... ...0 = Server: Client
A-RR result: 0 PTR-RR result: 0
Client name: Inspiron. Option 60: Vendor class identifier = "MSFT 5.0" Option
55: Parameter Request List 1 = Subnet Mask
15 = Domain Name 3 = Router 6
= Domain Name Server 44 = NetBIOS over TCP/IP Name Server
46 = NetBIOS over TCP/IP Node Type 47 = NetBIOS over TCP/IP Scope
31 = Perform Router Discover 33 = Static Route
Unknown Option Code: 249 43 = Vendor-Specific Information
End Option
No. Time
Source Destination
Protocol Info 4 450.055062 71.112.194.1
71.112.194.122 DHCP DHCP ACK
- Transaction ID 0x7e117b76
Frame 4 (336 bytes on wire, 336 bytes captured)
Arrival Time: Jun 8, 2005 20:37:29.868231000 Time delta from previous packet: 0.017861000 seconds
Time since reference or first frame: 450.055062000 seconds Frame Number: 4 Packet
Length: 336 bytes Capture Length: 336 bytes Protocols in frame: eth:ip:udp:bootp Ethernet
II, Src: 00:90:1a:41:54:0a, Dst: 00:08:74:e6:07:be Destination: 00:08:74:e6:07:be (DellComp_e6:07:be)
Source: 00:90:1a:41:54:0a (Unispher_41:54:0a) Type: IP (0x0800) Internet Protocol, Src Addr: 71.112.194.1
(71.112.194.1), Dst Addr: 71.112.194.122 (71.112.194.122) Version: 4 Header length:
20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable
Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length:
322 Identification: 0x0000 (0) Flags: 0x00
0... = Reserved bit: Not set .0.. = Don't fragment: Not set
..0. = More fragments: Not set Fragment offset: 0 Time to live: 64
Protocol: UDP (0x11) Header checksum: 0x664f (correct) Source: 71.112.194.1 (71.112.194.1)
Destination: 71.112.194.122 (71.112.194.122) User Datagram Protocol, Src Port: bootps (67), Dst Port: bootpc (68)
Source port: bootps (67) Destination port: bootpc (68) Length: 302
Checksum: 0x0000 (none) Bootstrap Protocol Message type: Boot Reply (2) Hardware
type: Ethernet Hardware address length: 6 Hops: 0 Transaction
ID: 0x7e117b76 Seconds elapsed: 0 Bootp flags: 0x0000 (Unicast)
0... .... .... .... = Broadcast flag: Unicast .000 0000 0000 0000 = Reserved
flags: 0x0000 Client IP address: 71.112.194.122 (71.112.194.122) Your (client)
IP address: 71.112.194.122 (71.112.194.122) Next server IP address: 0.0.0.0 (0.0.0.0)
Relay agent IP address: 0.0.0.0 (0.0.0.0) Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Server host name not given Boot file name not given Magic cookie: (OK)
Option 53: DHCP Message Type = DHCP ACK Option 54: Server Identifier = 71.112.194.1
Option 51: IP Address Lease Time = 15 minutes Option 58: Renewal Time Value = 7 minutes, 30 seconds
Option 59: Rebinding Time Value = 13 minutes, 7 seconds Option 1: Subnet Mask = 255.255.255.0
Option 3: Router = 71.112.194.1 Option 6: Domain Name Server
IP Address: 68.238.128.12 IP Address: 68.238.64.12 Option
46: NetBIOS over TCP/IP Node Type = Unknown (0) End Option Padding
No. Time
Source Destination
Protocol Info 5 900.077323 71.112.194.122
71.112.194.1 DHCP DHCP Request - Transaction
ID 0x1409a7af
Frame 5 (342 bytes on wire, 342 bytes captured)
Arrival Time: Jun 8, 2005 20:44:59.890492000 Time delta from previous packet: 450.022261000 seconds
Time since reference or first frame: 900.077323000 seconds Frame Number: 5 Packet
Length: 342 bytes Capture Length: 342 bytes Protocols in frame: eth:ip:udp:bootp Ethernet
II, Src: 00:08:74:e6:07:be, Dst: 00:90:1a:41:54:0a Destination: 00:90:1a:41:54:0a (Unispher_41:54:0a)
Source: 00:08:74:e6:07:be (DellComp_e6:07:be) Type: IP (0x0800) Internet Protocol, Src Addr: 71.112.194.122
(71.112.194.122), Dst Addr: 71.112.194.1 (71.112.194.1) Version: 4 Header length:
20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable
Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length:
328 Identification: 0xbaf6 (47862) Flags: 0x00
0... = Reserved bit: Not set .0.. = Don't fragment: Not set
..0. = More fragments: Not set Fragment offset: 0 Time to live: 128
Protocol: UDP (0x11) Header checksum: 0x6b52 (correct) Source: 71.112.194.122
(71.112.194.122) Destination: 71.112.194.1 (71.112.194.1) User Datagram Protocol, Src Port: bootpc
(68), Dst Port: bootps (67) Source port: bootpc (68) Destination port: bootps
(67) Length: 308 Checksum: 0x786d (correct) Bootstrap Protocol
Message type: Boot Request (1) Hardware type: Ethernet Hardware address length:
6 Hops: 0 Transaction ID: 0x1409a7af Seconds elapsed: 0
Bootp flags: 0x0000 (Unicast) 0... .... .... .... = Broadcast flag: Unicast
.000 0000 0000 0000 = Reserved flags: 0x0000 Client IP address: 71.112.194.122 (71.112.194.122)
Your (client) IP address: 0.0.0.0 (0.0.0.0) Next server IP address: 0.0.0.0 (0.0.0.0)
Relay agent IP address: 0.0.0.0 (0.0.0.0) Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Server host name not given Boot file name not given Magic cookie: (OK)
Option 53: DHCP Message Type = DHCP Request Option 61: Client identifier
Hardware type: Ethernet Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Option 12: Host Name = "Inspiron" Option 81: FQDN Flags:
0x00 0000 .... = Reserved flags: 0x00
.... 0... = Server DDNS: Some server updates .... .0..
= Encoding: ASCII encoding .... ..0. = Server overrides:
No override .... ...0 = Server: Client
A-RR result: 0 PTR-RR result: 0
Client name: Inspiron. Option 60: Vendor class identifier = "MSFT 5.0" Option
55: Parameter Request List 1 = Subnet Mask
15 = Domain Name 3 = Router 6
= Domain Name Server 44 = NetBIOS over TCP/IP Name Server
46 = NetBIOS over TCP/IP Node Type 47 = NetBIOS over TCP/IP Scope
31 = Perform Router Discover 33 = Static Route
Unknown Option Code: 249 43 = Vendor-Specific Information
End Option
No. Time
Source Destination
Protocol Info 6 900.097626 71.112.194.1
71.112.194.122 DHCP DHCP ACK
- Transaction ID 0x1409a7af
Frame 6 (336 bytes on wire, 336 bytes captured)
Arrival Time: Jun 8, 2005 20:44:59.910795000 Time delta from previous packet: 0.020303000 seconds
Time since reference or first frame: 900.097626000 seconds Frame Number: 6 Packet
Length: 336 bytes Capture Length: 336 bytes Protocols in frame: eth:ip:udp:bootp Ethernet
II, Src: 00:90:1a:41:54:0a, Dst: 00:08:74:e6:07:be Destination: 00:08:74:e6:07:be (DellComp_e6:07:be)
Source: 00:90:1a:41:54:0a (Unispher_41:54:0a) Type: IP (0x0800) Internet Protocol, Src Addr: 71.112.194.1
(71.112.194.1), Dst Addr: 71.112.194.122 (71.112.194.122) Version: 4 Header length:
20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..0. = ECN-Capable
Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length:
322 Identification: 0x0000 (0) Flags: 0x00
0... = Reserved bit: Not set .0.. = Don't fragment: Not set
..0. = More fragments: Not set Fragment offset: 0 Time to live: 64
Protocol: UDP (0x11) Header checksum: 0x664f (correct) Source: 71.112.194.1 (71.112.194.1)
Destination: 71.112.194.122 (71.112.194.122) User Datagram Protocol, Src Port: bootps (67), Dst Port: bootpc (68)
Source port: bootps (67) Destination port: bootpc (68) Length: 302
Checksum: 0x0000 (none) Bootstrap Protocol Message type: Boot Reply (2) Hardware
type: Ethernet Hardware address length: 6 Hops: 0 Transaction
ID: 0x1409a7af Seconds elapsed: 0 Bootp flags: 0x0000 (Unicast)
0... .... .... .... = Broadcast flag: Unicast .000 0000 0000 0000 = Reserved
flags: 0x0000 Client IP address: 71.112.194.122 (71.112.194.122) Your (client)
IP address: 71.112.194.122 (71.112.194.122) Next server IP address: 0.0.0.0 (0.0.0.0)
Relay agent IP address: 0.0.0.0 (0.0.0.0) Client MAC address: 00:08:74:e6:07:be (DellComp_e6:07:be)
Server host name not given Boot file name not given Magic cookie: (OK)
Option 53: DHCP Message Type = DHCP ACK Option 54: Server Identifier = 71.112.194.1
Option 51: IP Address Lease Time = 15 minutes Option 58: Renewal Time Value = 7 minutes, 30 seconds
Option 59: Rebinding Time Value = 13 minutes, 7 seconds Option 1: Subnet Mask = 255.255.255.0
Option 3: Router = 71.112.194.1 Option 6: Domain Name Server
IP Address: 68.238.128.12 IP Address: 68.238.64.12 Option
46: NetBIOS over TCP/IP Node Type = Unknown (0) End Option Padding
|