Author Topic: Hard lesson learned about DNS  (Read 1950 times)

0 Members and 1 Guest are viewing this topic.

Offline Thom

Hard lesson learned about DNS
« on: May 01, 2016, 12:56:55 am »
I just discovered that the DNSSEC record for a domain name I bought from NAMECHEAP is screwing up resolving its address. I have had this domain in place now for over 30 days and I have not changed anything in my DNS records for most of that time. Around 2 weeks ago suddenly certain users reported the website down, tho others had no issues. I checked many places around the world and most I checked said my domain name could not be found. Most users of the website had no problems resolving the name and reaching the site.

Moreover, I gave people the IP address to try to eliminate the DNS lookup entirely and they still could not reach my website. WTF? Not even directly by IP? Yes, that's right. It seems NAMECHEAP messed up the DNSSEC record which prohibited the distribution of the correct information. I just realized that DNS is more than a lookup database of names to numbers, it is the central map of the Internet as well. DNS is not an independent name lookup service for IP addresses maintained in some other scheme or database, DNS IS the database that creates the map of Internet node interconnections.

Here all this time I thought if I had the IP address of a server and DNS went down or was unavailable I could still access the server directly by IP address. I now know that isn't true. The DNSSEC records are nothing more than additional records containing cryptographic keys used to validate legitimate requests to change the DNS data so it can be propagated around the world. That is their only function; they do not encrypt any information.

Although namecheap offers names and services in exchange for Bitcoin, I will not be purchasing any more services from them in the future. Names ARE their business and mistakes like this are severe. It represents incompetence, mismanagement or both and should never have happened.

It's really a shame too, b/c with the fierce competition in the hosting space hosting services come & go often. When that happens it can tie up domains and make their ownership questionable. Namecheap got it's start in 2000 and added hosting to it's lineup as a secondary feature. That aspect of the company is quite small compared with the name services. I have used namecheap to insulate myself from the riskier volatility of the hosting market. If a hosting company or VPS provider suddenly goes out of business without notice all I have to do is buy another and point my domain to the new one. It has minimal impact with adequate backups of the old server.

Word to the wise - you might want to find a different place to buy your domains than your hosting company or Namecheap.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline pc

  • Hero Member
  • *****
  • Posts: 1530
    • View Profile
    • Bitcoin - Perspektive oder Risiko?
  • BitShares: cyrano
Re: Hard lesson learned about DNS
« Reply #1 on: May 01, 2016, 08:06:05 am »
I just realized that DNS is more than a lookup database of names to numbers, it is the central map of the Internet as well. DNS is not an independent name lookup service for IP addresses maintained in some other scheme or database, DNS IS the database that creates the map of Internet node interconnections.

Here all this time I thought if I had the IP address of a server and DNS went down or was unavailable I could still access the server directly by IP address. I now know that isn't true.

DNS has nothing to do with node interconnections.

You can access the *server* using only the IP address. Your problem is that many websites are hosted on a single server/IP-address, and the server distinguishes requests to these sites by looking at the "Host" in the browser request. That "host" is missing when you try to access the website by typing the server IP into your browser. That's why your website didn't turn up.
Bitcoin - Perspektive oder Risiko? ISBN 978-3-8442-6568-2 http://bitcoin.quisquis.de

Offline Thom

Re: Hard lesson learned about DNS
« Reply #2 on: May 01, 2016, 01:09:16 pm »
I just realized that DNS is more than a lookup database of names to numbers, it is the central map of the Internet as well. DNS is not an independent name lookup service for IP addresses maintained in some other scheme or database, DNS IS the database that creates the map of Internet node interconnections.

Here all this time I thought if I had the IP address of a server and DNS went down or was unavailable I could still access the server directly by IP address. I now know that isn't true.

DNS has nothing to do with node interconnections.

You can access the *server* using only the IP address. Your problem is that many websites are hosted on a single server/IP-address, and the server distinguishes requests to these sites by looking at the "Host" in the browser request. That "host" is missing when you try to access the website by typing the server IP into your browser. That's why your website didn't turn up.

Not so pc. The issue was the failure to propagate DNS information to ALL DNS zones. What you describe would be an issue I had control over. If what you hypothesized were true, nobody would be able to make requests of my server and that wasn't the case. Nothing changed on my vps or the vps provider.

Most ISPs worked great and had the correct IP address for the domain name, but there were others that didn't. Those that didn't had a different IP address than my server b/c it wasn't propagated in the DNS system due to a screwed up DNSSEC record.

I used to think the DNS was separate from how packets were routed on the Internet, but that isn't the case. DNS is central in that process, and hence why I said it forms the basis of how interconnections between IP addresses (through DNS zones) are mapped. If that were not the case DNS could stop working and all anyone would need to access a server is the IP address.

This incident proved to me that isn't true. My server has not changed, it always has had a virtual host definition with the IP address and domain name in place. When the registrar removed the DNSSEC record from my domain name it allowed the correct information to  be propagated around the world enabling the TCP/IP packets to find their way to my server on the other side of the planet. Until the DNS information was allowed to propagate, an IP address (alone, no name) in a browser in one of the dead zones (the zones which had incorrect information) had erroneous information to route the request to my server. The DNS was integral for routing the packets, even when no domain name is used. My server responds to requests just fine with only an IP address. The VPS provider routes all packets for that IP address to my server, but they never reached my server due to incorrect routing, controlled by way of DNS zones.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline pc

  • Hero Member
  • *****
  • Posts: 1530
    • View Profile
    • Bitcoin - Perspektive oder Risiko?
  • BitShares: cyrano
Re: Hard lesson learned about DNS
« Reply #3 on: May 01, 2016, 01:36:57 pm »
I can assure you that DNS has nothing to do with how packets are routed through the internet. If that was the case, how could you look up DNS records in the first place?

You may have seen several issues at once, like a DNSSEC problem and an (unrelated) routing issue.
If a DNSSEC record is screwed up, those that validate DNSSEC signed zones will just ignore the bad records, meaning they shouldn't have seen any records at all. If they did see an IP address for your name, that could have been inserted by their provider for example (I know that T-Online here in Germany responds with their own IP address so they can display a nice error page with their own logo and advertising SPAM when you mistype a domain in your browser.).
Bitcoin - Perspektive oder Risiko? ISBN 978-3-8442-6568-2 http://bitcoin.quisquis.de

Offline Thom

Re: Hard lesson learned about DNS
« Reply #4 on: May 01, 2016, 03:58:54 pm »
I can assure you that DNS has nothing to do with how packets are routed through the internet. If that was the case, how could you look up DNS records in the first place?

With all due respect pc, the way that "DNS records are looked up in the first place" is b/c every machine knows where to route it's packets. There is a chain of authority and every link in that chain (if it is unaware of a direct path to an IP address) passes the packet "upstream", to the next "higher authority", where it processes the packet's destination the same way. Where do you suppose those packets would go without such a hierarchy? What does "upstream" mean? It means to the closest authoritative DNS zone. They are not broadcast (except UDP packets) as in a blockchain to all nodes, that would be inefficient and waste bandwidth.

There were not multiple issues at play. There was simply no know route by machines in the "dead" zones to get to my server. The DNS was the sole problem. If an IP address was ALL that was necessary everybody could reach my server regardless of DNS and that simply wasn't the case.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12915
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: Hard lesson learned about DNS
« Reply #5 on: May 01, 2016, 04:03:52 pm »


I can assure you that DNS has nothing to do with how packets are routed through the internet. If that was the case, how could you look up DNS records in the first place?
i can confirm this too

Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline pc

  • Hero Member
  • *****
  • Posts: 1530
    • View Profile
    • Bitcoin - Perspektive oder Risiko?
  • BitShares: cyrano
Re: Hard lesson learned about DNS
« Reply #6 on: May 01, 2016, 07:46:24 pm »

With all due respect pc, the way that "DNS records are looked up in the first place" is b/c every machine knows where to route it's packets.

Exactly. What you're saying here is that routing is independent from DNS.

DNS is in fact organized hierarchically. There are root server known only by their IP address. If you want to resolve www.ibm.com you send a packet to the IP address of one root server and it will give you a list of IP addresses that are responsible for .com domains. Then you send your query to one of the adresses in that list. In response you will received a list of IP addresses responsible for ibm.com, and if you send your query to one of those you will finally receive the answer that you were looking for.

IP routing is not hierarchical. At home you'll most likely have a single "upstream" to which you send all your packets. Your ISP does not have a single upstream, though. There is no such thing as a "root router" in the internet. The internet is an inter-net, not an inter-tree. Your ISP has connections to other ISPs, typically via large peering points where many ISPs are connected. Each router in these peering points knows where to send IP packets. These routes are managed almost automatically, and that has nothing to do with DNS. Look up "Border Gateway Protocol" for details.
Bitcoin - Perspektive oder Risiko? ISBN 978-3-8442-6568-2 http://bitcoin.quisquis.de

Offline Thom

Re: Hard lesson learned about DNS
« Reply #7 on: May 02, 2016, 03:50:24 am »
OK guys, I'll go back and study this more. Empirically I just don't see how the Internet would function if DNS disappeared. On the other hand, thinking back in the early days of the Internet back in the 1980's, the net functioned globally without DNS. Back then many nodes had hard coded lists of IP addresses in their hosts file though. It wasn't nearly as dynamic as today. However the TCP/IP protocol has not changed substantially since then, so I do recognize the complete picture or map if you will is not dictated by DNS alone.

One thing is for certain tho, that a single mistake in the DNS screwed up how traffic was routed, irrespective of everything else. Nothing was changed on my VPS or by my VPS provider between functional and non-functional routing of packets. Some request originators failed to reach my server while I and others had no difficulty at all.

Quote
...send a packet to the IP address of one root server and it will give you a list of IP addresses...

Those "root servers" as you call them are authoritative TLD DNS repositories. If the data they contain is wrong, how do you think packets will reach the correct destination? Perhaps if the TTL were long enough they eventually would find a way to arrive at the right place, but the ping time would probably be outrageous and the tracert long enough to circle the globe!

Regardless of the detailed mechanics at the lower levels, the effective result is that DNS influences the path packets take in reaching a target IP address, and if that path is too long they won't have time to reach their destination.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12915
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: Hard lesson learned about DNS
« Reply #8 on: May 02, 2016, 06:26:55 am »
To find a route from computer A to computer B, you are using the "routing layer" of your network stack. In case of TCP/IP, that layer would be "IP". here you get an IP-address which identifies your network and your host. If you connect different networks, you have gateways or routers, they know what to do if packages from one network want to travel to another network.

DNS merely "resolves" a domain name into an IP address .. that is really everything DNS can do.
google.com -> 216.58.213.206

This is how such a DNS entry would look like

Code: [Select]
google.com.             376     IN      A       216.58.213.206
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline Thom

Re: Hard lesson learned about DNS
« Reply #9 on: May 02, 2016, 10:29:59 am »
I studied these things decades ago and lived through their development. I understood them better in the past, and what you say is not foreign to me. That's why this incident feels a bit like like cognitive dissonance.

However, you both seem to be missing the point I'm making concerning the empirical evidence this incident highlights. I see no point in repeating myself or the raw facts I've presented, but aside from pc's speculation that there were multiple problems coincidentally involved, neither of you have explained why huge swaths of the Internet could not reach my server's fixed IP address while others could. You seem to be saying that DNS was not the issue, and if so you haven't explained what took place. It seems clear that DNS was the entire reason for the incorrect routing despite IP addressing being correct on source and destination.

As I said in my last post, hypothetically if the DNS suddenly disappeared I suppose the transport layer could get packets from any IP address in the world to any other IP address, but even so, I suspect the delays would be so significant many if not all packets would be lost along the way unless major changes were made to TTL defaults used in routers and other networking devices.

No offense intended to either of you, I'm just trying to make sense of the raw facts I see and reconcile them with what I thought I knew about networking, particularly DNS. My previous understanding that DNS is merely a distributed name to number lookup is called into question by this incident, and so far I haven't heard a solid explanation to convince me that understanding is correct.
« Last Edit: May 02, 2016, 10:31:36 am by Thom »
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12915
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: Hard lesson learned about DNS
« Reply #10 on: May 02, 2016, 11:33:28 am »
However, you both seem to be missing the point I'm making concerning the empirical evidence this incident highlights. I see no point in repeating myself or the raw facts I've presented, but aside from pc's speculation that there were multiple problems coincidentally involved, neither of you have explained why huge swaths of the Internet could not reach my server's fixed IP address while others could. You seem to be saying that DNS was not the issue, and if so you haven't explained what took place. It seems clear that DNS was the entire reason for the incorrect routing despite IP addressing being correct on source and destination.

That happened to me aswell recently. The cause of this was probably the
"peering" of your datacenter and a technical issue there.

Technically, your datacenter has several "partner" internet providers,
such as Telekom, AT&T and what not. All of them have their own "peer" to
the datacenter and to that peer corresponds a set of "routes" that this
peer handles. For instance, someone from telekom would reach your server
from the telekom peering while vodafone customers would reach it via the
vodafon peering. If one of them fails (due to hardware or configuration
errors, or maybe even a malicious BGP (border gateway protocol)
package), those customers from that peering couldn't reach your server.
They also can't really find a route to your machine via another peering
(e.g. telekom, vodafon, what not) because either there are not other
peerings for the destination network, or the "costs" for that route are
too high (in network design, you want to prevent packages from being
routed in a circle!).

Stuff like that is a little similar to so called "net-splits" as they
are known from IRC.
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4509
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
Re: Hard lesson learned about DNS
« Reply #11 on: May 02, 2016, 04:23:45 pm »
Just a note that you don't have to use the NS servers provided by domain name registrars. There are professional 3rd DNS service providers for example UltraDNS.

//Edit:
By the way I think this topic belongs to another board.
« Last Edit: May 02, 2016, 04:26:07 pm by abit »
BTS account: abit
BTS committee member: abit
BTS witness: in.abit

Offline Thom

Re: Hard lesson learned about DNS
« Reply #12 on: May 03, 2016, 12:32:18 pm »
Just a note that you don't have to use the NS servers provided by domain name registrars. There are professional 3rd DNS service providers for example UltraDNS.

//Edit:
By the way I think this topic belongs to another board.

Interesting, never heard of UltraDNS.

As for the appropriateness of this topic to witnesses I see your point. I posted it here as I thought it was useful information for witnesses to be aware of. All of my nodes for example are listed in the code by domain name and are thus subject to this same type of disruption. Given the plans wackou & I had to implement a multi-tier backbone system it made sense to insulate the code from any shuffling of roles our vps systems needed to have. It also insulates the code from upgrading / downgrading servers, which often end up with different IP addresses afterwards.

There are probably other options if you have the budget for dedicated servers, but using DNS names in the code was our initial method to mitigate changes we anticipated would likely be necessary until the witnesses stabilized and our implementation of the backbone was established. There seems to be little interest in that now, but should the journey to the moon commence it could once again become important.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 4509
    • View Profile
    • Abit's Hive Blog
  • BitShares: abit
  • GitHub: abitmore
Re: Hard lesson learned about DNS
« Reply #13 on: May 03, 2016, 06:30:21 pm »
Just a note that you don't have to use the NS servers provided by domain name registrars. There are professional 3rd DNS service providers for example UltraDNS.

//Edit:
By the way I think this topic belongs to another board.

Interesting, never heard of UltraDNS.
There are some other providers as well: https://en.wikipedia.org/wiki/List_of_managed_DNS_providers

Quote
As for the appropriateness of this topic to witnesses I see your point. I posted it here as I thought it was useful information for witnesses to be aware of. All of my nodes for example are listed in the code by domain name and are thus subject to this same type of disruption. Given the plans wackou & I had to implement a multi-tier backbone system it made sense to insulate the code from any shuffling of roles our vps systems needed to have. It also insulates the code from upgrading / downgrading servers, which often end up with different IP addresses afterwards.

There are probably other options if you have the budget for dedicated servers, but using DNS names in the code was our initial method to mitigate changes we anticipated would likely be necessary until the witnesses stabilized and our implementation of the backbone was established. There seems to be little interest in that now, but should the journey to the moon commence it could once again become important.
When I was managing some tens of servers for a web site, I've ever used DNS-based load balancing, I know it's not bad, and cheap. Managed DNS service helped much.

By the way, we have more things to do once "the journey to the moon commence" . For example, current Graphene code only resolves DNS names at startup, and it even won't start if failed to resolve one of the address ( see https://github.com/cryptonomex/graphene/issues/618 ).
BTS account: abit
BTS committee member: abit
BTS witness: in.abit