Following up here with our solution in case anyone is experiencing the same issue.
Dyn thinks that this is due to: "Google Public DNS does some really weird routing, sometimes shipping requests from one POP to another for responses. If they are shipping a US-based request to a EU POP, this could explain this behavior.". Since we're blocking traffic in the EU, this could cause the weird resolution we're seeing in the US.
Per Dyn, "Google will receive a DNS query, append the originating /24 prefix. in the edns-client-subnet (ECS) data, and send that information to our nameservers. If we see that ECS information in the query, we will take advantage of it in our traffic director response."
Hope this helps someone else someday.
On Tuesday, January 15, 2019 at 5:50:06 PM UTC-5, Jeremy Sadwith wrote:
When using Google's DNS Server (220.127.116.11), we periodically see DNS resolution issues, but none of the other DNS servers have the same issue. You can reproduce by running `dig @18.104.22.168 krk.kargo.com
` a bunch of times in a row in comparison to `dig @22.214.171.124 krk.kargo.com
126.96.36.199 always returns the following answer section...
;; ANSWER SECTION:
krk.kargo.com. 227 IN CNAME kraken.production.us-east-1.kops.kargo.com.
kraken.production.us-east-1.kops.kargo.com. 8 IN A 188.8.131.52
kraken.production.us-east-1.kops.kargo.com. 8 IN A 184.108.40.206
kraken.production.us-east-1.kops.kargo.com. 8 IN A 220.127.116.11
Whereas 18.104.22.168 sometimes returns an authority section instead...
;; AUTHORITY SECTION:
krk.kargo.com. 1080 IN SOA ns1.p24.dynect.net. email.kargo.com. 2019011111 3600 600 604800 1800
There is one error, but we don't think it applies. Is that an incorrect assumption?
kargo.com/DNSKEY: The response had an invalid RCODE (SERVFAIL). (22.214.171.124, 126.96.36.199, 188.8.131.52, 184.108.40.206, 2001:500:90:1::24, 2001:500:94:1::24, UDP_0_EDNS0_32768_512, UDP_0_NOEDNS)
We talked to our DNS provider and were told that this is an issue within Google's DNS server. Any clue how to best handle this?