I should clarify I wasn’t a upper level sys admin managing those servers, I just used them or maintained accounts being a rank and file technician

While I get the fundamental concept of DNS as a phonebook for your IPs. I am not sure why it is joked around if something goes haywire or someone breaks something.

Is it because if you get no DNS, people can’t log in through their AD accounts, browse the Internet?

Afaik DNS is a bit of a rabbit hole topic, maybe that’s why people joke about it due to DNS being this “No one really knows how this magic name matching box works”?

Please correct me, I’d genuinely like to know why this is prevalent from you guys.

  • kobra@lemm.ee
    link
    fedilink
    arrow-up
    124
    ·
    edit-2
    3 months ago

    There’s even a haiku:

    It’s not DNS
    There’s no way it’s DNS
    It was DNS

  • Shadow@lemmy.ca
    link
    fedilink
    arrow-up
    87
    ·
    3 months ago

    100% of the internet depends on it, and 90% of technical people can’t be bothered to learn how it works and understand it. Partly because they only touch it once every 5 years. They get what they need done but don’t understand why it worked, so it ends up feeling like black magic to them.

  • Nougat@fedia.io
    link
    fedilink
    arrow-up
    43
    ·
    3 months ago

    I can speak to Windows DNS and AD.

    Yes, the basis of DNS is to match IP addresses with hostnames. But there are plenty of other kinds of DNS records. SPF, DKIM, DMARC use DNS for email validation; AD uses SRV records to point clients to find domain resources. There are also reverse zones, where you send a query for an IP address to learn the hostname(s) associated with it. There are CNAME records that associate a hostname with whatever IP address another hostname is using.

    Point being, there are a lot of different interconnected layers, and if some entry is wrong, it can easily have a cascading effect that creates a problem with something entirely different.

    I would also extend the “It’s always DNS” trope to “It’s always name resolution.” In a Windows environment, at least, DNS name resolution is only used when a FQDN (fully qualified domain name) is being queried. If your application is only looking for [ServerName], NetBIOS name resolution is used.

    When NetBIOS name resolution is used, that is querying a WINS server (if one is specified and exists). In the absence of a WINS server, the query is sent to the “master browser” on your network segment. What machine is the “master browser”? By default, all machines on a network segment (subnet) have a browser election behind the scenes to choose the master browser, based in large part on the machines’ hardware capabilities.

    This means that any machine on the segment (again, by default) can become the master browser and respond to NetBIOS queries. Laptop, desktop, server, doesn’t matter. There are registry entries that can be manipulated to have manual control over what machine is master browser, what machines are and are not allowed to participate in elections, but it is rare that those are configured.

    Why do I know all this? Because about 25 years ago, I came across a situation where different machines were configured with different overlapping subnet masks, and users were trying to browse \\servername\share, and getting only partial results. It took a long time to narrow that down to the misconfigured subnet masks. At the company I’m currently at, the developers still use NetBIOS names instead of FQDNs in applications.

    Always use FQDNs in any in-house development. Always use FQDNs when accessing resources by hostname. And if “some weird, inexplicable, possibly intermittent issue” is happening, check name resolution.

  • shalafi@lemmy.world
    link
    fedilink
    English
    arrow-up
    28
    ·
    3 months ago

    I got a story that perfectly illustrates the meme.

    Had done a solid job of setting up my first domain, from scratch. Everything was tooling along nicely for months until my administrator account kept locking itself, every few minutes.

    Logged in with another admin account and hunted for the issue for a month. Since it was affecting only my account, none of the users had issues. Finally found a single sentence in an obscure forum that pointed me. My DNS issue was buried deep in the DHCP settings.

    Despite knowing better, I had used my personal account to authorize interactions between the DNS and DHCP services. When I changed my password, DNS was still trying to use the old credentials, over and over again, locking my account.

    HOLY SHIT! If you google “dns haiku” my image is in the front page twice! Love it!

  • tiny@midwest.social
    link
    fedilink
    English
    arrow-up
    25
    ·
    3 months ago

    When it is the cause of a problem it’s not always obvious at first so you spend hours troubleshooting the broken app until you look at DNS and a simple DNS issue

    • qaz@lemmy.world
      link
      fedilink
      arrow-up
      3
      ·
      3 months ago

      I’ve had a problem like this yesterday. I couldn’t access my paperless instance. I eventually figured out I could access it with an IP + port combination and the DNS lookup failed.

    • Landless2029@lemmy.world
      link
      fedilink
      arrow-up
      1
      ·
      3 months ago

      Often because we know how badly things can go wrong with so many components we start at the end of the app instead of the beginning.

      Similar to how tech support always asks of you rebooted. We often don’t confirm basic connectivity issues.

  • bfg9k@lemmy.world
    link
    fedilink
    arrow-up
    17
    ·
    3 months ago

    DNS failure can manifest in strange ways and have a sysadmin scratching their head as to why some devices are working fine (statically configured/running from DNS cache), but others cannot access the internet or any of their work services.

    It’s usually the last thing you suspect, because DNS always just works, right?

    • PlexSheep@infosec.pub
      link
      fedilink
      arrow-up
      9
      ·
      3 months ago

      You’d think so until you have dealt with a few DNS problems. At some point, the mantra of “It’s always DNS” stays in your head when troubleshooting. It’s often the first thing I try nowadays.

  • IphtashuFitz@lemmy.world
    link
    fedilink
    English
    arrow-up
    15
    ·
    3 months ago

    I never would have thought of it but I recently saw a novel use of DNS to exfiltrate data from a compromised server.

    My employer takes security very seriously. Our public facing web servers are very thoroughly locked down, or so we thought. We contract with companies like HackerOne to perform penetration testing etc. One of their white hat hackers managed a remote command attack, and copied data off of the server via a string of DNS queries.

    Suppose the hacker owned the domain example.com, and he had his own authoritative nameserver for it. He just ran a series of commands that took, for example, a password file, and ran DNS queries for line1.example.com, line2.example.com, line3.example.com and so on for each line in the file. As a result the log file on his DNS server collected each line of the password file as it responded to each query.

    • archomrade [he/him]@midwest.social
      link
      fedilink
      English
      arrow-up
      4
      ·
      3 months ago

      I’m trying to digest this

      You’re saying he was stealing data from the target server by appending it line-by-line to dns requests sent to his nameserver? Wouldn’t he have needed to both be on the target server and already have access to the data?

      • IphtashuFitz@lemmy.world
        link
        fedilink
        English
        arrow-up
        7
        ·
        3 months ago

        Our web servers are locked down in such a way that you can’t copy data off of them using standard protocols like scp, ftp, and even http, etc. Our firewall blocks all such outbound traffic.

        This hacker found a bug in a framework used on our web servers that let him execute commands remotely. When commands to copy data off the server failed using those more typical methods he switched to a more novel (and difficult) method of leveraging DNS instead. He discovered we weren’t locking DNS down the same way we were locking other protocols down and used that as a way to extract data from our server.

        • archomrade [he/him]@midwest.social
          link
          fedilink
          English
          arrow-up
          8
          arrow-down
          1
          ·
          3 months ago

          Ah, ok, that makes sense! So there was a separate bug in the framework that granted him limited remote access, but because the server had tight control over outbound connections he had to use a novel way of getting the data back out

          Basically: He crawled in through the sewer and then robbed the bank one stack of bills at a time via pigeon courier.

      • JackbyDev@programming.dev
        link
        fedilink
        English
        arrow-up
        4
        ·
        3 months ago

        Yes, but it’s not necessarily as simple as having full ssh access or something like that. Plus getting data out by DNS queries is probably much harder to detect than something like sftp or http posts.

      • PlexSheep@infosec.pub
        link
        fedilink
        arrow-up
        2
        ·
        3 months ago

        Could be used to exfiltrate data when you only can make commands, but not see their output. There might be other and easier exfiltration possibilities then, but this is a creative solution that uses a very common protocol and will probably be available on any machine.

  • JustARegularNerd@lemmy.world
    link
    fedilink
    English
    arrow-up
    13
    ·
    3 months ago

    Only a support tech chiming in, so far I’ve found when it goes wrong, it causes errors or behaviours that are unusual and could be hard to trace back. Clients might be confused as to why their laptop isn’t connecting to some services but their co-worker still can.

    I’ve currently got an infuriating issue where the DNS on my modem just dies at seemingly random intervals. I set up a monitor using Uptime Kuma to let me know when it goes down, and ever since it just hasn’t been a problem yet so I have no idea why it’s going down. I might just set up a pihole and just work around the problem.

  • brygphilomena@lemmy.world
    link
    fedilink
    arrow-up
    12
    ·
    3 months ago

    Because so few understand it and so many things use it.

    If you read a guide on setting up a website. They might have you change a DNS record and you might not realize it’s doing something else. Web developers frequently want to make changes to DNS and will change the name servers away to theirs but not migrate any of the records for anything besides the website. They’ll break EVERYTHING but hey, the website will load.

    If you read a guide on connecting some service like hubspot. They will have you add spf records. But those need to be included with the existing ones and not just replacing what’s already there. Mess it up and every single email you send will get sent to spam folders.

    • dreugeworst@lemmy.ml
      cake
      link
      fedilink
      arrow-up
      3
      ·
      3 months ago

      clueless dev who very rarely touches web apps here, what things would break if you dont touch other records besides those for tour website?

      • brygphilomena@lemmy.world
        link
        fedilink
        arrow-up
        5
        ·
        3 months ago

        A records are usually fine for web dev, and some cname records. But if you move DNS to different name servers it doesn’t take the MX, srv, or txt with you.

        Name servers are where all the DNS records are hosted. It tells every computer who is the authority on the information. If you change those without moving all the DNS records to the new location. All the old ones are no longer used at all. Even if they still exist on the old system.

        Also spf records might also reference the websites a record to allow the website to send email without getting flagged as spam. If the site moves, but emails are still sent from the old IP address for a myriad of reasons it would break email. Like if they used some provider that hosted the site and emails on the same system, which used to be more common than it is today.

        • Natanael@slrpnk.net
          link
          fedilink
          arrow-up
          2
          ·
          3 months ago

          Some dumb shit I see is setting SPF so Google is a trusted origin for email “to solve issues with sending to Gmail addresses” when what you’re supposed to do is add your mail servers as trusted origin.

          Directionality, how does it work?

  • dbx12@programming.dev
    link
    fedilink
    arrow-up
    12
    ·
    3 months ago

    Might be related to long TTL on DNS entries making it hard to connect one failure to a thing you changed days ago.

  • some_guy@lemmy.sdf.org
    link
    fedilink
    arrow-up
    12
    ·
    3 months ago

    Sorry this doesn’t explain anything, cause I think others have already put in the work. At my old job, there was a slack icon for “Is it DNS?” because it’s often DNS. You already know and that’s why you asked, but I’m just reinforcing that this is how common it is.

  • tomkatt@lemmy.world
    link
    fedilink
    English
    arrow-up
    9
    ·
    edit-2
    3 months ago

    DNS is often misconfigured.

    On the linux side of things, people like to manually edit /etc/resolv.conf when it’s actually a symlink and changes to it don’t persist on boot (the real file location varies, but it’s usually in something like /etc/system/resolve). And forget bind9, if it’s not MS DNS it’s not DNS to some folks.

    On the Windows side, people love to ignore that reverse DNS exists, even though so many things use it. They also freaking love CNAME aliases and break stuff in interesting ways (for example, a “load balanced” configuration that’s all just the first node acting as all three nodes of a cluster or pool).

    Many people only know enough DNS to be dangerous and come up with really jank workarounds to get things running because they don’t understand the proper solutions.

  • scsi@lemm.ee
    link
    fedilink
    arrow-up
    9
    ·
    3 months ago

    In addition to the other comments which more directly address your question, DNS has been / can be used to exfiltrate data from “secure” networks. Search “dns data exfiltration” in your favourite search engine and you’ll get several high quality articles. Typical mitigations might be to limit which DNS servers your network can contact, restrict packet sizes to the bare minimum which valid use would have and so forth.

  • hendrik@palaver.p3x.de
    link
    fedilink
    English
    arrow-up
    4
    ·
    edit-2
    3 months ago

    What’s an example of a joke? Because that’s not what I experienced… In my world people make similar jokes if you mess up any aspect of networking. Same if you break the routing. Or mess up the servers, replication or virtualization infrastructure. DNS is one aspect and people will crack a similar amount of jokes as they do with any other f…-up. (It’s a way to cope with a stressful day or whatever. Or to get some relief.)

  • xmunk@sh.itjust.works
    link
    fedilink
    arrow-up
    2
    ·
    3 months ago

    Because it’s a fucking unauthoritative clown show that is necessarily decentralized which leads to all sorts of difficult to diagnose (and sometimes impossible to quickly fix) issues.

    It’s like 100% footguns.