A firewall regular expression rule caused widespread HTTP 502 errors throughout the internet…
Last week, Verizon caused a major BGP misroute that took large chunks of the Internet, including CDN company Cloudflare, partially down for a day. This week, the rest of the Internet has apparently asked Verizon to hold its beer.
Facebook and its properties WhatsApp and Instagram have suffered widespread outages relating to image display for most of today. The problem seems to be bad timestamp data being fed to the company’s CDN in some image tags; when I looked into the broken images littering my own Facebook timeline, I discovered different timestamp arguments embedded in the same URLs. Loading an image from fbcdn.net with bad “oh=” and “oe=” arguments—or no arguments at all—results in an HTTP 403 “Bad URL timestamp”.
If you dig into the code, Facebook refers to the same image URL with different arguments in its anchor tags and its image tags; in the posts with broken images, the URLs in the anchor tags work, but the ones in the image tags don’t. (On the plus side, now everybody is suddenly noticing the AI-driven hidden alt tags the company has been embedding in its images since 2017.)
Twitter has suffered some as-yet-unexplained hiccups in its direct messaging service
today as well. The outages appear to be mostly in the Eastern United States and Europe, with few or no reports shown in other regions on several third-party outage-tracking sites.
No common threads appear to link any of these outages—it’s just a bad week to be an Internet engineer.