Skip to content

Player FM Status

Just the facts ma'am about upgrades and downtime

Linode has advised it will be doing some critical updates this weekend, which might result in some downtime. While the app can survive most servers being knocked off on an individual basis, this downtime promises to be “extra special” as each window entails downtime of multiple servers.

I’m preparing to show status updates within the app, in the event connections can’t be made, and will be online when it happens to ensure downtime is minimal.

I will shortly be upgrading server infrastructure.

I should be able to isolate servers to avoid any disruption, though there’s a small risk of downtime or slightly slower performance during the update.

We’re about to update the catalogue to use a new underlying structure and storage mechanism. It’s unlikely to cause any downtime, but heads-up in case you notice new content or topics are slower to load.

A few components to upgrade today, should be some short downtime ahead.

Update: Took site down approx 16:10 UTC, back up approx 16:20 UTC.

Servers will go down soon for a planned upgrade of various infrastructure. Expect 10-15 minutes downtime.

I’m upgrading remaining machines to the new Linode setup today. Expecting just a few minutes downtime.

There has been a syncing issue over the past few days which caused some feeds to not be updated correctly.

The main cause was over-excited logging. After the recent infrastructure upgrade, there was a problem that caused logs to not be cleared correctly, impacting on both performance and memory usage. This has been fixed now and should also avert website and API slowdown. I’ve also simulated the cleanup process to ensure it won’t happen again.

The main feed-processing servers have also now been turbo-charged, thanks to Linode’s recent and delightful mega-upgrade. It doubles RAM (which was already doubled a year ago!), brings faster processing by way of SSDs and processor upgrades, and improves network speed. Altogether a big win.

Having said that, I’m still aware there may be another issue with some feeds being marked as stale and not reverting from that. I’m still monitoring carefully and meanwhile I’ve also been developing an open source library which will ultimately make the whole process more robust and performant. UPDATE: Identified and fixed the issue. The regular re-fetch cycle wasn’t happening due to a software bug that slipped through (yes, there’s now test coverage on it). The reason it wasn’t detected is feeds were still being updated upon receiving push notifications, so it was only those where a notification was missed that were never re-tried.

Linode’s Newark center went down around 10 minutes ago. This affects core Player FM infrastructure.

The mobile client will continue to work, playing and streaming, but the website is down and actions such as searching and discovering new channels are not possible. Subscribing and favoriting on the mobile client is possible and will be sent back to the server after downtime resumes.

Watching Linode’s status blog closely and hopefully back online shortly …

Update: 1:43am PST – I’m relieved to report service is now restored. Sorry for this morning’s outage. It seems to have been a rare one-time incident at Linode’s center involving power and network outages, then compounded by hardware failure.

An issue has arisen with the web host, outside of my control and taking down the website and services for mobile apps. The host, Linode, is aware and fixing it now.

The web server went down about 20 minutes ago due to a security measure Linode had suddenly applied to a neighbouring node. It’s running again now, sorry for the convenience.