Skip to content

Player FM Status

Just the facts ma'am about upgrades and downtime

As part of Linode’s scheduled maintenance I referenced here earlier this week, there will be a period of anywhere between 30 minutes and 2 hours downtime starting at 19:00 PST tonight (3am UK time). International times here.

There are a number of scheduled downtime windows this weekend where the system can be reconfigured, load-balance, or gracefully degrade to work around the downtime (one is happening as I type this), but 19:00 is the slot which knocks out the main database, so it’s an inevitable downtime.

The Android app has been updated to show downtime details should any user-initiated requests not reach the server. Of course, most app functionality will work as normal, including streaming episodes (since Player FM doesn’t host or proxy them). Browsing shows, adding to Play Later and even subscriptions are possible, but they will be delayed getting to the server until it’s back and running. The main issue with downtime is how it affects new users, as not much can be done without initial setup.

Linode has advised it will be doing some critical updates this weekend, which might result in some downtime. While the app can survive most servers being knocked off on an individual basis, this downtime promises to be “extra special” as each window entails downtime of multiple servers.

I’m preparing to show status updates within the app, in the event connections can’t be made, and will be online when it happens to ensure downtime is minimal.

I will shortly be upgrading server infrastructure.

I should be able to isolate servers to avoid any disruption, though there’s a small risk of downtime or slightly slower performance during the update.

We’re about to update the catalogue to use a new underlying structure and storage mechanism. It’s unlikely to cause any downtime, but heads-up in case you notice new content or topics are slower to load.

A few components to upgrade today, should be some short downtime ahead.

Update: Took site down approx 16:10 UTC, back up approx 16:20 UTC.

Servers will go down soon for a planned upgrade of various infrastructure. Expect 10-15 minutes downtime.

I’m upgrading remaining machines to the new Linode setup today. Expecting just a few minutes downtime.

There has been a syncing issue over the past few days which caused some feeds to not be updated correctly.

The main cause was over-excited logging. After the recent infrastructure upgrade, there was a problem that caused logs to not be cleared correctly, impacting on both performance and memory usage. This has been fixed now and should also avert website and API slowdown. I’ve also simulated the cleanup process to ensure it won’t happen again.

The main feed-processing servers have also now been turbo-charged, thanks to Linode’s recent and delightful mega-upgrade. It doubles RAM (which was already doubled a year ago!), brings faster processing by way of SSDs and processor upgrades, and improves network speed. Altogether a big win.

Having said that, I’m still aware there may be another issue with some feeds being marked as stale and not reverting from that. I’m still monitoring carefully and meanwhile I’ve also been developing an open source library which will ultimately make the whole process more robust and performant. UPDATE: Identified and fixed the issue. The regular re-fetch cycle wasn’t happening due to a software bug that slipped through (yes, there’s now test coverage on it). The reason it wasn’t detected is feeds were still being updated upon receiving push notifications, so it was only those where a notification was missed that were never re-tried.

Linode’s Newark center went down around 10 minutes ago. This affects core Player FM infrastructure.

The mobile client will continue to work, playing and streaming, but the website is down and actions such as searching and discovering new channels are not possible. Subscribing and favoriting on the mobile client is possible and will be sent back to the server after downtime resumes.

Watching Linode’s status blog closely and hopefully back online shortly …

Update: 1:43am PST – I’m relieved to report service is now restored. Sorry for this morning’s outage. It seems to have been a rare one-time incident at Linode’s center involving power and network outages, then compounded by hardware failure.

An issue has arisen with the web host, outside of my control and taking down the website and services for mobile apps. The host, Linode, is aware and fixing it now.