Good morning, we are about to perform a database upgrade. This is a software change that will allow us to make faster, zero-downtime, data migrations in the future, as new features are added.
Expect a few minutes downtime, this post will be updated once done, thanks!
UPDATE: Done. This was completed with a few minutes of downtime and has been tested with success.
A database upgrade will take place this morning, a few minutes downtime expected. I’ll update this post once it’s done.
UPDATE: all done now System was out for approx 8 minutes earlier.
The main database server is running on an old KVM instance, which Linode have advised should be migrated. With v3.6 releasing this week, now is the time to do it, so please expect some downtime this week. I’ll comment here once complete.
Surprise, surprise! Linode just decided to suddenly kick off a database migration – yes, during the peak time in the week, and this will take 3.5 hours. I’ve been on the phone to their support and nothing can be done about it.
UPDATE (Tues 00:30 PST)
The migration did not go well and the database was restored to an earlier state. For reasons that are not clear, backups are also not able to recover recent data, though a scheduled backup plan is in place and has been tested in the past (but not on the current server, which was recently upgraded).
As it would take more than a day to investigate the cause, we’ve made the hard decision to go ahead and restore an earlier version of the database, which will mean some users will see subscriptions reverted to an earlier state. (Fortunately, Discover and Play Later lists will mostly be unaffected.) I’m very sorry that’s happened, but it was considered better to restore the service than delay any further.
I’ll continue to investigate to see if the more recent subscriptions can be restored, and will plan to upgrade the backup and restore process longer term.
(Fortunately, Play Later lists will be relatively unaffected.)
The core database will be down for a few minutes today as we increase capacity on it. There’s also a new read-only mode for the website and app which will be operating for most of this time, to minimise downtime.
Some extra web servers were added in past 24 hours. Downtime was mostly avoided, though there two periods of approx 30 minutes with outages or slow performance.
Currently a background processor is being upgraded, meaning that feeds won’t be fetched and importing will be suspended for next 2-3 hours. Once it’s ready, it will be possible to fetch more feeds and at a faster rate.
Remaining Linodes are being upgraded shortly, stay tuned!
Update Job complete, all Linodes are now running on Linode’s new KVM infrastructure. Downtime was approx 60 minutes.
Servers will be cycled this morning to take advantage of Linode’s new KVM setup. It’s been running a while in staging without hassle.
Update: 20:24 PST – This is now resolved. If you were seeing this issue, please pull-to-refresh and all should be restored. 
Some users have reported an app crash today, which seems to be the result of a server update involving how timestamps are stored .
We’re still investigating the issue. For most users, it should be fixed now. You may need to perform a pull-to-refresh (under Shows/Subscriptions) first. The only issue we’re still aware of is some users seeing a crash when scrolling down the series grid.
We should be able to release an update in the next few hours, but until then, the playlist view and detail screens should work fine for browsing.
Sorry for the inconvenience, we’ll get this sorted soon.
Short detail: we’ve moved to millisecond-level resolution to improve caching performance. Although a number of tests were in place, we didn’t have a test specifically for the format of the time in server responses. The phone did not taken kindly to those decimal places, which we knew, but in one field they inadvertently remained, only once new episodes started to be updated, and that’s what caused this.
This was a related issue where a few series had their episodes not showing the series ID, causing the crash. The API’s been updated to ensure episodes always honor the series from whence they came.
As part of Linode’s scheduled maintenance I referenced here earlier this week, there will be a period of anywhere between 30 minutes and 2 hours downtime starting at 19:00 PST tonight (3am UK time). International times here.
There are a number of scheduled downtime windows this weekend where the system can be reconfigured, load-balance, or gracefully degrade to work around the downtime (one is happening as I type this), but 19:00 is the slot which knocks out the main database, so it’s an inevitable downtime.
The Android app has been updated to show downtime details should any user-initiated requests not reach the server. Of course, most app functionality will work as normal, including streaming episodes (since Player FM doesn’t host or proxy them). Browsing shows, adding to Play Later and even subscriptions are possible, but they will be delayed getting to the server until it’s back and running. The main issue with downtime is how it affects new users, as not much can be done without initial setup.