A traditional software release circle starts with intensive development. The release early, release often model suggests to release incomplete, unstable versions as early as possible, making it very clear that they shouldn’t be used in production. At some point the branch is declared feature frozen, so from that date only bug fix and low risk optimisations will be merged.
The point of this article is that MySQL 8.0 and 5.7 follow a new release policy that exposes users to unreasonable risks. MariaDB follows the policy historically used by old MySQL versions, which is reasonably safe.
UPDATE: Both the problems described here were solved on Percona side. I will leave the article as it is, because I still consider these events a good example of the problems that MySQL current policy can cause.
The risks of MySQL current policies
MariaDB follows this policy, though not too strictly. MySQL used to do the same. But something changed:
- Years ago Oracle decided to stopped commiting changes to public repository before they appeared in official releases. Because of this, one doesn’t know what MySQL is going to do before it happens. As we’ll see, this is a huge problem.
- Oracle 8.0 started introducing several new features at every release, including changes that break backward compatibility.
- Recently even 5.7 is introduding risky changes.
The consequences of this may include more bugs and broken compatibility with existing tools.
We have two unpleasant examples of this:
- MySQL 8.0.22 and MySQL 5.7.32 broke compatibility with Percona Xtrabackup, which is the preferred tool to take backups. There is no workaround for this. Percona is working to fix the problem on their side, but this may take time. In the meanwhile, they recommend to delay the upgrade. The kind of recommendation we should never, never see in a normal situaiton.
- This already happened earlier this year: MySQL 8.0.20 also broke compatibility with Xtrabackup. At this point one could suspect that this kind of incidents happen on purpose, or at least Oracle doesn’t pay any attention in avoiding them. Because it was very clear that changing InnoDB transaction logs format would break Xtrabackup, especially after the first time.
Let me quote Vadim Tkachenko from Percona:
What to do
As a consultant company, Vettabase sees the problem on users standpoint. Users are left with these options:
- Not knowing about the problem at all. This is the case for most users, the exceptions are those that test their backups (please do it, we can help), and those that closely follow community social network profiles.
- Purposely ignore the problem, and have no remedy for data losses.
- Don’t upgrade or downgrade temporarily – keep in mind that downgrading requires a logical backup in these cases.
- Quickly build automation for a different backup method.
- “Quickly” migrate to another DBMS. The most obvious choice would be MariaDB, because the effort to migrate to a completely diferent DBMS could simply be unacceptable.
All of these options have unpleasant drawbacks.
How MariaDB differs
Incidentally, MariaDB compatibility with Xtrabackup was also abandoned, but the way it happened is completely different. MariaDB development diverged from MySQL, and Percona chose not to keep their tool compatible with recent versions. This happened naturally, without MariaDB adding major changes in GA branches. Both companies did nothing wrong.
MariaDB users can enjoy mariabackup, a fork of Xtrabackup developed by MariaDB itself that fully supports MariaDB features.
That said, am I implying that MariaDB release model is perfect? No, I’m not. MariaDB introduced massive changes to InnoDB in10.5.4, which was the first GA release of the 10.5 branch. In the release notes, you can see the links to several InnoDB tasks, mostly related to contention reduction.
This is risky for GA branches early adopters. But once a branch is declared GA, major features will not be added to it.
Some people don’t want to wait for a long time to see new features. While I don’t see any urgency, I would recommend them to use beta versions. MariaDB still publishes beta versions, something that Oracle doesn’t do nowadays. Using them is not necessarily more risky than using a GA branch with new important features in InnoDB, in the optimiser and in the security area (something that happens often with MySQL). But my recommendation is to use these branches for replicas that are not critical.
Note that MariaDB still uses public repositories (you can currently see the early commits for 10.6) and
What to do
We discussed the risks of MySQL release policy, but what to do to avoid or mitigate them?
Considering migration form MySQL and MariaDB
⇒ Recommended
At Vettabase, we don’t like simplistic statements like “technology A is always better than technology B”. MySQL has many unique features that MariaDB doesn’t have. The opposite is also true of course, but the point is that you may depend on MySQL, and you may enjoy its features. Maybe it’s sensibly faster for your specific workload. Maybe you take backups with snapshots, which is quite common between AWS users. Maybe changing your existing MySQL automation to work with MariaDB would be too expensive.
But new releases of a GA (stable) version should be predictable. This has a great value for the whole organisation. So, if this were the only criteria (and it is indeed an important one), we would definitely suggest to switch to MariaDB.
Considering Percona Server
⇒ Partly recommended, read carefully
Percona Server is an obvious alternative to MySQL. In theory no change in tools and applications is needed to switch from MySQL to Percona Server – please test if it’s true in your particular case before migrating.
But this move wouldn’t really solve problems, because Percona Server is based on MySQL. This means that:
- It imports all its new features and… bugs. If MySQL introduces non-mature features, Percona Server will do the same.
- Percona policy seems to be to delay updates when MySQL breaks compatibility with its tools. But what if a MySQL update contains urgent security fixes? In this case upgrading or not will be a hard choice, for both Percona and users.
Most importantly, making an informed choice is impossible, because Oracle does not release information about security fixes.
Staying on MySQL
⇒ Not recommended
As mentioned, staying on MySQL is definitely an option. But we recommend to be careful. Before upgrading production servers, test each minor upgrade thoroughly. Use pt-upgrade to check if your queries work fine and be fast with the new version. Test your tools, including backups and monitoring.
Migrating to a different DBMS?
⇒ Not recommended for existing projects
We don’t recommend to switch to a different technology. There can be reasons to do so, but MySQL is not one of them. You would have to refactor queries, code, and tooling. The cost may not be reasonable, give that you could switch to MariaDB. We often highlight that MariaDB and MySQL are different products. But at least they are similar, which helps reducing the migration cost and risks.
Also, before switching to another DBMS, learn more about their release policy. For example, do you know that PostgreSQL does not have a public bug tracker? This makes it hard to find out which known bugs affect a certain version, or if/when a particular bug was fixed. And what about proprietary DBMSs? We know very little about their bugs and release policies.
Federico Razzoli
0 Comments