- The purpose of this file is to provide a place where the administrators can record changes and repairs to the system.
- Unless you are an administrator, please do not touch this page.
July 23 2018: mediawiki can't talk to the mysql server
- The following error was reported: The user links in OEIS are broken. For example, Jeffrey Shallit appears as "_Jeffrey Shallit_" in A317177 instead of a link to his user page.
- There were also other reported errors. For example, several people reported getting the error message “An unexpected error happened while authenticating.”
- David Applegate said:
I've taken a brief look.
Top-level problem is that mediawiki can't talk to the mysql server. That breaks the wiki, and also breaks login to the main OEIS.
The cause of that looks like either the mysql database is corrupted, or there's some kind of version mismatch. mysql crashed last night around 9pm, and during the restart tossed out a few errors (from /var/log/mysql/error.log.1):
2018-07-23 21:57:11 139686236439104 [ERROR] Column count of mysql.db is wrong. Expected 22, found 20. The table is probably corrupted 2018-07-23 21:57:11 139686236439104 [ERROR] mysql.user has no `Event_priv` column at position 29 2018-07-23 21:57:11 139686236439104 [ERROR] mysqld: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler.
From an earlier message: 2018-07-23 21:57:11 139686236439104 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 42434347594 It appears to be running Percona XtraDB. I don't know if this is intended or not, or whether perhaps the mysql version got upgraded after the server was started and the db never migrated.
A few other notes: In the syslog, I noticed a couple of errors we probably need to address:
An error from mailgun-sendmail (via cron) complaining about "flag provided but not defined: -o" A warning from oeisdb: "Server is runing in secure-auth mode, but wikiuser@localhost has a password in the old format..."
From /var/log/messages, it appearss that the mysql restart was because the system ran out of memory "mysqld invoked oom-killer". And that might be because of a huge number of php-cgi processes.
From poking around on Google, it appears that we may need to run mysql_upgrade to adapt for an upgrade of MariaDB. But it would be nice to check with Russ before doing that.
I'm going to try executing the following sequence:
service mysqld stop cp -rp /var/mysql /var/mysql.backup.20180724 mysql_upgrade service mysqld start
to attempt to upgrade the mysql databases.
Notes: running "mysql_upgrade" requires mysqld to be running, so restarted mysqld first. Then got error
david@oeis4:~$ sudo mysql_upgrade Version check failed. Got the following error when calling the 'mysql' command line client ERROR 1275 (HY000): Server is running in --secure-auth mode, but 'root'@'localhost' has a password in the old format; please change the password to the new format FATAL ERROR: Upgrade failed
stopped mysqld via "sudo service mysqld stop". restarted mysqld without authentication via "sudo mysqld --skip-grant-tables" More progress, but failed, presumably because the privilege files weren't loaded, and it wanted to upgrade them.
Reset mysql root@localhost password to *******************************
ran "sudo mysql_upgrade -u root -p' and gave it the password. This appeared to work
Stopped and restarted mysqld (sudo service mysqld stop; sudo service mysqld start)
Checked that /var/log/mysql/error.log no longer complains about table sizes.
Suspect problem is messages of the form ERROR 1275 (HY000): Server is running in --secure-auth mode, but 'wikiuser'@'localhost' has a password in the old format; please change the password to the new format
Verified that I couldn't log into mysql using user wikiuser (with same error).
Reset the wikiuser password using
update mysql.user set Password=PASSWORD('...') where User = 'wikiuser';
The above is a longer email detailing the steps. But the short form is 1. I didn't know the mysql root@localhost password, so reset it (value contained in previous email, but you may want to reset it again to one that hasn't been in email). 2. Ran mysql_upgrade to upgrade some of the tables to reflect new version. 3. Reset the mysql wikiuser@localhost password to the secure_auth form (that is, using PASSWORD('password')) but didn't change the actual password (used the one from LocalSetttings_pw.php). 4. Restarted the mysql server.
Along the way, I made a copy of /var/lib/mysql in /var/lib/mysql.backup.20180724. Once you've confirmed that things seem to be working, that can be deleted.
This seemed to fix the problems.