Recovering from v20.3.0.13059 upgrade failure

I used the filecloudcp CLI to upgrade FileCloud on my CentOS 7.9 VM, but there were errors. I didn’t save the log from that process.

Since my installation was working at v20.2.4.12080 and since I’m on a Digital Ocean droplet with weekly backups, I chose to restore my droplet from a backup. But now when I go to https:///admin the webpage is empty.

This wasn’t expected. I thought that FileCloud would simply work. What is the recommended process for recovering from a restored backup of VM, in order to get FileCloud to work again?

Hi @Unimatrix

Full restoration from the snapshot usually works fine. Please check and confirm MongoDB service is up and running on the server.

MongoDB is running. I should have mentioned that I receive a daily FileCloud Admin Summary email, so at least something in the background appears to be working. I just can’t get into the UI.

Any recommendations for how to investigate, reset and/or repair the FileCloud installation especially with regard to accessing the UI?

@Unimatrix

Can you please upload the below mentioned logs

  • Apache logs
    Linux : /var/logs/apache2/access.log
  • Server logs
    Linux : /var/www/html/scratch/logs/log_YYYY-MM-DD.txt
    -Mongodb logs
    Linux: (Ubuntu): /var/log/mongodb/mongd.log

You can upload the logs to this link

I’ve uploaded the files requested (access, log, scratch logs from the day I posted this question and one from Jan 3, and the mongodb log.

@Unimatrix

We will review the logs and get back to you today itself

Any news about this? Thank you.

@Unimatrix

Can you check if you can access the user portal and also can you please share us with the developer console window when accessing admin portal.

I can’t access the user portal or the admin portal over the web. That’s the reason I posted this support request. I’m not sure what you mean by the “developer console window”. Perhaps you mean an SSH session in a terminal window?

But it doesn’t matter. This is taking too long to resolve. I’ve decided to reinstall FileCloud on a fresh VM and give it one more try. I’ll be more careful about backing up the VM just before I use the filecloudcp CLI to run a FileCloud update. This problem was caused indirectly because the update failed the last time I did that. It’s still a mystery as to why a VM snapshot didn’t bring everything back online, but for now that will have to remain a mystery.

One more thing, it says I’m watching this forum thread but I am not receiving notices when you post a new comment.

With a completely new install on Centos 7.6, which worked at first when setting it up, ultimately the same empty webpage ended up happening after I rebooted the server, following all basic configurations in the Admin UI were completed, and all basic and extended checks passed.

I now suspect that the blank web UI has something to do with the fact that I use Letsencrypt certificates and virtual hosts — as recommended by the Letsencrypt documentation.

The daily Summary Email from FileCloud is still being delivered, as before, but, also as before, it shows this:

Error: —Mod Rewrite Apache Configuration Setup Failed

To reiterate, when I first installed FileCloud the UI was working. Then after I rebooted, after all configurations were completed, the UI became blank.

According to your documentation at Mod Rewrite Setup Check Fails during Install there are only three things to verify in case of the above error message:

  1. Verify that the mod_rewrite apache module is properly installed and activated
  2. In Apache configuration file for the site, ensure “AllowOverride All” is set correctly for the site.
  3. Ensure, that the Apache .htaccess file is present in the WWW Root (say /var/www or c:\xampp\htdocs)

These are all true in my server (though directories differ because I’m not using XAMPP).

Here is the apache run configuration:

[root@filecloud httpd]# httpd -t -D DUMP_RUN_CFG
ServerRoot: "/etc/httpd"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/etc/httpd/logs/error_log"
Mutex mpm-accept: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex authdigest-client: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex authn-socache: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/run/httpd/" mechanism=default
PidFile: "/run/httpd/httpd.pid"
Define: _RH_HAS_HTTPPROTOCOLOPTIONS
Define: DUMP_RUN_CFG
User: name="apache" id=48
Group: name="apache" id=48

Here are my vhosts:

[root@filecloud ~]# httpd -D DUMP_VHOSTS
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server cans-mci.com (/etc/httpd/conf.d/cans-mci.com-le-ssl.conf:2)
         port 443 namevhost cans-mci.com (/etc/httpd/conf.d/cans-mci.com-le-ssl.conf:2)
                 alias cans-mci.com
         port 443 namevhost filecloud.cans-mci.com (/etc/httpd/conf.d/ssl.conf:56)
         port 443 namevhost www.cans-mci.com (/etc/httpd/conf.d/www.cans-mci.com-le-ssl.conf:2)
                 alias www.cans-mci.com
*:80                   is a NameVirtualHost
         default server cans-mci.com (/etc/httpd/conf.d/cans-mci.com.conf:1)
         port 80 namevhost cans-mci.com (/etc/httpd/conf.d/cans-mci.com.conf:1)
                 alias cans-mci.com
         port 80 namevhost www.cans-mci.com (/etc/httpd/conf.d/www.cans-mci.com.conf:1)
                 alias www.cans-mci.com

Notice that I did not create the vhost

filecloud.cans-mci.com

I don’t know where its configuration resides. I wonder if your scripts depend on it. Notice also that it points to /etc/httpd/conf.d/ssl.conf, so here are a few lines from that file that might be relevant to this discussion:

SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

Those are NOT the Letscencrypt cert and key. I did not create them. So, although my server name is cans-mci.com and there is a vhost for cans-mci.com, I now wonder if FileCloud uses the filecloud subdomain with the filecloud.cans-mci.com vhost, which points to /etc/httpd/conf.d/ssl.conf.

So I’m wondering if the existence of the cans-mci.com vhost is messing up the operation of your scripts.

I want to provide complete information here for the record. Here is the file listing of /etc/httpd/conf.d

drwxr-xr-x. 2 root root  236 Jan 15 06:25 .
drwxr-xr-x. 5 root root   92 Jan 15 06:25 ..
-rw-r--r--. 1 root root 2926 Nov 16 16:18 autoindex.conf
-rw-r--r--. 1 root root  283 Jan 15 06:25 cans-mci.com.conf
-rw-r--r--. 1 root root  437 Jan 15 06:25 cans-mci.com-le-ssl.conf
-rw-r--r--. 1 root root  625 Oct  1 13:50 php.conf
-rw-r--r--. 1 root root  366 Nov 16 16:19 README
-rw-r--r--. 1 root root 9443 Nov 16 14:44 ssl.conf
-rw-r--r--. 1 root root 1252 Nov 16 14:44 userdir.conf
-rw-r--r--. 1 root root  824 Nov 16 14:44 welcome.conf
-rw-r--r--. 1 root root  295 Jan 15 06:25 www.cans-mci.com.conf
-rw-r--r--. 1 root root  445 Jan 15 06:25 www.cans-mci.com-le-ssl.conf

These files are loaded because of this declaration in /etc/httpd/conf/httpd.conf:

# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf

For the record here are the contents of /etc/httpd/conf.d/cans-mci.com.conf

<VirtualHost *:80>
    ServerName cans-mci.com
    DocumentRoot "/var/www/html"
    ServerAlias cans-mci.com

    # Other directives here
RewriteEngine on
RewriteCond %{SERVER_NAME} =cans-mci.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

For the record, here are the contents of /etc/httpd/conf.d/cans-mci.com-le-ssl.conf

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName cans-mci.com
    DocumentRoot "/var/www/html"
    ServerAlias cans-mci.com

    # Other directives here
SSLCertificateFile /etc/letsencrypt/live/cans-mci.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/cans-mci.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/cans-mci.com/chain.pem
</VirtualHost>
</IfModule>

For the record here is the ServerName definition in /etc/httpd/conf/httpd.conf:

ServerName cans-mci.com

And here is the section of the same file where AllowOverride All is written (with standard comments removed):

# Further relax access to the default document root:
<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>

Here is the first part of /var/www/html/.htaccess:

RewriteEngine On
RewriteRule .*\.gz$ - [F,NC]
#--------------------------------------------------
# Let's Encrypt Support
RewriteRule ^.well-known/(.*)$ .well-known/$1 [L]
#--------------------------------------------------

#RewriteRule server-status server-status [L]
#--------------------------------------------------
# ADVANCED CUSTOMIZATION SECTION - START
#--------------------------------------------------
#Use this rule for customization for Apache 2.4
RewriteRule ^custom/css/(.*)$ resources/customization/css/$1 [END]

#Use this rule for customization for Apache 2.2
#RewriteRule ^custom/css/(.*)$ resources/customization/css/$1 [L]

#RewriteCond %{REQUEST_FILENAME} !-f

#--------------------------------------------------
# ADVANCED CUSTOMIZATION SECTION - END
#--------------------------------------------------

#Route all requests to our handler
RewriteRule ^(.*)/?$ public/index.php [L]

I’m sorry that I have very low expectations that I will get a useful response after this post, but I’m putting down all this information in case others eventually can benefit from it.

I am going to see if removing the vhosts resolves this problem. A result of adding them seems to be overlapping, conflicting, and/or mislaid apache declarations that screw up FileCloud’s hardcoded configuration. Anyway, that’s my theory in the absence of additional feedback from FileCloud engineers.

@Unimatrix

Please let us know if you have a firewall in front of the server. In the meantime, we will also check this with our team.

I do have a firewall at digitalocean’s network layer; its rules are unchanged since I set up the VM a couple months ago; it allows incoming 80 and 443 from all IP’s;

I tried installing again on Centos 7.8 and it succeeded up to the Basic Checks, but Extended Checks gave back a blank webpage. I gave up on Centos.

Then I installed on Ubuntu 16.04 LTS and so far so good. I didn’t find official documentation about how to use Letsencrypt for SSL, so I did this:

Created one vhost:
nano /etc/apache2/sites-available/cans-mci.com.conf

<VirtualHost *:80>
    ServerAdmin <obscured-email>
    ServerName cans-mci
    DocumentRoot /var/www/html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Enabled my domain vhost:
a2ensite cans-mci.com.conf

Disabled the default vhost:
a2dissite 000-default.conf

Added the letsencrypt cert:
certbot --apache -d cans-mci.com

Uncommented this line in /var/www/html/.htaccess
RewriteRule ^.well-known/(.*)$ .well-known/$1 [L]

One oddity during the FileCloud installation was a repeating message toward what seemed to be the end of process (no way to know because this message was in a separate cursor). So chose to Control-C out of that. I feared the worst, but it seems not to have crapped the install:

Jan 20 01:03:06 cans-mci solr[9154]: *** [WARN] *** Your open file limit is currently 1024.
Jan 20 01:03:06 cans-mci solr[9154]: It should be set to 65000 to avoid operational disruption.
Jan 20 01:03:06 cans-mci solr[9154]: If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.
Jan 20 01:03:06 cans-mci solr[9154]: *** [WARN] *** Your Max Processes Limit is currently 15735.
Jan 20 01:03:06 cans-mci solr[9154]: It should be set to 65000 to avoid operational disruption.
Jan 20 01:03:06 cans-mci solr[9154]: If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.
Jan 20 01:03:10 cans-mci solr[9154]: [146B blob data]
Jan 20 01:03:10 cans-mci solr[9154]: Started Solr server on port 8983 (pid=9207). Happy searching!
Jan 20 01:03:11 cans-mci solr[9154]: [14B blob data]
Jan 20 01:03:11 cans-mci systemd[1]: Started LSB: Controls Apache Solr as a Service.

So far so good on this install.