If your Grape instance does seems to not have started properly after an upgrade or reboot please follow this procedure.
Note: Make sure to check common server problems like:
network not available
high CPU load (
Grape is offline / HTTP 502¶
grape status or
grape status performs some basic checks on the stack, such as container and service status.
A successful run looks like this:
grape@grapetest:~$ grape status __ ____ __________ _____ ___ ________ / /___ ______ / __ `/ ___/ __ `/ __ \/ _ \ / ___/ _ \/ __/ / / / __ \ / /_/ / / / /_/ / /_/ / __/ (__ ) __/ /_/ /_/ / /_/ / \__, /_/ \__,_/ .___/\___/ /____/\___/\__/\__,_/ .___/ /____/ /_/ /_/ Grape SETUP Version: 4.0.1 Options for selected command 'status': TODO General options: * verbose False * debug False Running Grape status check * Proxy: Container: running * Sentry: Container: running * Grape: Image: onpremises-4.0.1 Container: running Processes: chatgrape-ws: running grape-check: stopped chatgrape-celery-worker-emails: running chatgrape-celery-beat: running chatgrape-rr: running chatgrape-celery-worker-notifications: running chatgrape-celery-worker: running Command 'status' returned no errors.
If a container shows “migrating” the necessary database migrations are still being run, in this case please be patient and try again after some minutes.
You can also run
docker ps which lists all containers and their status.
Look at the Service status¶
If the service is not running it is necessary to look at the status of the corresponding service.
docker-compose ps in the configuration directory for:
If the status shows that the service has failed please try restarting it with
docker-compose restart and wait until it either succeeds or fails again.
Note: Certain combinations of docker and linux distributions sometimes lead to errors like “Cannot create/destroy containers” or a general hangup while trying to restart containers.
Should this happen you will have to fully restart the docker daemon (e.g.
systemctl restart docker.service) and/or reboot the VM/Server.
Try to restart the proxy¶
docker restart grape-proxy
Before you dig deeper into the cause, you can try to just restart the proxy and see if this recovers the service.
Check the output of the docker containers¶
docker-compose logs -f
This only affects Sentry and Grape.
Check individual container logs¶
The following containers should be checked for Grape:
The command to run is
docker logs -f CONTAINERNAME --tail 100. This shows the last 100 lines of output and follows the log.
If the grape-runtime container failed with
ERROR: The finalization script failed! ERROR: Please try redeploying grape, if the problem persists please contact the Grape support team! ERROR: Exiting
The most common cause is a failed migration due to the database containers failing/starting up too slow.
You can confirm missing migrations by entering the runtime container and checking manually. Use the following commands:
docker exec -it application_grape-runtime_1 bash
if any migrations are not run, they won’t be checked like so
... teams [X] 0001_initial [X] 0002_auto_20170627_0948 [X] 0003_pre_populate_default_teams [X] 0004_pre_populate_default_teams_fixed [ ] 0005_ad_imports_blank # <- not run [ ] 0006_team_crowd_imports # <- not run thumbnail [X] 0001_initial trello [X] 0001_initial
If this is the case, you need to report that problem to Grape immediately.
Restart the runtime container with
docker restart application_grape-runtime_1.
If this didn’t help check the PostgreSQL, Redis and Elasticsearch containers next.
If they are not running try restarting them with
docker restart CONTAINERNAME.
Once the databases are up and running please restart the grape-runtime container next by running
docker restart application_grape-runtime_1.
If PostgreSQL failed with the error
FATAL: the database system is starting up the reason is most likely an unclean shutdown of the database caused by a container or VM/Server crash.
To fix this run the
fix_postgresql_grape.sh helper script located in
Grape is online¶
Problems using Grape (Error 500) - check Sentry¶
Check if you can see an error in Sentry at the time the problem happened. It helps the Grape team if you have an error in sentry that correlates to you problem.
You can open the self-test page in the browser for a quick overview:
Note: only Grape admin users can access this page
Problems after upgrading Grape¶
My Grape instance just show the “We are busy updating our product” page (HTTP 502)¶
Please check the output of the runtime container:
docker logs application_grape-runtime_1 | tail -n 30
Grape (gunicorn process) uses ~100%¶
First try restarting the grape-application by running
cd /data/grape/config/application && docker-compose restart
and wait until it comes back up.
If this does not help:
Inspect what the process is doing:
Find the process id of
gunicornE.g. with ps (as root on the host machine)
ps aux | grep gunicorn
Inspect it using strace
strace -p PID_OF_GUNICORN
A bug appeared after upgrading to a new version¶
Use your VM snapshot to recover the last working version.
If you don’t have a VM snapshot:
Take a VM snapshot!
Look up the previous release in the changelog or your inbox.
Downgrade to that version, e.g.
grape install --custom-version onpremises-3.3.9
YMMV with this approach. Always make a backup before an upgrade!
I want see what
grape commands have been called in the past¶
grape commands logs all it’s output to the file
/data/grape/logs/grape_setup.log, you’ll find all the information about previous runs in there.