Troubleshooting

If your Grape instance does seems to not have started properly after an upgrade or reboot please follow this procedure.

Note: Make sure to check common server problems like:

  • insufficient diskspace

  • insufficient memory

  • network not available

  • high CPU load (top, uptime)

Troubleshooting Flowchart:

Grape is offline / HTTP 502

grape status or docker ps

The command grape status performs some basic checks on the stack, such as container and service status.

A successful run looks like this:

grape@grapetest:~$ grape status
                                             __
   ____ __________ _____  ___     ________  / /___  ______
  / __ `/ ___/ __ `/ __ \/ _ \   / ___/ _ \/ __/ / / / __ \
 / /_/ / /  / /_/ / /_/ /  __/  (__  )  __/ /_/ /_/ / /_/ /
 \__, /_/   \__,_/ .___/\___/  /____/\___/\__/\__,_/ .___/
/____/          /_/                               /_/

Grape SETUP Version: 4.0.1


Options for selected command 'status':
TODO

General options:
 * verbose              False
 * debug                False

Running Grape status check
 * Proxy:
    Container: running
 * Sentry:
    Container: running
 * Grape:
    Image:     onpremises-4.0.1
    Container: running
    Processes:
      chatgrape-ws: running
      grape-check: stopped
      chatgrape-celery-worker-emails: running
      chatgrape-celery-beat: running
      chatgrape-rr: running
      chatgrape-celery-worker-notifications: running
      chatgrape-celery-worker: running

Command 'status' returned no errors.

If a container shows “migrating” the necessary database migrations are still being run, in this case please be patient and try again after some minutes.

You can also run docker ps which lists all containers and their status.

Look at the Service status

If the service is not running it is necessary to look at the status of the corresponding service.

Please run docker-compose ps in the configuration directory for:

  • Proxy: /data/grape/config/proxy

  • Sentry: /data/grape/config/sentry

  • Grape: /data/grape/config/application

If the status shows that the service has failed please try restarting it with docker-compose restart and wait until it either succeeds or fails again.

Note: Certain combinations of docker and linux distributions sometimes lead to errors like “Cannot create/destroy containers” or a general hangup while trying to restart containers. Should this happen you will have to fully restart the docker daemon (e.g. systemctl restart docker.service) and/or reboot the VM/Server.

Try to restart the proxy

docker restart grape-proxy

Before you dig deeper into the cause, you can try to just restart the proxy and see if this recovers the service.

Check the output of the docker containers

docker-compose logs -f

This only affects Sentry and Grape.

Check individual container logs

The following containers should be checked for Grape:

  • application_grape-runtime_1

  • application_postgresql_1

  • application_elasticsearch_1

  • application_redis_1

  • application_pgbouncer_1

The command to run is docker logs -f CONTAINERNAME --tail 100. This shows the last 100 lines of output and follows the log.

grape-runtime

If the grape-runtime container failed with

ERROR: The finalization script failed!
ERROR: Please try redeploying grape, if the problem persists please contact the Grape support team!
ERROR: Exiting

The most common cause is a failed migration due to the database containers failing/starting up too slow.

You can confirm missing migrations by entering the runtime container and checking manually. Use the following commands:

  • docker exec -it application_grape-runtime_1 bash

  • source venv/bin/activate

  • cd code

  • ./manage.py showmigrations

if any migrations are not run, they won’t be checked like so

...
teams
 [X] 0001_initial
 [X] 0002_auto_20170627_0948
 [X] 0003_pre_populate_default_teams
 [X] 0004_pre_populate_default_teams_fixed
 [ ] 0005_ad_imports_blank  # <- not run
 [ ] 0006_team_crowd_imports  # <- not run
thumbnail
 [X] 0001_initial
trello
 [X] 0001_initial

If this is the case, you need to report that problem to Grape immediately.

Restart the runtime container with docker restart application_grape-runtime_1.

If this didn’t help check the PostgreSQL, Redis and Elasticsearch containers next.

If they are not running try restarting them with docker restart CONTAINERNAME.

Once the databases are up and running please restart the grape-runtime container next by running docker restart application_grape-runtime_1.

postgresql

If PostgreSQL failed with the error FATAL: the database system is starting up the reason is most likely an unclean shutdown of the database caused by a container or VM/Server crash.

To fix this run the fix_postgresql_grape.sh helper script located in /data/grape/data/setup/current/extra.

Grape is online

Problems using Grape (Error 500) - check Sentry

Check if you can see an error in Sentry at the time the problem happened. It helps the Grape team if you have an error in sentry that correlates to you problem.

More details on Sentry

Self-test page

You can open the self-test page in the browser for a quick overview:

https://your_grape_server/debug/self-test/

Note: only Grape admin users can access this page

Problems after upgrading Grape

My Grape instance just show the “We are busy updating our product” page (HTTP 502)

Please check the output of the runtime container:

docker logs application_grape-runtime_1 | tail -n 30

Grape (gunicorn process) uses ~100%

First try restarting the grape-application by running

cd /data/grape/config/application && docker-compose restart

and wait until it comes back up.

If this does not help:

Inspect what the process is doing:

  1. Find the process id of gunicorn E.g. with ps (as root on the host machine)

ps aux | grep gunicorn
  1. Inspect it using strace

strace -p PID_OF_GUNICORN

A bug appeared after upgrading to a new version

  1. Use your VM snapshot to recover the last working version.

  2. If you don’t have a VM snapshot:

    1. Take a VM snapshot!

    2. Look up the previous release in the changelog or your inbox.

    3. Downgrade to that version, e.g. onpremises-3.3.9

    grape install --custom-version onpremises-3.3.9
    

    YMMV with this approach. Always make a backup before an upgrade!

I want see what grape commands have been called in the past

The grape commands logs all it’s output to the file /data/grape/logs/grape_setup.log, you’ll find all the information about previous runs in there.