SSL errors after upgrading to qcluster version 1.1.0

[marcelolima]

Hi,

Is anyone else having SSL Errors when using the new version (1.1.0)? I tried upgrading it in production, but started to get "django.db.utils.OperationalError: SSL error: decryption failed or bad record mac" whenever django ORM's performs a query from within a django-q task (traceback below).

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django_q/cluster.py", line 379, in worker
    res = f(*task['args'], **task['kwargs'])
  File "/home/docker/src/tasks.py", line 8, in wake_up_driver_app
    for company in Company.objects.all():
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 274, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 446, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: SSL error: decryption failed or bad record mac

Configuration:

• broker: AWS SQS
• database: AWS RDS Postgres 11.5
• Django version: 2.2.10
• Qcluster version: 1.1.0

I found an old similar issue: #79 but it seems to have been solved.

Does anyone have a clue on how to investigate this issue? For now I'm keeping the previous version (1.0.2) that doesn't have those issues but I need some of the fixes that are part of 1.1.0 release.

Thanks in advance!

[yyken]

I had the same issue
psycopg2.OperationalError: SSL error: decryption failed or bad record mac

• broker: AWS Redis
• database: AWS RDS Postgres 11.5
• Django version: 3.0.4
• Qcluster version: 1.2.1

[KonstantinSchubert]

A similar issue was fixed here: ckan/ckan#3247

[KonstantinSchubert]

Could it be possible that the issue was introduced here: 66588a2 ?

@Koed00 Note that db.close_old_connections() and db.connections.close_all() and db.connection.close() are not exactly the same thing. Was this an intentional change in the linked PR?

db.close_old_connections() will only close those connections that have errors or are beyond their maximum age, it calls this method internally: https://github.com/django/django/blob/master/django/db/backends/base/base.py#L502

[KonstantinSchubert]

This here may be a potential fix: https://github.com/sennder/django-q/commit/b951a19e8ce6fdad47199e7c406b1581434df90f

Unfortunately, i am currently unable to reproduce the original bug, and thus unable to verify that this really fixes it. I'm mostly guessing here.

@yyken If you want, you could try to see if this fixes it for you.

[cveilleux]

I am having the same issue with postgresql.

I have applied sennder@b951a19 on top of master and will report back if it helps.

[aaronn]

Having the same issue here as well. @cveilleux did your fix work for you?

[cveilleux]

Yes, the fix works. I just checked the logs, there has been zero SSL error in failed tasks since I applied the fix.

Before the fix, there would be dozens of these SSL errors every day. All gone now.

FYI I am running my own fork with b951a19 cherry-picked on top of a more recent version of django-q.

requirements.txt:

-e git://github.com/cveilleux/django-q.git@b11b663272bf0cc41faf2e27f4f886ee683ed21c#egg=django_q

[edmenendez]

This also worked for me.

[stackbomb]

@Koed00 hey could you merge this fix to the master?
I keep getting SSL error: decryption failed or bad record mac even when I use the conn_max_age = 0

[Koed00]

I'm currently on holiday, but if someone can find or create a PR for this that passes tests and coverage, then I'm more than happy to give it a quick merge.

[stackbomb]

Amazing, I will try to create a PR today. Enjoy your holiday dude!

[nittolese]

I was getting the same errors as OP, but thanks to the fix of @cveilleux and @KonstantinSchubert I have create a PR for @Koed00