This is part of the Semicolon&Sons Code Diary - consisting of lessons learned on the job. You're in the dumb-mistakes-and-gotchas category.
Last Updated: 2024-04-26
Check the max tries setting for any queue software before putting it into
production. With Laravel, its default was infinity. When we had a buggy job,
the maximum retries eventually caused 2nd order errors in the (database backed)
queue system, such as SQL errors like "number too big" for number_retries
column.
I could imagine other situations where infinite retries could have consumed costly external APIs, or - worse - bombarded someone with emails.
Always set converative max number of retries for jobs on a queue.