Tuesday, September 24, 2019

I started getting intermittent sets of error messages like this:

error: rpmdb: BDB0113 Thread/process 26154/140393252489024 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm

when deploying changes to a bunch of AWS EC2 instances.

The error messages are misleading as (in this case at least) the RPM database is not corrupted; the underlying issue was this:

[67897.740241] Out of memory: Kill process 28759 (yum) score 330 or sacrifice child
[67905.749492] yum invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

i.e. simply a lack of memory (the instances were just too small).

Posted at 11:12 PM |