Penguin, small TECH.BARWICK.DE
Start
 

Recent posts

Categories

Archive

Syndication

 



Powered By

Info

Tuesday, September 24, 2019  11:12 PM

Yum and "Thread died in Berkeley DB library" errors

I started getting intermittent sets of error messages like this:

error: rpmdb: BDB0113 Thread/process 26154/140393252489024 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm

when deploying changes to a bunch of AWS EC2 instances.

The error messages are misleading as (in this case at least) the RPM database is not corrupted; the underlying issue was this:


[67897.740241] Out of memory: Kill process 28759 (yum) score 330 or sacrifice child
[67905.749492] yum invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

i.e. simply a lack of memory (the instances were just too small).


Posted in Linux | add a comment

The Prometheus PostgreSQL storage adapter does not seem amenable to being executed directly from a systemd service file.

As a workaround I created a wrapper script like this (adjust parameters as required):

#!/bin/bash

# Wrapper to launch prometheus-postgresql-adapter, as calling
# it directly from the systemd service file doesn't seem to work.
#
# Disclaimer: there is probably a better way of doing this.

nohup /usr/local/bin/prometheus-postgresql-adapter \
-pg-host=... \
-pg-port=... \
-pg-database="..." \
-pg-user="..." \
>> /var/log/prometheus-postgresql-storage-adapter/prometheus-pg-adapter.log 2>&1

and a service file like this:

[Unit]
Description=Prometheus PostgreSQL Storage Adapter
Documentation=https://github.com/timescale/prometheus-postgresql-adapter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus-postgresql-adapter-wrapper
Restart=on-failure

[Install]
WantedBy=multi-user.target

which works fine (YMMV of course).

There may of course be a more elegant way of solving this issue, if so feel free to share.


Posted in Linux | add a comment