Re: Prometheus not starting

Hi,

Alright, finally figured it out, sort of. I found a
file /var/run/rc.d/prometheus with some old settings. When looking at
the rcctl output, I noticed:

> doing _rc_parse_conf /var/run/rc.d/prometheus

So apparently the settings in /var/run/rc.d were overriding the rest. I
read the man page for rc.d and it says about /var/run/rc.d: "Directory
containing files recording the variables of currently running daemons."

I don't know why that file was there. Some failed stop attempt or I had
just killed prometheus without stopping it? Anyway, after removal it
finally picked up all the flags I was setting.

Fabian


On Sun, 1 Mar 2020 23:18:28 +0200 Fabian wrote:

> Am Sun, 01 Mar 2020 13:44:16 +0000
> schrieb Stuart Henderson <stu@spacehopper.org>:
>
> > Why have you set 'prometheus_flags=""'? Just omit it if you want
> > the default flags.
>
> Because "man rc.conf.local" says:
>
> > If set to the empty string, the daemon is run with the default
> > command line arguments defined in its rc.d(8) daemon script, or
> > without command line arguments if no such default exists.
>
> But good point nevertheless. Removed prometheus_flags from
> rc.conf.local entirely, ran "rcctl start prometheus" and still:
>
> ...
> prometheus_flags empty, using default
> >--config.file /etc/prometheus/prometheus.yml --storage.tsdb.path
> >'/var/prometheus'< ...
> prometheus: level=error ts=2020-03-01T21:10:47.505Z
> caller=main.go:717 err="opening storage failed: mkdir data/:
> permission denied" ...
>
> Fabian
>
>
> > On 1 March 2020 01:29:22 Fabian <openbsd@mailfri.com> wrote:
> >
> > > Hi,
> > >
> > > I have been trying to run the current Prometheus port version
> > > 2.9.2 on OpenBSD 6.6 and I just cannot figure out why Prometheus
> > > fails to start. My rc.conf.local looks like this:
> > >
> > > node_exporter_flags="--web.listen-address=localhost:9100"
> > > prometheus_flags=""
> > > pkg_scripts=node_exporter prometheus
> > >
> > > That works nicely for node_exporter but prometheus is giving me
> > > this:
> > >
> > > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > > caller=main.go:285 msg="no time or size retention was set so using
> > > the default time retention" duration=15d
> > > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > > caller=main.go:321 msg="Starting Prometheus"
> > > version="(version=2.9.2, branch=non-git, revision=non-git)"
> > > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > > caller=main.go:322 build_context="(go=go1.13.1,
> > > user=_pbuild@amd64.ports.openbsd.org, date=20191012-03:10:07)"
> > > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > > caller=main.go:323 host_details=(openbsd)
> > > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > > caller=main.go:324 fd_limits="(soft=128, hard=1024)"
> > > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > > caller=main.go:325 vm_limits="(soft=34359738368b,
> > > hard=34359738368b)" prometheus: level=info
> > > ts=2020-03-01T00:04:05.108Z caller=main.go:640 msg="Starting
> > > TSDB ..." prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > > caller=main.go:509 msg="Stopping scrape discovery manager..."
> > > prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > > caller=main.go:523 msg="Stopping notify discovery manager..."
> > > prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > > caller=main.go:545 msg="Stopping scrape manager..."
> > > prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > > caller=main.go:519 msg="Notify discovery manager stopped"
> > > prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > > caller=web.go:416 component=web msg="Start listening for
> > > connections" address=localhost:9090 prometheus: level=info
> > > ts=2020-03-01T00:04:05.111Z caller=main.go:505 msg="Scrape
> > > discovery manager stopped" prometheus: level=info
> > > ts=2020-03-01T00:04:05.112Z caller=manager.go:743 component="rule
> > > manager" msg="Stopping rule manager..." prometheus: level=info
> > > ts=2020-03-01T00:04:05.112Z caller=manager.go:749 component="rule
> > > manager" msg="Rule manager stopped" prometheus: level=info
> > > ts=2020-03-01T00:04:05.112Z caller=notifier.go:527
> > > component=notifier msg="Stopping notification manager..."
> > > prometheus: level=info ts=2020-03-01T00:04:05.112Z
> > > caller=main.go:708 msg="Notifier manager stopped" prometheus:
> > > level=info ts=2020-03-01T00:04:05.112Z caller=main.go:539
> > > msg="Scrape manager stopped" prometheus: level=error
> > > ts=2020-03-01T00:04:05.112Z caller=main.go:717 err="opening
> > > storage failed: mkdir data/: permission denied"
> > >
> > >
> > > Apparently it is reading its TSDB from the built-in default
> > > "data/" instead of what the port packager set in the rc.d script:
> > >
> > > # cat rc.d/prometheus
> > > #!/bin/sh
> > > #
> > > # $OpenBSD: prometheus.rc,v 1.1.1.1 2018/01/10 16:26:19 claudio
> > > Exp $
> > >
> > > daemon="/usr/local/bin/prometheus"
> > > daemon_flags="--config.file /etc/prometheus/prometheus.yml"
> > > daemon_flags="${daemon_flags} --storage.tsdb.path
> > > '/var/prometheus'" daemon_user=_prometheus
> > >
> > > . /etc/rc.d/rc.subr
> > >
> > > pexp="${daemon}.*"
> > > rc_bg=YES
> > > rc_reload=NO
> > >
> > > rc_start() {
> > > ${rcexec} "${daemon} ${daemon_flags} < /dev/null 2>&1 | \
> > > logger -p daemon.info -t prometheus"
> > > }
> > >
> > > rc_cmd $1
> > >
> > >
> > > Strangely, it seems to me like the config.file option is being
> > > picked up, otherwise it would bail out even earlier. rcctl seems
> > > to recognize the flags:
> > >
> > > # rcctl -d start prometheus
> > > doing _rc_parse_conf
> > > doing _rc_quirks
> > > prometheus_flags empty, using default >--config.file
> > > /etc/prometheus/prometheus.yml --storage.tsdb.path
> > > '/var/prometheus'< doing _rc_parse_conf /var/run/rc.d/prometheus
> > > doing _rc_quirks
> > > doing rc_check
> > > prometheus
> > > doing rc_start
> > > doing _rc_wait start
> > > No home directory /nonexistent!
> > > Logging in with home = "/".
> > > doing rc_check
> > > doing _rc_write_runfile
> > > (ok)
> > >
> > >
> > > When I run prometheus manually like this, it starts fine:
> > >
> > > su -l -s /bin/sh _prometheus -c 'prometheus
> > > --config.file=/etc/prometheus/prometheus.yml
> > > --storage.tsdb.path=/var/prometheus'
> > >
> > >
> > > I am running out of ideas what to do here. It looks like an issue
> > > with the rc.d script?
> > >
> > > Fabian