Re: Prometheus not starting

Am Sun, 01 Mar 2020 13:44:16 +0000
schrieb Stuart Henderson <stu@spacehopper.org>:

> Why have you set 'prometheus_flags=""'? Just omit it if you want the
> default flags.

Because "man rc.conf.local" says:

> If set to the empty string, the daemon is run with the default
> command line arguments defined in its rc.d(8) daemon script, or
> without command line arguments if no such default exists.

But good point nevertheless. Removed prometheus_flags from
rc.conf.local entirely, ran "rcctl start prometheus" and still:

...
prometheus_flags empty, using default >--config.file /etc/prometheus/prometheus.yml --storage.tsdb.path '/var/prometheus'<
...
prometheus: level=error ts=2020-03-01T21:10:47.505Z caller=main.go:717 err="opening storage failed: mkdir data/: permission denied"
...

Fabian


> On 1 March 2020 01:29:22 Fabian <openbsd@mailfri.com> wrote:
>
> > Hi,
> >
> > I have been trying to run the current Prometheus port version 2.9.2
> > on OpenBSD 6.6 and I just cannot figure out why Prometheus fails to
> > start. My rc.conf.local looks like this:
> >
> > node_exporter_flags="--web.listen-address=localhost:9100"
> > prometheus_flags=""
> > pkg_scripts=node_exporter prometheus
> >
> > That works nicely for node_exporter but prometheus is giving me
> > this:
> >
> > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > caller=main.go:285 msg="no time or size retention was set so using
> > the default time retention" duration=15d
> > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > caller=main.go:321 msg="Starting Prometheus"
> > version="(version=2.9.2, branch=non-git, revision=non-git)"
> > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > caller=main.go:322 build_context="(go=go1.13.1,
> > user=_pbuild@amd64.ports.openbsd.org, date=20191012-03:10:07)"
> > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > caller=main.go:323 host_details=(openbsd)
> > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > caller=main.go:324 fd_limits="(soft=128, hard=1024)"
> > prometheus: level=info ts=2020-03-01T00:04:05.104Z
> > caller=main.go:325 vm_limits="(soft=34359738368b,
> > hard=34359738368b)" prometheus: level=info
> > ts=2020-03-01T00:04:05.108Z caller=main.go:640 msg="Starting
> > TSDB ..." prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > caller=main.go:509 msg="Stopping scrape discovery manager..."
> > prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > caller=main.go:523 msg="Stopping notify discovery manager..."
> > prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > caller=main.go:545 msg="Stopping scrape manager..."
> > prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > caller=main.go:519 msg="Notify discovery manager stopped"
> > prometheus: level=info ts=2020-03-01T00:04:05.109Z
> > caller=web.go:416 component=web msg="Start listening for
> > connections" address=localhost:9090 prometheus: level=info
> > ts=2020-03-01T00:04:05.111Z caller=main.go:505 msg="Scrape
> > discovery manager stopped" prometheus: level=info
> > ts=2020-03-01T00:04:05.112Z caller=manager.go:743 component="rule
> > manager" msg="Stopping rule manager..." prometheus: level=info
> > ts=2020-03-01T00:04:05.112Z caller=manager.go:749 component="rule
> > manager" msg="Rule manager stopped" prometheus: level=info
> > ts=2020-03-01T00:04:05.112Z caller=notifier.go:527
> > component=notifier msg="Stopping notification manager..."
> > prometheus: level=info ts=2020-03-01T00:04:05.112Z
> > caller=main.go:708 msg="Notifier manager stopped" prometheus:
> > level=info ts=2020-03-01T00:04:05.112Z caller=main.go:539
> > msg="Scrape manager stopped" prometheus: level=error
> > ts=2020-03-01T00:04:05.112Z caller=main.go:717 err="opening storage
> > failed: mkdir data/: permission denied"
> >
> >
> > Apparently it is reading its TSDB from the built-in default "data/"
> > instead of what the port packager set in the rc.d script:
> >
> > # cat rc.d/prometheus
> > #!/bin/sh
> > #
> > # $OpenBSD: prometheus.rc,v 1.1.1.1 2018/01/10 16:26:19 claudio Exp
> > $
> >
> > daemon="/usr/local/bin/prometheus"
> > daemon_flags="--config.file /etc/prometheus/prometheus.yml"
> > daemon_flags="${daemon_flags} --storage.tsdb.path '/var/prometheus'"
> > daemon_user=_prometheus
> >
> > . /etc/rc.d/rc.subr
> >
> > pexp="${daemon}.*"
> > rc_bg=YES
> > rc_reload=NO
> >
> > rc_start() {
> > ${rcexec} "${daemon} ${daemon_flags} < /dev/null 2>&1 | \
> > logger -p daemon.info -t prometheus"
> > }
> >
> > rc_cmd $1
> >
> >
> > Strangely, it seems to me like the config.file option is being
> > picked up, otherwise it would bail out even earlier. rcctl seems to
> > recognize the flags:
> >
> > # rcctl -d start prometheus
> > doing _rc_parse_conf
> > doing _rc_quirks
> > prometheus_flags empty, using default >--config.file
> > /etc/prometheus/prometheus.yml --storage.tsdb.path
> > '/var/prometheus'< doing _rc_parse_conf /var/run/rc.d/prometheus
> > doing _rc_quirks
> > doing rc_check
> > prometheus
> > doing rc_start
> > doing _rc_wait start
> > No home directory /nonexistent!
> > Logging in with home = "/".
> > doing rc_check
> > doing _rc_write_runfile
> > (ok)
> >
> >
> > When I run prometheus manually like this, it starts fine:
> >
> > su -l -s /bin/sh _prometheus -c 'prometheus
> > --config.file=/etc/prometheus/prometheus.yml
> > --storage.tsdb.path=/var/prometheus'
> >
> >
> > I am running out of ideas what to do here. It looks like an issue
> > with the rc.d script?
> >
> > Fabian
>
>
>