Thursday, May 31, 2018

Logging to Elasticsearch with syslog-ng

Hi Misc,

I am revisiting the idea of storing log files in Elasticsearch DB for
quick search, analytics, and visualization (Kibana). I would like to
keep my current OpenBSD syslog-ng centralized logging server and just
write logs into ElasticsearchDB instead of flat files. Looks like
Elastricsearch runs happily on OpenBSD

http://openports.se/textproc/elasticsearch

just like Kibana

http://openports.se/www/kibana

I was wondering if the syslog-ng version in ports 3.12.1 (the latest
release seems to be 3.15.1) supports Java plugin needed to send logs
from syslog-ng to Elasticsearch. It looks like 3.12.1 is high enough
version which supports syslog-ng-incubator which was not the case last
time

https://marc.info/?l=openbsd-misc&m=143249546020820&w=2

However I don't see incubator in ports

https://github.com/balabit/syslog-ng-incubator

To be frank by looking quickly through incubator GitHub pages it is not
even clear to me that Java module currently necessary to send things to
Elasticsearch is even the part of the incubator. I stumbled somewhere on
Balabit official documentation which recommends Linux (binary blob
plugins) as the syslog-ng server OS for that very reason.

I do see that Balabit is contemplating writing a native Elasticsearch
destination driver per Google Summer of Code

https://github.com/balabit/syslog-ng/wiki/GSoC-2018-Proposal-:-ElasticSearch-destination:-native(C)-REST-API

Can anybody who is more informed than I on the topic shed some light
onto this topic?

Best,
Predrag

No comments:

Post a Comment