| Age | Commit message (Collapse) | Author |
|---|
|
This release relicensing the software to GPLv3, add support for forwarding
logs to InfluxDB v2 and questdb [1].
[1] https://questdb.io.
Signed-off-by: Shulhan <ms@kilabit.info>
|
|
While at it, mention how to install it using pre-build Arch Linux
package through build.kilabit.info.
|
|
|
|
See https://kilabit.info/journal/2022/gpl/ for more information.
|
|
This changes convert the README to AsciiDoc format, add section about
configuring forwarder using questdb.
|
|
Questdb [1] is one the time-series database.
We experiment to forward the HTTP log using Influx Line Protocol (ILP).
[1]: https://questdb.io/
|
|
Instead of single forwarder, Influxd, the Config struct now can have
one or more forwarders.
The kind of forwarders is defined by it subsection name, for example
`[forwarder "influxd"]` defined a forwarder for influxd.
|
|
|
|
Later we will have multiple forwarders, not only influxd.
|
|
It is up to the user of haminer library (in this case the cmd/haminer)
on how to Start and Stop the process, not at the library level.
|
|
Halog contains parsed HTTP log, so its make more readable if we rename
the type name.
|
|
|
|
|
|
|
|
Since the influxdb v2, and since the flux query language introduced,
the way the log read and queried kinda changes.
Things that we can query on v1, is not possible (or maybe I forgotten)
anymore.
This changes move all haproxy log fields value that is not number (except
HTTP status code) to tags and left all numbers (like time, number of
connections, bytes read) in the fields.
|
|
Instead of using struct UDPPacket to read UDP packet from HAproxy log,
simplify it by using fixed, reusable size of []byte directly.
|
|
Previously, if no logs received after 15 seconds (or any interval user
defined in Config.ForwardInterval), the haminer process send an empty
halog to forwarders (empty halog is the one that use "-" for backend,
frontend, server name, and HTTP method).
This cause bogus traffic and need additional filter when analyzed.
|
|
This changes replace the "influxdb_api_write" with new section
`[forwarder "influxd"]`.
The section contains version, url, org, bucket, user, password, and
token.
The version field define the API version to be used when writing log
to Influxd.
|
|
Set the minimum Go to 1.18 and update the share modules.
When this program written, the ini library does not have the Unmarshal
function, so we load the configuration by reading each key and parse it
manually.
Now that we have Unmarshal function, the way we parse the configuration
is simplified so does the way configuration written.
This changes the accept_backend, capture_request_header written.
Instead of using comma to set multiple values, now it must be written
one key and one value on different line.
|
|
|
|
An administrator may need to monitor if the haminer service is running or
not through kapacitor. With heartbeat, we can tell if haminer is not
running if no data is send along, say, one hour.
|
|
|
|
|
|
Each endpoint (http_url) may have two or more HTTP methods on them.
|
|
|
|
Previously, we forward the logs only if total collected logs in buffer
is greater or equal to 10.
This commit change the model into using time interval, where the logs
will be send every N seconds (default to 15 seconds).
|
|
|
|
|
|
|
|
|
|
This is to make it consistent with separator on "accept_backend".
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
