elasticsearch - How to extract variables from log file path, test log file name for pattern in Logstash? -

-

i have aws elasticbeanstalk instance logs on s3 bucket.

path logs is:

resources/environments/logs/publish/e-3ykfgdfgmp8/i-cf216955/_var_log_nginx_rotated_access.log1417633261.gz

which translates :

resources/environments/logs/publish/e-[random environment id]/i-[random instance id]/

the path contains multiple logs:

_var_log_eb-docker_containers_eb-current-app_rotated_application.log1417586461.gz _var_log_eb-docker_containers_eb-current-app_rotated_application.log1417597261.gz _var_log_rotated_docker1417579261.gz _var_log_rotated_docker1417582862.gz _var_log_rotated_docker-events.log1417579261.gz _var_log_nginx_rotated_access.log1417633261.gz

notice there's random number (timestamp?) inserted aws in filename before ".gz"

problem need set variables depending on log file name.

here's configuration:

input {         s3 {                 debug => "true"                 bucket => "elasticbeanstalk-us-east-1-something"                 region => "us-east-1"                 region_endpoint => "us-east-1"                 credentials => ["..."]                 prefix => "resources/environments/logs/publish/"                 sincedb_path => "/tmp/s3.sincedb"                 backup_to_dir => "/tmp/logstashed/"                 tags => ["s3","elastic_beanstalk"]                 type => "elastic_beanstalk"         } }  filter {  if [type] == "elastic_beanstalk" {   grok {     match => [ "@source_path", "resources/environments/logs/publish/%{environment}/%{instance}/%{file}<unnecessary_number>.gz" ]   }  } }

in case want extract environment , instance , file name path. in file name need ignore random number. doing right way? full, correct solution this?

another question how can specify fields custom log format particular log file above?

this like: (meta-code)

filter {      if [type] == "elastic_beanstalk" {        if [file_name] begins "application_custom_log" {          grok {              match => [ "message", "%{ip:client} %{word:method} %{uripathparam:request} %{number:bytes} %{number:duration}" ]           }        }         if [file_name] begins "some_other_custom_log" {         ....        }      }     }

how test file name pattern?

for first question, , assuming @source_path contains full path, try:

match => [ "@source_path", "logs/publish/%{notspace:env}/%{notspace:instance}/%{notspace:file}%{number}%{notspace:suffix}" ]

this create 4 logstash field you:

  • env
  • instance
  • file
  • suffix

more information available on grok man page , should test grok debugger.

to test fields in logstash, use conditionals, e.g.

if [field] == "value" if [field] =~ /regexp/

etc.

note it's not necessary grok. can have multiple 'match' arguments, , (by default) stop after hitting first 1 matches. if patterns exclusive, should work you.


Comments

Post a Comment

Popular posts from this blog

javascript - How to synchronize the Three.js and HTML/SVG coordinate systems (especially w.r.t. the y-axis)? -

-
i'm combining 3d content three.js html , svg content. three.js cssloader pretty job synchronizing placement of html , svg content in 3d world. but svg/html coordinate systems 'left-handed', whereas three.js coordinate system 'right-handed'. means y-axes reversed. in svg/html, y / top goes go down screen, , three.js uses more standard mathematical convention of y going down go down screen. i have continually convert 1 other, pretty error prone. know not first run (for example, look here ). has come general solution? here's tried: do in object3d .scale.y = -1 . may suspect, turns out disaster. turns inside-out, , don't try put camera in there. do in object3d .rotate.x = math.pi . more promising, z axis no longer consistent html concept of z-index. still, i'm using now. in html, don't use top , use bottom . in svg , inside <g transform="scale(1, -1)"> inside <g transform="translate(0, imageheight)"> . h...
Read more

javascript - How do I find how many occurences are there of a highlighted string, and which occurence is it? -

-
as title says, have website text inside div, there way highlight part of , know occurence is? if text has 4 occurences of word "something", , highlight 3rd, how information using javascript/jquery? http://jsfiddle.net/g09g35xa/6/ here's method seems work. highlight string want , when click button alert index of highlighted word in div. markup: <div id="thediv"> <p>the quick brown quick fox jumps on lazy dog</p> </div> <br /> <input id="findindex" type="button" value="find index of highlighted string" /> and js: function getselectedhtml() { try { if (window.activexobject) { var c = document.selection.createrange(); return c.htmltext; } return getselection().getrangeat(0).tostring(); } catch (e) { if (window.activexobject) { return document.selection.createrange(); } else { return ...
Read more

java - Reading data from multiple zip files and combining them to one -

-
i want read data lets 4 zip files called zip1, zip2, zip3, zip4. of these zip files split 1 big zip file called "bigzip". want combine zip files 1 , compare bytes if 1 bigzip file matches size of bytes combined zip file of (zip1+zip2+zip3+zip4). getting small file size when combine size of 4 zip files. doing wrong? here code same: targetfilepath1, targetfilepath2, targetfilepath3, targetfilepath4 belongs path of 4 zip files. sourcefilepath path bigzip file class test { public static void main(string args[]) { zipoutputstream outstream = new zipoutputstream(new fileoutputstream(sourcebigzip)); readzip(sourcefilepath, targetfilepath1); readzip(sourcefilepath, targetfilepath2); readzip(sourcefilepath, targetfilepath3); readzip(sourcefilepath, targetfilepath4); outstream.close(); } static void readzip(string sourcebigzip, string targetfile) throws exception { zipinputstream instream = new zipinputstream(new file...
Read more