From 554b784420dbb3c45c3795502985e6ca8d5ba224 Mon Sep 17 00:00:00 2001 From: Orien Vandenbergh Date: Fri, 17 Mar 2017 07:43:34 -0600 Subject: [PATCH] Add notes from some splunk virtual classes --- .../notes/SplunkDataAdministration/day1.txt | 58 +++++++++++++++ .../notes/SplunkDataAdministration/day2.txt | 70 ++++++++++++++++++ .../notes/SplunkDataAdministration/day3.txt | 33 +++++++++ .../notes/SplunkSystemAdministration/day1.txt | 74 +++++++++++++++++++ .../notes/SplunkSystemAdministration/day2.txt | 55 ++++++++++++++ 5 files changed, 290 insertions(+) create mode 100644 ghetto/notes/SplunkDataAdministration/day1.txt create mode 100644 ghetto/notes/SplunkDataAdministration/day2.txt create mode 100644 ghetto/notes/SplunkDataAdministration/day3.txt create mode 100644 ghetto/notes/SplunkSystemAdministration/day1.txt create mode 100644 ghetto/notes/SplunkSystemAdministration/day2.txt diff --git a/ghetto/notes/SplunkDataAdministration/day1.txt b/ghetto/notes/SplunkDataAdministration/day1.txt new file mode 100644 index 0000000..889ebc0 --- /dev/null +++ b/ghetto/notes/SplunkDataAdministration/day1.txt @@ -0,0 +1,58 @@ +Teacher: joanna@splunk.com + +My Intro: + I'm Orien (sounds like the constellation Orion), just started as a Splunk + Engineer at Defense Point Security. I've finished a couple Splunk classes + over the last month, nothing practical yet, Linux for 20+ years. Dogs. + +Goals: + Manage and deploy forwarders with management (Module 4&5, critically important) + configure common splunk data inputs + customize input parsing process + - Not covering creating splunk indexes + +Schedule 1-4 today, 4-7 tomorrow, 8-12 Friday + +Module 1: Introduction + Input > Parsing > Indexing > Searching + Primary Components: Forwarder, Indexer, Search Head + Additional: Heavy Forwarder, Deployment Server + + Splunk Data Administrator Role + data onboarding and management + work with users requesting new data, define events and fields for ingest + prioritize requests + document everything + design and manage inputs for UF/HF to capture data + manage parsing, line breaking, timestamp extraction + move from testing to production +Lab 1: + Path: /opt/splunk + +Module 2: Getting Data in - Staging + Input phase - broad strokes only + most configuration in input.conf + some configuration occurs in props.conf + Parsing phase - fine tuned tweaks + most configuration in props.conf + also uses transforms.conf + _thefishbucket contains file monitoring audit information + custom indexes control access, improve performance and control retention time for each index individually. + Index-Time precedence, local/default file processing under apps occurs in ascii sort order + splunk btool list + options: --debug --user= --app= + examples: splunk bool inputs list monitor:///var/log/secure.log --debug + --debug shows the config files that created the settings. + +Module 3: getting Data in - Productin + Universal Forwarder bandwidth limited to 256KBps by default + UF only forward to splunk instances, and only 1 at a time + HF can forward to other products, and more than 1 at a time + HF can be used as a mid stage forwarder for multi-tier forwarding setups. + HF no longer best practice + +Module 4: Deployment Server + Server classes have one or more apps + A server has one or more classes + so a server gets apps via the classes it belongs to + diff --git a/ghetto/notes/SplunkDataAdministration/day2.txt b/ghetto/notes/SplunkDataAdministration/day2.txt new file mode 100644 index 0000000..02335f7 --- /dev/null +++ b/ghetto/notes/SplunkDataAdministration/day2.txt @@ -0,0 +1,70 @@ +Module 5 - Monitor Inputs + Question: How does splunk handle file rotations if it happens during a restart? Data lost? + Answer: Directory Monitors do, File Monitors don't + slide 112 + splunk cmd btprobe -d SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file --reset + + tcp/udp default source name: : + + scripted input: + * $splunk_home/etc/apps//bin # This is the best place for it. + * $splunk_home/bin/scripts + * $splunk_home/etc/system/bin + test script: ./splunk cmd /script.sh # doesn't run script, just tests that splunk can access it. + scripted inputs can also buffer data, similar to the network collectors + Better to have cron run the script, and dump the data to a logfile. Make splunk monitor the logfile instead + + Module 7: windows & Agentless + Windows + input types: admon perfmon WinEventLog WinHostMon WinPrintMon WinRegMon + Warning from fellow student: + Just throwing this out there. If you monitor the registry in a way that + causes the Universal Forwarders to send you their entire registry you + are likely to clog WAN links. I saw a 16 Gbps WAN link go down because + of this went thousands of Windows systems were sending over their + registry. + + [WinEventLog://Security] + whitelist1= "Stuff" + whitelist2= "Other stuff" + blacklist + Maximum of 10 whitelists and blacklists per universal forwarder stanza + Can do WMI remote inputs, not recommended for environments bigger than small, scales poorly, requires an AD account + Special field extractions + IIS: frequently reconfigured on the fly by admins. OBvs this is a problem. + Use indexed field extraction on the windows forwarder to correct this. + Ensure that the header is in the same place and never moves. Then the forwarder can use that header to pre-parse the data. + Powershell input, otherwise teh same as the scripted input, still better to have windows schedule it instead + Agentless + Splunk App for Stream + essentially a packet capture agent + monitors the network and collects it's data there, then sends it into splunk + HTTP Event Collector + Splunk listens for http inputs, clients send their data to the http listener. + Distributed HEC (HTTP Event Collector) Deployment Options + Can scale because every splunk system can act as a collector to receive data from a load balancer + Disabled by default Settings > Data Inputs > HTTP Inputs + Create a token, then define metadata for the stream + Data can be transmitted as JSON + Can send acks, but requires additional handshaking for the response channel + Multi-event JSON posts are possible, but in non-standard format: { stuff }{ stuff 2 }{ stuff 3} rather than standard [{}{}{}] + My Token: 3372606C-6D24-48A4-A28D-09C616A277E7 + + Module 8: Fine-Tuning Inputs + props.conf is very important + inputs phase: + character encoding (default is utf8) + fine tuned source types + can override the defaults on a per file basis + + parsing: + event breaks + time extraction + event transformation + + Module 9: Parsing Phase and Data Preview + props.conf.spec - LINE_BREAKER is best way to split lines, ProServ recommended + Take extra time to ensure timestamps are correct + Either TZ in timestamp, or specified in props.conf or tz of indexer + + diff --git a/ghetto/notes/SplunkDataAdministration/day3.txt b/ghetto/notes/SplunkDataAdministration/day3.txt new file mode 100644 index 0000000..bbc32ce --- /dev/null +++ b/ghetto/notes/SplunkDataAdministration/day3.txt @@ -0,0 +1,33 @@ +Module 10: + Modifying raw data before it's indexed + use per event source types only in a last chance scenario, everything else is better + to set metadata in transforms.conf + SOURCE_KEY = _raw + REGEX = server:(\w+) + DEST_KEY = MetaData:Host + FORMAT = host::$1 + Host => host::, + + To change the index at index-time (note the additional underscore here) + REGEX = (Error|Warning) + DEST_KEY = _MetaData:Index + FORMAT = itops + + Filter Events + FORMAT = nullQueue + + http:///debug/refresh - forces splunk to refresh it's config(?) + at a minimum it does the inputs configurations, definitely doesn't do the indexer + + + I need to go over modules 10 and 11. Missed too much i fear + +Module 12: Diag + Creates diagnostic package for shipment to experts. + ./splunk diag + Create and index a diag + + +Course Review: + Mod 1 - + joanna@splunk.com diff --git a/ghetto/notes/SplunkSystemAdministration/day1.txt b/ghetto/notes/SplunkSystemAdministration/day1.txt new file mode 100644 index 0000000..4d8d585 --- /dev/null +++ b/ghetto/notes/SplunkSystemAdministration/day1.txt @@ -0,0 +1,74 @@ +Mitch Fleischman +mitchf@splunk.com + +studentid: 06 +ipaddress: 52.53.200.165 10.0.0.206 +ssh username: btv_splunker06 + +set servername and hostname to splunk06 +also set sessiontimeout, to something helpful for class + +Modules 1-6.5 today, 6.5-11 tomorrow + +When do you add more indexers? + Partly based on how much searching, but add a new indexer every 100 = 250GB daily + with Enterprise Security, you'll trend closer to the lower number (aka more indexers) + +Search Heads? + 8-12 users per search head + user might mean scheduled searches, etc + +hardware + 12G ram + indexer: 12@2Ghz 800iops + search: 16@2Ghz 2x10k SAS RAID1 + +splunk kv store is mongodb + +Linux OS tuning: pg 20 + ulimit -c 1073741824 + ulimit -n 48 x default + ulimit -u 12 x default + + disable THP + + change root password, insert sha256 checksum (I believe) into $SPLUNK_HOME/etc/passwd to change admin password + + ./splunk enable boot-start -user + + +Windows + Autostarts automatically + +$SPLUNK_DB = $SPLUNK_HOME/var/lib/splunk + +Licensing: + 3 warnings for free splunk, 5 for paid + 30 day rolling window + +Module 3: Installing Apps + App is collection of files (inputs, indexes, sourcetypes, extractions, transformations), (eventtypes, tags, reports, dashboards, other KOs), (Scripts, web assets) + Addon is an App subset (like the bits needed to make a forwarder work) + Remove an app: + splunk remove app + Permissions: + read - to see and interact with it + write - to add delete modify the KO in the app + Default is read only + +Module 4: Configuration files + */default - comes with splunk + */local - user overrides + .meta files determine how global a configuration file setting is. + app/metadata/local.meta + [tags/action%3Daddtocart/browser] + access = read : [ * ] + export = (none|system) + owner + version + modtime + + splunk btool check + splunk btool (inputs|) list (|monitor:///var/log{, --debug}) # debug shows which file the line came from + splunk btool tags list (list all tags configured) --debug (also show the file they came from) + splunk btool tags list --debug --app=search --user= diff --git a/ghetto/notes/SplunkSystemAdministration/day2.txt b/ghetto/notes/SplunkSystemAdministration/day2.txt new file mode 100644 index 0000000..8cce69b --- /dev/null +++ b/ghetto/notes/SplunkSystemAdministration/day2.txt @@ -0,0 +1,55 @@ +Module 5 - Buckets and indexes (lots of data from DataAdmin class) + Show data utilization for an index with details + |dbinspect index= + Can set the search default window per app + ui-prefs.conf + [search] + dispatch.earliest_time = -24h@h + dispatch.latest_time = now + +Module 6 - Splunk Index Management + Recommend + rolling hot buckets daily, + maxHotBuckets - limit of 10 hot buckets for a high volume index (default 3) + frozenTimePeriodInSecs - how long to wait before freezing buckets + index.conf + [volume:fast] + path = <> + maxVolumeDataSizeMB = + + [soc] + homePath = volume:fast/soc/db # homePath is hot and warm buckets + homePath.maxDataSizeMB = + coldPath # Same thing for cold + + Backups + $SPLUNK_HOME/var/lib/splunk // indexes + $SPLUNK_HOME/etc // configs + + Hot buckets cannot be backed up without stopping splunk, or using snapshots + Alternatively, forhigh volume, multiple daily incremental backups to grab data frequently + Moving an index + stop splunk + then move the directories + then update indexes.conf to point at the new locations + if a global move, update SPLUNK_DB environment variable + Removing data + wait for expiration + delete command marks as deleted, doesn't free space need to the the special can_delete role for that + Search> search for some records | delete + > splunk clean [eventdata|userdata|all] [-index name] + Actually removes the data from the index entirely, frees space + if no index is provided, deletes all the data! + Restoring Data from frozen + only raw data is frozen, no indexes + copy archive directory into the index specific thaweddb directory + The rebuild the index for that data, doesn't recharge for licensing + > splunk rebuild + + Index replication + +Module 8: Authentication Integration + LDAP, PAM, RADIUS, AD, etc + + +