Add notes from some splunk virtual classes

ghetto/notes/SplunkDataAdministration/day1.txt

@@ -0,0 +1,58 @@
+Teacher: joanna@splunk.com
+
+My Intro:
+  I'm Orien (sounds like the constellation Orion), just started as a Splunk
+  Engineer at Defense Point Security. I've finished a couple Splunk classes
+  over the last month, nothing practical yet, Linux for 20+ years. Dogs.
+
+Goals:
+  Manage and deploy forwarders with management (Module 4&5, critically important)
+  configure common splunk data inputs
+  customize input parsing process
+  - Not covering creating splunk indexes
+
+Schedule 1-4 today, 4-7 tomorrow, 8-12 Friday
+
+Module 1: Introduction
+  Input > Parsing > Indexing > Searching
+  Primary Components: Forwarder, Indexer, Search Head
+  Additional: Heavy Forwarder, Deployment Server
+  
+  Splunk Data Administrator Role
+    data onboarding and management
+    work with users requesting new data, define events and fields for ingest
+    prioritize requests
+    document everything
+    design and manage inputs for UF/HF to capture data
+    manage parsing, line breaking, timestamp extraction
+    move from testing to production
+Lab 1:
+  Path: /opt/splunk
+
+Module 2: Getting Data in - Staging
+  Input phase - broad strokes only
+    most configuration in input.conf
+    some configuration occurs in props.conf 
+  Parsing phase - fine tuned tweaks
+    most configuration in props.conf
+    also uses transforms.conf
+  _thefishbucket contains file monitoring audit information
+  custom indexes control access, improve performance and control retention time for each index individually.
+  Index-Time precedence, local/default file processing under apps occurs in ascii sort order
+  splunk btool <conf-name> list <options>
+    options: --debug --user=<user> --app=<app>
+    examples: splunk bool inputs list monitor:///var/log/secure.log --debug
+      --debug shows the config files that created the settings.
+
+Module 3: getting Data in - Productin
+  Universal Forwarder bandwidth limited to 256KBps by default
+  UF only forward to splunk instances, and only 1 at a time
+  HF can forward to other products, and more than 1 at a time
+  HF can be used as a mid stage forwarder for multi-tier forwarding setups.
+  HF no longer best practice
+
+Module 4: Deployment Server
+  Server classes have one or more apps
+  A server has one or more classes
+  so a server gets apps via the classes it belongs to
+

ghetto/notes/SplunkDataAdministration/day2.txt

@@ -0,0 +1,70 @@
+Module 5 - Monitor Inputs
+  Question: How does splunk handle file rotations if it happens during a restart? Data lost?
+  Answer: Directory Monitors do, File Monitors don't
+  slide 112
+  splunk cmd btprobe -d SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file <source> --reset
+
+  tcp/udp default source name: <host>:<port>
+
+  scripted input:
+    * $splunk_home/etc/apps/<app_name>/bin # This is the best place for it.
+    * $splunk_home/bin/scripts
+    * $splunk_home/etc/system/bin
+  test script: ./splunk cmd <path>/script.sh # doesn't run script, just tests that splunk can access it.
+  scripted inputs can also buffer data, similar to the network collectors
+  Better to have cron run the script, and dump the data to a logfile.  Make splunk monitor the logfile instead
+
+  Module 7: windows & Agentless
+    Windows
+      input types: admon perfmon WinEventLog WinHostMon WinPrintMon WinRegMon
+      Warning from fellow student:
+          Just throwing this out there. If you monitor the registry in a way that
+          causes the Universal Forwarders to send you their entire registry you
+          are likely to clog WAN links. I saw a 16 Gbps WAN link go down because
+          of this went thousands of Windows systems were sending over their
+          registry.
+
+      [WinEventLog://Security]
+        whitelist1= "Stuff"
+        whitelist2= "Other stuff"
+        blacklist
+      Maximum of 10 whitelists and blacklists per universal forwarder stanza
+      Can do WMI remote inputs, not recommended for environments bigger than small, scales poorly, requires an AD account
+      Special field extractions
+        IIS: frequently reconfigured on the fly by admins.  OBvs this is a problem.
+        Use indexed field extraction on the windows forwarder to correct this.
+        Ensure that the header is in the same place and never moves.  Then the forwarder can use that header to pre-parse the data.
+        Powershell input, otherwise teh same as the scripted input, still better to have windows schedule it instead
+    Agentless
+      Splunk App for Stream
+        essentially a packet capture agent
+        monitors the network and collects it's data there, then sends it into splunk
+      HTTP Event Collector
+        Splunk listens for http inputs, clients send their data to the http listener.
+    Distributed HEC (HTTP Event Collector) Deployment Options
+      Can scale because every splunk system can act as a collector to receive data from a load balancer
+      Disabled by default Settings > Data Inputs > HTTP Inputs
+      Create a token, then define metadata for the stream
+      Data can be transmitted as JSON
+      Can send acks, but requires additional handshaking for the response channel
+      Multi-event JSON posts are possible, but in non-standard format: { stuff }{ stuff 2 }{ stuff 3} rather than standard [{}{}{}]
+      My Token: 3372606C-6D24-48A4-A28D-09C616A277E7
+
+  Module 8: Fine-Tuning Inputs
+    props.conf is very important
+      inputs phase:
+        character encoding (default is utf8)
+        fine tuned source types
+          can override the defaults on a per file basis
+
+      parsing:
+        event breaks
+        time extraction
+        event transformation
+
+  Module 9: Parsing Phase and Data Preview
+    props.conf.spec - LINE_BREAKER is best way to split lines, ProServ recommended
+    Take extra time to ensure timestamps are correct
+    Either TZ in timestamp, or specified in props.conf or tz of indexer
+
+      

ghetto/notes/SplunkDataAdministration/day3.txt

@@ -0,0 +1,33 @@
+Module 10:
+  Modifying raw data before it's indexed
+  use per event source types only in a last chance scenario, everything else is better
+  to set metadata in transforms.conf
+    SOURCE_KEY = _raw
+    REGEX = server:(\w+)
+    DEST_KEY = MetaData:Host
+    FORMAT = host::$1
+  Host => host::,
+
+  To change the index at index-time (note the additional underscore here)
+    REGEX = (Error|Warning)
+    DEST_KEY = _MetaData:Index
+    FORMAT = itops
+
+  Filter Events
+    FORMAT = nullQueue
+
+  http://<splunk>/debug/refresh - forces splunk to refresh it's config(?)
+    at a minimum it does the inputs configurations, definitely doesn't do the indexer
+
+
+  I need to go over modules 10 and 11. Missed too much i fear
+
+Module 12: Diag
+  Creates diagnostic package for shipment to experts.
+  ./splunk diag
+  Create and index a diag
+
+
+Course Review:
+  Mod 1 - 
+  joanna@splunk.com