上一篇文章中讲述了如何对Graylog中日志目的IP地址进行自动化调查，在更早的合集文章中记录了如何将Windows、Linux端点日志接入到Graylog中，接下里需要将网络侧的日志接入到Graylog中，由于网络流量记录了攻击者时如何到达你的业务服务器上，哪些流量是无害的，哪些流量是有害的，影响范围有多大。所以端点日志结合网络流量日志会具备更好的安全可见性。
这一篇会介绍如何接入、提取、富化pfSense防火墙日志。
The pfSense project is a free network firewall distribution, based on the FreeBSD operating system with a custom kernel and including third party free software packages for additional functionality.
https://www.pfsense.org/getting-started/
根据官方文档可以快速将pfSense部署起来，官方文档地址如下
https://docs.netgate.com/pfsense/en/latest/install/download-installer-image.html?_gl=1*mv72zm*_ga*ODAwNzU4MzM5LjE2NzEzNjE2MDA.*_ga_TM99KBGXCB*MTY3MTYyNTc0NS4yLjEuMTY3MTYyNjIxMy4wLjAuMA..部署包下载地址如下
https://www.pfsense.org/download/Architecture选择AMD64，Installer选择DVD，Mirror都是国外站点
下载完成之后参考部署文档进行部署
https://docs.netgate.com/pfsense/en/latest/install/install-walkthrough.html虚拟化部署参考以下链接
https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-esxi.html针对实验室环境部署在VMware Workstation上关键步骤如下图所示
调整网络，分为内网口以及外网口，内网口使用仅主机模式，关闭DHCP，外网口使用NAT模式，在实验室环境分别为VMnet1以及VMnet8，内网使用网段为192.168.13.0/24，外网使用网段为192.168.116.0/24
pfSense虚拟机网络配置如下图所示
部署完成如下图所示
部署完成之后内网口地址允许自己修改，注意开启DHCP服务方便内网电脑自动获取IP地址。使用内网地址登录pfSense控制台（默认使用https协议，由于实验室修改过使用http协议），部署完成默认密码为admin/pfsense
http://192.168.13.10/index.php登录后即可看到pfSense防火墙首页
基础部署完成之后，需要给pfSense安装上snort包，使其具备IDS/IPS功能。
点击System-Package Manager，选择Available Packages，在搜索框中搜索snort，点击Install
点击完后，点击Confirm按钮后等待安装完成。如下图所示
安装完成后，点击Services-Snort，配置Snort参数。
点击Add，在Alert Settings栏中勾选Send Alerts to System Log以及Enable Packet Captures，点击Save
点击WAN Settings旁边的WAN Categories栏，点击Select All，选择所有规则，点击页面最底下的Save
在Global Settings中勾选Enable Snort VRT、Enable Snort GPLv2、Enable ET Open、Enable OpenAppID以及Enable FEODO Tracker Botnet C2 IP Rules，Snort Oinkmaster Code需要到以下网址进行注册获取
https://www.snort.org/users/sign_up把获取到的Oinkmaster Code填入
在Updates标签页中点击Update Rules
更新完成如下图所示，由于没有订阅所以只能使用社区规则
接下来将pfSense日志接入Graylog中。
1.日志接入
查看官方文档pfSense支持Syslog，默认使用UDP协议进行发送，发送时数据为明文
登录Graylog控制台配置日志接入，点击System-Inputs，根据上述信息选择Raw/Plaintext UDP，点击Launch new input
在弹出窗口中把Title设置为pfSense，Port修改成5556，如下图所示，点击Save
完成后如下图所示，接下来还需要对pfSense防火墙传过来的日志添加上标记，类似之前Wazuh server的日志做标记一样，点击刚才新增的Input中右侧More actions选项中Add static field
在弹出窗口中，Field name设置为log_source，Field value设置为pfSense，设置完点击Add field
添加完如下图所示，后续所有pfSense防火墙传过来的日志中都会被添加上这个标记
Graylog接收到pfSense防火墙传过来之后需要进行存储索引，接下来需要创建pfSense防火墙日志使用的索引，点击System-Indices，点击右上角Create index set
在弹出窗口中，把Title以及Description都设置成pfSense alert，Index prefix设置成pfsense-alert，其他参数保持不变，点击Save，具体的参数解读参考合理中第五篇文章开源SOC实现（五）-SIEM日志处理
完成以上操作Graylog区分pfSense防火墙传过来的日志，并且在接收到日志之后默认加上log_source字段用于区分，Graylog中创建了索引用于存储pfSense告警日志，由于Graylog接收到日之后默认存放于All messages这个流的索引中，所以接下来需要把pfSense防火墙传过来的日志存放到刚才创建的pfSense alert索引中，通过定义Stream实现，点击Stream页面，点击Create Stream创建流
在弹出窗口中，Title以及Description设置成pfSense alert，Index Set选择刚才创建的pfSense alert，如下图所示，点击Save
创建完后点击Manage Rules
在弹出窗口中点击Add stream rule
把Field设置成log_source，Type选择match exactly，Value设置成pfSense，如下图所示，点击Save
配置完，点击Start Stream，开启后如下图所示
登录pfSense控制台，点击Status-System Logs
在Remote Logging Options栏中勾选Enable Remote Logging，Remote log servers中填写Graylog中Inputs信息192.168.116.201:5556，在Remote Syslog Contents中取消勾选Everything，勾选System Events以及Firewall Events，点击Save
配置完成后当pfSense中由新的Firewall Events产生可以看到Graylog中Inputs中pfSense右侧中Network IO出现数值变化
2.日志解析
接着需要将pfSense传过来的日志进行正确解析，登录Graylog控制台，点击System-Inputs，找到之前添加的pfSense，点击右侧Manage extractor
在弹出窗口中选择Actions下的Import extractors
把以下内容进行黏贴
{"extractors": [{"title": "pfSense filterlog: IPv6 TCP","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,Class,FlowLabel,HopLimit,Protocol,ProtocolID,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCPFlags,Sequence,ACK,Window,URG,Options"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),6,.*,tcp,.*$"},{"title": "pfSense filterlog: IPv4 UDP","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,(?i)UDP,.*$"},{"title": "pfSense filterlog: IPv4 TCP","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCPFlags,Sequence,ACK,Window,URG,Options"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,tcp,.*$"},{"title": "pfSense filterlog: IPv4 ICMP Unreachable Protocol","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,ICMP_Type,ICMP_DestIP,ICMP_ProtocolID"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,icmp,.*,unreachproto,.*$"},{"title": "pfSense filterlog: IPv4 ICMP Echo","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,ICMP_Type,ICMP_ID,ICMP_Sequence"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,icmp,.*,(request|reply),.*$"},{"title": "pfSense filterlog: IPv4 ICMP Unreachable Port","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,ICMP_Type,ICMP_DestIP,ICMP_ProtocolID,ICMP_Port"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,icmp,.*,unreachport,.*$"},{"title": "pfSense filterlog: IPv6 UDP","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,Class,FlowLabel,HopLimit,Protocol,ProtocolID,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),6,.*,(?i)UDP,.*$"},{"title": "pfSense filterlog: IPv4 ICMP Unreachable Other","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,ICMP_Type,ICMP_Description"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,icmp,.*,(unreach|timexceed|paramprob|redirect|maskreply),.*$"},{"title": "pfSense filterlog: IPv4 ICMP Need Frag","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,ICMP_Type,ICMP_DestIP,ICMP_MTU"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,icmp,.*,needfrag,.*$"},{"title": "pfSense filterlog: IPv4 ICMP TStamp Reply","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,ICMP_Type,ICMP_ID,ICMP_Sequence,ICMP_otime,ICMP_rtime,ICMP_ttime"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,icmp,.*,tstampreply,.*$"},{"title": "pfSense filterlog: IPv4 ICMP TStamp","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,ICMP_Type,ICMP_ID,ICMP_Sequence"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,icmp,.*,tstamp,.*$"},{"title": "pfSense filterlog: IPv4 ICMP Default","extractor_type": "regex","converters": [{"type": "csv","config": {"column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,ICMP_Type,ICMP_Description"}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "FilterData","extractor_config": {"regex_value": "^filterlog:\\s+(.*)$"},"condition_type": "regex","condition_value": "^filterlog:\\s+.*,(in|out),4,.*,icmp,.*,(?!(request|reply|unreachproto|unreachport|unreach|timexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)),.*$"},{"title": "Puppet error - DevOps","extractor_type": "regex","converters": [],"order": 1,"cursor_strategy": "copy","source_field": "message","target_field": "puppetError","extractor_config": {"regex_value": "(err)"},"condition_type": "regex","condition_value": "^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+err+\\)"},{"title": "Snort Destination IP","extractor_type": "regex","converters": [],"order": 34,"cursor_strategy": "copy","source_field": "message","target_field": "DestIp","extractor_config": {"regex_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:.+\\]\\s\\[Priority:\\s\\d\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s(\\S+):\\d+"},"condition_type": "regex","condition_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:.+\\]\\s\\[Priority:\\s\\d\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},{"title": "Snort Priority","extractor_type": "regex","converters": [],"order": 33,"cursor_strategy": "copy","source_field": "message","target_field": "snort_priority","extractor_config": {"regex_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:.+\\]\\s\\[Priority:\\s(\\d)\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},"condition_type": "regex","condition_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:.+\\]\\s\\[Priority:\\s\\d\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},{"title": "Snort Message","extractor_type": "regex","converters": [],"order": 31,"cursor_strategy": "copy","source_field": "message","target_field": "snort_message","extractor_config": {"regex_value": "\\[\\d+:\\d+:\\d\\] (.*) \\[Classification:.+\\]\\s\\[Priority:\\s\\d\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},"condition_type": "regex","condition_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:.+\\]\\s\\[Priority:\\s\\d\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},{"title": "Snort Classification","extractor_type": "regex","converters": [],"order": 32,"cursor_strategy": "copy","source_field": "message","target_field": "snort_classification","extractor_config": {"regex_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:(.+)\\]\\s\\[Priority:\\s\\d\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},"condition_type": "regex","condition_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:.+\\]\\s\\[Priority:\\s\\d\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},{"title": "Snort Protocol","extractor_type": "regex","converters": [],"order": 35,"cursor_strategy": "copy","source_field": "message","target_field": "snort_protocol","extractor_config": {"regex_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:.+\\]\\s\\[Priority:\\s\\d\\]\\s\\{(\\S+)\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},"condition_type": "regex","condition_value": "\\[\\d+:\\d+:\\d\\].*\\[Classification:.+\\]\\s\\[Priority:\\s\\d\\]\\s\\{\\S+\\}\\s\\S+:\\d+\\s->\\s\\S+:\\d+"},{"title": "Puppet: Puppet Configuration Version","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 3,"cursor_strategy": "copy","source_field": "full_message","target_field": "puppetConfigVersion","extractor_config": {"regex_value": "^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+info+\\)+:\\s+Applying+\\s+configuration+\\s+version+\\s+'(\\d[0-9]{0,9})+'"},"condition_type": "regex","condition_value": "^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+info+\\)+:\\s+Applying+\\s+configuration+\\s+version+\\s+'(\\d[0-9]{0,9})+'"},{"title": "Time Request from message IIS","extractor_type": "regex","converters": [],"order": 8,"cursor_strategy": "copy","source_field": "message","target_field": "request_time","extractor_config": {"regex_value": "^\\d+-\\d+-\\d+\\s+(\\d+:\\d+:\\d+)"},"condition_type": "regex","condition_value": "^\\d+-\\d+-\\d+\\s+(\\d+:\\d+:\\d+)+\\s+\\d+.\\d+.\\d+.\\d+"},{"title": "pfSense Action","extractor_type": "regex","converters": [],"order": 24,"cursor_strategy": "copy","source_field": "message","target_field": "action","extractor_config": {"regex_value": ".* (pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},"condition_type": "regex","condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},{"title": "Puppet main Module name","extractor_type": "split_and_index","converters": [],"order": 9,"cursor_strategy": "copy","source_field": "full_message","target_field": "puppetModule","extractor_config": {"index": 3,"split_by": "/"},"condition_type": "regex","condition_value": "^\\d+-\\d+-\\d+\\s+\\d+:\\d+:\\d+\\s+-\\d+\\s+\\/+Stage+\\[+main+\\]+\\/(.*?)+\\/"},{"title": "pfSense Interface","extractor_type": "regex","converters": [],"order": 25,"cursor_strategy": "copy","source_field": "message","target_field": "interface","extractor_config": {"regex_value": ".* (?:pass|block) .* on ([^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},"condition_type": "regex","condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},{"title": "Puppet: Puppet run time - DevOps","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 2,"cursor_strategy": "copy","source_field": "full_message","target_field": "puppetRunTime","extractor_config": {"regex_value": "Finished catalog run in ((\\d[0-9]{0,9}\\.[0-9]{0,9})) seconds"},"condition_type": "regex","condition_value": "^\\d+-+\\d+-\\d+\\s+\\d+:+\\d+:\\d+\\s+-\\d+\\s+Puppet+\\s+\\(+notice+\\)+:\\s+Finished+\\s+catalog+\\s+run+\\s+in+\\s+(\\d[0-9]{0,9}\\.[0-9]{0,9})+\\s+seconds"},{"title": "pfSense Source IP","extractor_type": "regex","converters": [],"order": 27,"cursor_strategy": "copy","source_field": "message","target_field": "srcip","extractor_config": {"regex_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},"condition_type": "regex","condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},{"title": "pfSense Protocol","extractor_type": "regex","converters": [],"order": 26,"cursor_strategy": "copy","source_field": "message","target_field": "protocol","extractor_config": {"regex_value": ".* (?:pass|block) .* on (?:[^:]+):.* (TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},"condition_type": "regex","condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},{"title": "pfSense Source Port","extractor_type": "regex","converters": [],"order": 28,"cursor_strategy": "copy","source_field": "message","target_field": "srcport","extractor_config": {"regex_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},"condition_type": "regex","condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},{"title": "Sudo Executor","extractor_type": "regex","converters": [],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "sudo_executor","extractor_config": {"regex_value": "sudo:\\s+(\\S+)\\s+:"},"condition_type": "string","condition_value": "sudo:"},{"title": "pfSense Destination Port","extractor_type": "regex","converters": [],"order": 30,"cursor_strategy": "copy","source_field": "message","target_field": "destport","extractor_config": {"regex_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(\\d*): (?:.*)"},"condition_type": "regex","condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},{"title": "pfSense Destination IP","extractor_type": "regex","converters": [],"order": 29,"cursor_strategy": "copy","source_field": "message","target_field": "destip","extractor_config": {"regex_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},"condition_type": "regex","condition_value": ".* (?:pass|block) .* on (?:[^:]+):.* (?:TCP|UDP|IGMP|ICMP) .* (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*) [<|>] (?:\\d+\\.\\d+\\.\\d+\\.\\d+)\\.?(?:\\d*): (?:.*)"},{"title": "Sudo Command","extractor_type": "regex","converters": [],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "sudo_command","extractor_config": {"regex_value": "sudo:.+COMMAND=(.+);?"},"condition_type": "string","condition_value": "sudo:"},{"title": "Sudo Command User","extractor_type": "regex","converters": [],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "sudo_command_user","extractor_config": {"regex_value": "sudo:.+USER=(\\S+)"},"condition_type": "string","condition_value": "sudo:"},{"title": "Level/Severity","extractor_type": "regex","converters": [{"type": "syslog_pri_level","config": {}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "level","extractor_config": {"regex_value": "\\d <(.+)>"},"condition_type": "none","condition_value": ""},{"title": "HTTP method","extractor_type": "regex","converters": [{"type": "lowercase","config": {}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "http_method","extractor_config": {"regex_value": "method=(.+?)(\\s|$)"},"condition_type": "string","condition_value": "method="},{"title": "Path","extractor_type": "regex","converters": [],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "path","extractor_config": {"regex_value": "path=(.+?)(\\s|$)"},"condition_type": "string","condition_value": "path="},{"title": "Request duration","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "request_duration","extractor_config": {"regex_value": "duration=(.+?)(\\s|$)"},"condition_type": "string","condition_value": "duration"},{"title": "Facility","extractor_type": "regex","converters": [{"type": "syslog_pri_facility","config": {}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "facility","extractor_config": {"regex_value": "\\d <(.+)>"},"condition_type": "none","condition_value": ""},{"title": "View duration","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "view_duration","extractor_config": {"regex_value": "view=(.+?)(\\s|$)"},"condition_type": "none","condition_value": ""},{"title": "DB Duration","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "db_duration","extractor_config": {"regex_value": "db=(.+?)(\\s|$)"},"condition_type": "none","condition_value": ""},{"title": "Remote Address","extractor_type": "regex","converters": [],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "remote_addr","extractor_config": {"regex_value": "nginx:\\s+(\\S+)"},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "Request ID","extractor_type": "regex","converters": [],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "request_id","extractor_config": {"regex_value": "\\[([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\\]"},"condition_type": "regex","condition_value": "\\[([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\\]"},{"title": "Request Timestamp","extractor_type": "regex","converters": [{"type": "date","config": {"date_format": "dd/MMM/YYYY:HH:mm:ss Z"}}],"order": 2,"cursor_strategy": "copy","source_field": "message","target_field": "timestamp","extractor_config": {"regex_value": "nginx:.+?\\[(.+?)\\]"},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "Request Verb","extractor_type": "regex","converters": [],"order": 3,"cursor_strategy": "copy","source_field": "message","target_field": "request_verb","extractor_config": {"regex_value": "nginx:.+\\[.+\\] \"(\\S+)"},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "Remote User","extractor_type": "regex","converters": [],"order": 1,"cursor_strategy": "copy","source_field": "message","target_field": "remote_user","extractor_config": {"regex_value": "nginx: \\S+ - (\\S+)"},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "Request Path","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 4,"cursor_strategy": "copy","source_field": "message","target_field": "request_path","extractor_config": {"regex_value": "nginx:.+?\"\\S+ (\\S+).+\""},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "HTTP Version","extractor_type": "regex","converters": [],"order": 5,"cursor_strategy": "copy","source_field": "message","target_field": "http_version","extractor_config": {"regex_value": "nginx:.+HTTP/(\\S+)\""},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "Response Status","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 6,"cursor_strategy": "copy","source_field": "message","target_field": "response_status","extractor_config": {"regex_value": "nginx:.+?HTTP/\\S+\" (\\d+)"},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "HTTP Referer","extractor_type": "regex","converters": [],"order": 9,"cursor_strategy": "copy","source_field": "message","target_field": "http_referer","extractor_config": {"regex_value": "nginx:.+?HTTP/\\S+\" \\d+ \\d+ \"(.+?)\""},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "Response Bytes","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 7,"cursor_strategy": "copy","source_field": "message","target_field": "response_bytes","extractor_config": {"regex_value": "nginx:.+?HTTP/\\S+\" \\d+ (\\d+)"},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "HTTP User Agent","extractor_type": "regex","converters": [],"order": 8,"cursor_strategy": "copy","source_field": "message","target_field": "http_user_agent","extractor_config": {"regex_value": "nginx:.+?HTTP/\\S+\" \\d+ \\d+ \".+?\" \"(.+?)\""},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "Connection requests","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 11,"cursor_strategy": "copy","source_field": "message","target_field": "connection_requests","extractor_config": {"regex_value": "connection_requests=(.+?)\\|"},"condition_type": "regex","condition_value": ".+connection_requests=.+"},{"title": "Connection ID","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 10,"cursor_strategy": "copy","source_field": "message","target_field": "connection_id","extractor_config": {"regex_value": "connection=(.+?)\\|"},"condition_type": "regex","condition_value": ".+connection=.+"},{"title": "Response time","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 12,"cursor_strategy": "copy","source_field": "message","target_field": "millis","extractor_config": {"regex_value": "millis=(.+?)>"},"condition_type": "regex","condition_value": ".+millis=.+"},{"title": "Message","extractor_type": "regex","converters": [],"order": 13,"cursor_strategy": "copy","source_field": "message","target_field": "message","extractor_config": {"regex_value": "nginx:.+?\\\"(\\S+.+HTTP\\/\\S+)\\\" \\d+"},"condition_type": "regex","condition_value": "^\\S+\\s+nginx:"},{"title": "Response Status","extractor_type": "regex","converters": [{"type": "numeric","config": {}}],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "response_status","extractor_config": {"regex_value": "status=(.+?)(\\s|$)"},"condition_type": "string","condition_value": "status="},{"title": "Remote Address","extractor_type": "regex","converters": [],"order": 0,"cursor_strategy": "copy","source_field": "message","target_field": "remote_addr","extractor_config": {"regex_value": "\\[(\\b(?:\\d{1,3}\\.){3}\\d{1,3}\\b)\\]"},"condition_type": "regex","condition_value": "\\[(\\b(?:\\d{1,3}\\.){3}\\d{1,3}\\b)\\]"}],"version": "2.4.3"}
点击Add extractors to input
此时页面不会自动回到上级页面，手动返回看到刚添加的extractor，如下图所示
点击System-Inputs中pfSense，点击Show received messages，查看刚才添加的extractor是否正常解析数据
可以看到正常进行日志解析
接下来对防火墙传过来的日志中目的IP地址字段做自动化调查。
由于已经在开源SOC实现（七）-自动调查之目的IP地址这篇文章中已经创建了Lookup Tables就不需要重复创建了，详细的过程参考该合集中对应文章。
接下来创建Pipelines，用于通知Graylog，日志中什么字段为目的IP需要进行自动化调查。点击System-Pipelines，点击Manage rules
在弹出窗口中点击Create Rule
把Description字段设置为GreyNoise Lookup on DestIp，对应的Rule source唯一下所示，通过之前查看日日志目的IP地址字段为DestIp，使用时对目的IP字段进行替换，配置完点击Apply，再点击Save&Close
配置完如下图所示
点击Manage Pipelines
点击Add new pipeline
在弹出窗口中Title以及Description都设置成GreyNoise pipeline 2，完成后点击Save
在弹出页面中点击在Pipeline connections后点击Edit connections，选择pfSense alert这个Stream，点击Save
点击Stage 0后的Edit按钮
Stage rule选择刚才创建的rule，点击Save
完成后如下图所示
最后点击System-Configurations，把Message Processors Configuration栏中，Pipeline Processor位置调整到Message Filter Chain下方
完成之后查看新收到的日志中已经带有GreyNoise_字段，如下图所示
接下来用开源SOC实现（六）-Grafana Dashboard中说过创建Grafana面板来对自动调查IP结果进行一个展示，详细过程请参考该合集对应文章，该面板配置如下
结合上篇文章讲述的内容进行配置完展示如下
这样子就完成pfSense防火墙日志接入并进行自动化调查。到这边就完成了终端日志接入以及防火墙的流量日志接入，可以发现接入的数据源越多，对于安全的可见性越高，但随之带来的一个困恼就是各数据源虽然包含的日志字段差不多，但字段名称多少有差异，这就不利于进行Grafana展示以及自动化调查，甚至是人工狩猎，于是需要对各数据源的关键日志字段进行统一处理，欢迎订阅收看开源SOC实现（九）-多源数据处理。
文章来源: http://mp.weixin.qq.com/s?__biz=MzI3NDYwMzI4Mg==&mid=2247486172&idx=1&sn=3a1afe6dd99ae6465c37ed49931b5975&chksm=eb10c6a1dc674fb7779407c5c1def19a69dd8665bfc853c70d3457b3632be4389c457e4e455f#rd
如有侵权请联系:admin#unsafe.sh
如有侵权请联系:admin#unsafe.sh