httpx 是一个快速且多用途的 HTTP 工具包,它允许使用 retryablehttp 库运行多个探测。
2023-4-4 08:1:17 Author: Ots安全(查看原文) 阅读量:81 收藏

httpx是一个快速且多用途的 HTTP 工具包,允许使用retryablehttp库运行多个探测器。它旨在通过增加线程数来保持结果的可靠性。【可直接滑到底部点击阅读原文跳转到项目地址

  • 简单和模块化的代码库使其易于贡献。

  • 快速且完全可配置的标志,用于探测多个元素。

  • 支持多个基于 HTTP 的探测。

  • 默认情况下从 https 到 http 的智能自动回退。

  • 支持主机、URL 和 CIDR 作为输入。

  • 处理边缘情况,为处理 WAF 进行重试、退避等。

支持的标头:

httpx需要go1.19才能安装成功。运行以下命令以获取存储库:

go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
httpx -h

这将显示该工具的帮助。这是它支持的所有开关。

Usage:  ./httpx [flags]
Flags:INPUT: -l, -list string input file containing list of hosts to process -rr, -request string file containing raw request -u, -target string[] input target host(s) to probe
PROBES: -sc, -status-code display response status-code -cl, -content-length display response content-length -ct, -content-type display response content-type -location display response redirect location -favicon display mmh3 hash for '/favicon.ico' file -hash string display response body hash (supported: md5,mmh3,simhash,sha1,sha256,sha512) -jarm display jarm fingerprint hash -rt, -response-time display response time -lc, -line-count display response body line count -wc, -word-count display response body word count -title display page title -server, -web-server display server name -td, -tech-detect display technology in use based on wappalyzer dataset -method display http request method -websocket display server using websocket -ip display host ip -cname display host cname -asn display host asn information -cdn display cdn in use -probe display probe status
MATCHERS: -mc, -match-code string match response with specified status code (-mc 200,302) -ml, -match-length string match response with specified content length (-ml 100,102) -mlc, -match-line-count string match response body with specified line count (-mlc 423,532) -mwc, -match-word-count string match response body with specified word count (-mwc 43,55) -mfc, -match-favicon string[] match response with specified favicon hash (-mfc 1494302000) -ms, -match-string string match response with specified string (case insensitive) (-ms admin) -mr, -match-regex string match response with specified regex (-mr admin) -mcdn, -match-cdn string[] match host with specified cdn provider (oracle, google, azure, cloudflare, cloudfront, fastly, incapsula, leaseweb, akamai, sucuri) -mrt, -match-response-time string match response with specified response time in seconds (-mrt '< 1') -mdc, -match-condition string match response with dsl expression condition
EXTRACTOR: -er, -extract-regex string[] display response content with matched regex -ep, -extract-preset string[] display response content matched by a pre-defined regex (url,ipv4,mail)
FILTERS: -fc, -filter-code string filter response with specified status code (-fc 403,401) -fl, -filter-length string filter response with specified content length (-fl 23,33) -flc, -filter-line-count string filter response body with specified line count (-flc 423,532) -fwc, -filter-word-count string filter response body with specified word count (-fwc 423,532) -ffc, -filter-favicon string[] filter response with specified favicon hash (-mfc 1494302000) -fs, -filter-string string filter response with specified string (-fs admin) -fe, -filter-regex string filter response with specified regex (-fe admin) -fcdn, -filter-cdn string[] filter host with specified cdn provider (oracle, google, azure, cloudflare, cloudfront, fastly, incapsula, leaseweb, akamai, sucuri) -frt, -filter-response-time string filter response with specified response time in seconds (-frt '> 1') -fdc, -filter-condition string filter response with dsl expression condition
RATE-LIMIT: -t, -threads int number of threads to use (default 50) -rl, -rate-limit int maximum requests to send per second (default 150) -rlm, -rate-limit-minute int maximum number of requests to send per minute
MISCELLANEOUS: -pa, -probe-all-ips probe all the ips associated with same host -p, -ports string[] ports to probe (nmap syntax: eg http:1,2-10,11,https:80) -path string path or list of paths to probe (comma-separated, file) -tls-probe send http probes on the extracted TLS domains (dns_name) -csp-probe send http probes on the extracted CSP domains -tls-grab perform TLS(SSL) data grabbing -pipeline probe and display server supporting HTTP1.1 pipeline -http2 probe and display server supporting HTTP2 -vhost probe and display server supporting VHOST -ldv, -list-dsl-variables list json output field keys name that support dsl matcher/filter
OUTPUT: -o, -output string file to write output results -sr, -store-response store http response to output directory -srd, -store-response-dir string store http response to custom directory -csv store output in csv format -csvo, -csv-output-encoding string define output encoding -json store output in JSONL(ines) format -irr, -include-response include http request/response in JSON output (-json only) -irrb, -include-response-base64 include base64 encoded http request/response in JSON output (-json only) -include-chain include redirect http chain in JSON output (-json only) -store-chain include http redirect chain in responses (-sr only)
CONFIGURATIONS: -r, -resolvers string[] list of custom resolver (file or comma separated) -allow string[] allowed list of IP/CIDR's to process (file or comma separated) -deny string[] denied list of IP/CIDR's to process (file or comma separated) -sni, -sni-name string custom TLS SNI name -random-agent enable Random User-Agent to use (default true) -H, -header string[] custom http headers to send with request -http-proxy, -proxy string http proxy to use (eg http://127.0.0.1:8080) -unsafe send raw requests skipping golang normalization -resume resume scan using resume.cfg -fr, -follow-redirects follow http redirects -maxr, -max-redirects int max number of redirects to follow per host (default 10) -fhr, -follow-host-redirects follow redirects on the same host -vhost-input get a list of vhosts as input -x string request methods to probe, use 'all' to probe all HTTP methods -body string post body to include in http request -s, -stream stream mode - start elaborating input targets without sorting -sd, -skip-dedupe disable dedupe input items (only used with stream mode) -ldp, -leave-default-ports leave default http/https ports in host header (eg. http://host:80 - https//host:443 -ztls use ztls library with autofallback to standard one for tls13
DEBUG: -health-check, -hc run diagnostic check up -debug display request/response content in cli -debug-req display request content in cli -debug-resp display response content in cli -version display httpx version -stats display scan statistic -profile-mem string optional httpx memory profile dump file -silent silent mode -v, -verbose verbose mode -si, -stats-interval int number of seconds to wait between showing a statistics update (default: 5) -nc, -no-color disable colors in cli output
OPTIMIZATIONS: -nf, -no-fallback display both probed protocol (HTTPS and HTTP) -nfs, -no-fallback-scheme probe with protocol scheme specified in input -maxhr, -max-host-error int max error count per host before skipping remaining path/s (default 30) -ec, -exclude-cdn skip full port scans for CDNs (only checks for 80,443) -retries int number of retries -timeout int timeout in seconds (default 5) -delay duration duration between each http request (eg: 200ms, 1s) (default -1ns) -rsts, -response-size-to-save int max response size to save in bytes (default 2147483647) -rstr, -response-size-to-read int max response size to read in bytes (default 2147483647)

网址探测

这将针对所有主机和子域运行该工具hosts.txt,并返回运行 HTTP 网络服务器的 URL。

cat hosts.txt | httpx 
__ __ __ _ __ / /_ / /_/ /_____ | |/ / / __ \/ __/ __/ __ \| / / / / / /_/ /_/ /_/ / | /_/ /_/\__/\__/ .___/_/|_| v1.1.1 /_/
projectdiscovery.io
[WRN] Use with caution. You are responsible for your actions[WRN] Developers assume no liability and are not responsible for any misuse or damage.
https://mta-sts.managed.hackerone.comhttps://mta-sts.hackerone.comhttps://mta-sts.forwarding.hackerone.comhttps://docs.hackerone.comhttps://www.hackerone.comhttps://resources.hackerone.comhttps://api.hackerone.comhttps://support.hackerone.com

文件输入

这将针对hosts.txt-probe中的所有主机运行带有标志的工具,并返回具有探测状态的 URL。

httpx -list hosts.txt -silent -probe
http://ns.hackerone.com [FAILED]https://docs.hackerone.com [SUCCESS]https://mta-sts.hackerone.com [SUCCESS]https://mta-sts.managed.hackerone.com [SUCCESS]http://email.hackerone.com [FAILED]https://mta-sts.forwarding.hackerone.com [SUCCESS]http://links.hackerone.com [FAILED]https://api.hackerone.com [SUCCESS]https://www.hackerone.com [SUCCESS]http://events.hackerone.com [FAILED]https://support.hackerone.com [SUCCESS]https://gslink.hackerone.com [SUCCESS]http://o1.email.hackerone.com [FAILED]http://info.hackerone.com [FAILED]https://resources.hackerone.com [SUCCESS]http://o2.email.hackerone.com [FAILED]http://o3.email.hackerone.com [FAILED]http://go.hackerone.com [FAILED]http://a.ns.hackerone.com [FAILED]http://b.ns.hackerone.com [FAILED]

网段输入

echo 173.0.84.0/24 | httpx -silent
https://173.0.84.29https://173.0.84.43https://173.0.84.31https://173.0.84.44https://173.0.84.12https://173.0.84.4https://173.0.84.36https://173.0.84.45https://173.0.84.14https://173.0.84.25https://173.0.84.46https://173.0.84.24https://173.0.84.32https://173.0.84.9https://173.0.84.13https://173.0.84.6https://173.0.84.16https://173.0.84.34

AS号输入

echo AS14421 | httpx -silent
https://216.101.17.248https://216.101.17.249https://216.101.17.250https://216.101.17.251https://216.101.17.252

工具链

subfinder -d hackerone.com -silent| httpx -title -tech-detect -status-code
__ __ __ _ __ / /_ / /_/ /_____ | |/ / / __ \/ __/ __/ __ \| / / / / / /_/ /_/ /_/ / |/_/ /_/\__/\__/ .___/_/|_| /_/ v1.1.1
projectdiscovery.io
Use with caution. You are responsible for your actionsDevelopers assume no liability and are not responsible for any misuse or damage.https://mta-sts.managed.hackerone.com [404] [Page not found · GitHub Pages] [Varnish,GitHub Pages,Ruby on Rails]https://mta-sts.hackerone.com [404] [Page not found · GitHub Pages] [Varnish,GitHub Pages,Ruby on Rails]https://mta-sts.forwarding.hackerone.com [404] [Page not found · GitHub Pages] [GitHub Pages,Ruby on Rails,Varnish]https://docs.hackerone.com [200] [HackerOne Platform Documentation] [Ruby on Rails,jsDelivr,Gatsby,React,webpack,Varnish,GitHub Pages]https://support.hackerone.com [301,302,301,200] [HackerOne] [Cloudflare,Ruby on Rails,Ruby]https://resources.hackerone.com [301,301,404] [Sorry, no Folders found.]

图标哈希

subfinder -d hackerone.com -silent | httpx -favicon
__ __ __ _ __ / /_ / /_/ /_____ | |/ / / __ \/ __/ __/ __ \| / / / / / /_/ /_/ /_/ / |/_/ /_/\__/\__/ .___/_/|_| /_/ v1.1.5
projectdiscovery.io
Use with caution. You are responsible for your actions.Developers assume no liability and are not responsible for any misuse or damage.https://docs.hackerone.com/favicon.ico [595148549]https://hackerone.com/favicon.ico [595148549]https://mta-sts.managed.hackerone.com/favicon.ico [-1700323260]https://mta-sts.forwarding.hackerone.com/favicon.ico [-1700323260]https://support.hackerone.com/favicon.ico [-1279294674]https://gslink.hackerone.com/favicon.ico [1506877856]https://resources.hackerone.com/favicon.ico [-1840324437]https://api.hackerone.com/favicon.ico [566218143]https://mta-sts.hackerone.com/favicon.ico [-1700323260]https://www.hackerone.com/favicon.ico [778073381]

JARM指纹

subfinder -d hackerone.com -silent | httpx -jarm    __    __  __       _  __   / /_  / /_/ /_____ | |/ /  / __ \/ __/ __/ __ \|   / / / / / /_/ /_/ /_/ /   |/_/ /_/\__/\__/ .___/_/|_|             /_/              v1.2.1
projectdiscovery.io
Use with caution. You are responsible for your actions.Developers assume no liability and are not responsible for any misuse or damage.https://www.hackerone.com [29d3dd00029d29d00042d43d00041d5de67cc9954cc85372523050f20b5007]https://mta-sts.hackerone.com [29d29d00029d29d00042d43d00041d2aa5ce6a70de7ba95aef77a77b00a0af]https://mta-sts.managed.hackerone.com [29d29d00029d29d00042d43d00041d2aa5ce6a70de7ba95aef77a77b00a0af]https://docs.hackerone.com [29d29d00029d29d00042d43d00041d2aa5ce6a70de7ba95aef77a77b00a0af]https://support.hackerone.com [29d3dd00029d29d00029d3dd29d29d5a74e95248e58a6162e37847a24849f7]https://api.hackerone.com [29d3dd00029d29d00042d43d00041d5de67cc9954cc85372523050f20b5007]https://mta-sts.forwarding.hackerone.com [29d29d00029d29d00042d43d00041d2aa5ce6a70de7ba95aef77a77b00a0af]https://resources.hackerone.com [2ad2ad0002ad2ad0002ad2ad2ad2ad043bfbd87c13813505a1b60adf4f6ff5]

ASN 指纹

subfinder -d hackerone.com -silent | httpx -asn    __    __  __       _  __   / /_  / /_/ /_____ | |/ /  / __ \/ __/ __/ __ \|   / / / / / /_/ /_/ /_/ /   |/_/ /_/\__/\__/ .___/_/|_|             /_/              v1.2.1
projectdiscovery.io
Use with caution. You are responsible for your actions.Developers assume no liability and are not responsible for any misuse or damage.https://mta-sts.managed.hackerone.com [AS54113, FASTLY, US]https://gslink.hackerone.com [AS16509, AMAZON-02, US]https://www.hackerone.com [AS13335, CLOUDFLARENET, US]https://mta-sts.forwarding.hackerone.com [AS54113, FASTLY, US]https://resources.hackerone.com [AS16509, AMAZON-02, US]https://support.hackerone.com [AS13335, CLOUDFLARENET, US]https://mta-sts.hackerone.com [AS54113, FASTLY, US]https://docs.hackerone.com [AS54113, FASTLY, US]https://api.hackerone.com [AS13335, CLOUDFLARENET, US]

文件/路径暴力破解

httpx -l urls.txt -path /v1/api -sc
__ __ __ _ __ / /_ / /_/ /_____ | |/ / / __ \/ __/ __/ __ \| / / / / / /_/ /_/ /_/ / |/_/ /_/\__/\__/ .___/_/|_| /_/ v1.1.5
projectdiscovery.io
Use with caution. You are responsible for your actions.Developers assume no liability and are not responsible for any misuse or damage.https://mta-sts.managed.hackerone.com/v1/api [404]https://mta-sts.hackerone.com/v1/api [404]https://mta-sts.forwarding.hackerone.com/v1/api [404]https://docs.hackerone.com/v1/api [404]https://api.hackerone.com/v1/api [401]https://hackerone.com/v1/api [302]https://support.hackerone.com/v1/api [404]https://resources.hackerone.com/v1/api [301]https://gslink.hackerone.com/v1/api [404]http://www.hackerone.com/v1/api [301]

Docker运行

cat sub_domains.txt | docker run -i projectdiscovery/httpx
__ __ __ _ __ / /_ / /_/ /_____ | |/ / / __ \/ __/ __/ __ \| / / / / / /_/ /_/ /_/ / |/_/ /_/\__/\__/ .___/_/|_| /_/ v1.1.2
projectdiscovery.io
Use with caution. You are responsible for your actionsDevelopers assume no liability and are not responsible for any misuse or damage.https://mta-sts.forwarding.hackerone.comhttps://mta-sts.hackerone.comhttps://mta-sts.managed.hackerone.comhttps://www.hackerone.comhttps://api.hackerone.comhttps://gslink.hackerone.comhttps://resources.hackerone.comhttps://docs.hackerone.comhttps://support.hackerone.com

httpx作为库使用

httpxOption可以通过创建结构的实例并使用将通过 CLI 指定的相同选项填充它来用作库。验证后,该结构应传递给运行器实例(将在程序结束时关闭)并RunEnumeration调用该方法。以下是如何执行此操作的最小示例:

package main
import ( "log"
"github.com/projectdiscovery/goflags" "github.com/projectdiscovery/gologger" "github.com/projectdiscovery/gologger/levels" "github.com/projectdiscovery/httpx/runner")
func main() { gologger.DefaultLogger.SetMaxLevel(levels.LevelVerbose) // increase the verbosity (optional)
options := runner.Options{ Methods: "GET", InputTargetHost: goflags.StringSlice{"scanme.sh", "projectdiscovery.io"}, //InputFile: "./targetDomains.txt", // path to file containing the target domains list }
if err := options.ValidateOptions(); err != nil { log.Fatal(err) }
httpxRunner, err := runner.New(&options) if err != nil { log.Fatal(err) } defer httpxRunner.Close()
httpxRunner.RunEnumeration()}
  • 默认情况下,仅当无法访问HTTPShttpx时才检查HTTPS探测并回退到HTTP 。

  • -no-fallback标志可用于显示HTTPHTTPS结果

  • 可以定义端口的自定义方案,例如

    -ports http:443,http:80,https:8443
  • 以下标志应该用于特定用例,而不是将它们作为其他探测器的默认值运行:

      -favicon,,,,,,,, -vhost_ -http2_ -pipeline_ -ports_ -csp-probe_ -tls-probe_-path
  • 使用该-json标志时,所有默认探测结果都包含在 JSON 输出中。

  • 自定义解析器以(例如)的形式支持多种协议(doh|tcp|udp )

    protocol:resolver:portudp:127.0.0.1:53
  • 无效的自定义解析器/文件将被忽略。

项目地址:https://github.com/projectdiscovery/httpx


文章来源: http://mp.weixin.qq.com/s?__biz=MzAxMjYyMzkwOA==&mid=2247497181&idx=1&sn=61b38ace3b26673fce0bbd558fde9c66&chksm=9badbc96acda3580623160bcc61be670fa2f159e0637c5fd2fc87941c06ee9302cd3ad1c71b4#rd
如有侵权请联系:admin#unsafe.sh