Last year I took a very quick look at NSRL hash set. Being de facto golden standard of good hashes I was curious what sort of data is actually included in it. There is no better way of looking at it than actually looking at it so I downloaded the files and started some basic analysis.

The NSRLFile.txt.zip stores 26G file NSRLFile.txt that includes the following number of entries:

192677750 NSRLFile.txt

Yay, that’s a lot!

Now that we know how many entries we have in it, let’s try to see what sort of file extensions we can find inside the file. After parsing the data set I came up with the following results (top 30):

class 44776614
<no ext> 36040752
png 8968329
manifest 6258021
js 4846976
foliage 4513590
py 3938119
dll 3658422
java 3263630
nasl 3256040
html 3068834
h 2671295
xml 2582138
cat 2355352
htm 2197584
mum 2164820
txt 1906815
uasset 1859095
dat 1779063
o 1749951
c 1697767
gz 1691747
mui 1377019
svg 1018434
properties 1004380
mo 1000680
gif 973653
ogg 842367
dds 822591
upk 772181

This is a very interesting statistic — it tells us that a substantial number of records relate to .class files – compiled Java files. This is most likely a result of an automated processing where every single .JAR file is being unzipped and every embedded .class files is accounted for. There are tones of non-executables as well (media files, source code, etc.). It’s a majority of them, really.

Looking specifically for executable / installer files we quickly realize that these are indeed quite scarce:

dll    3658422
o    1749951
bin    557524
so    504442
exe    346292
sys    98390
msi    24763
bat    31482
ps1    13433
cmd    10931
scr    8567
drv    5350

File extensions tell us roughly what file types we deal with, but another great way to look at the NSRL set is statistical analysis of its file names. A quick & dirty histogram I came up with looks like this (top 30):

1221091 ".text"
744897 "1"
719540 "text" 
641856 ".reloc" 
630123 "__bitcode" 
579898 "version.txt" 
530652 ".data" 
457126 "__compact_unwind" 
393021 "__gcc_except_tab" 
387403 "__eh_frame" 
312569 ".rodata" 
312238 "CERTIFICATE" 
265511 "__init.py"
235670 ".rdata"
230514 "Makefile"
214357 ".dynamic"
212477 ".dynsym"
208787 "pathname"
208787 "asset.meta"
208594 ".dynstr"
201074 ".eh_frame"
198758 ".symtab"
194931 ".init"
194760 ".note.gnu.build-id"
194215 ".strtab"
185339 "__const"
184099 "English.dat"
182945 ".gnu.version"
176370 "Master.js"
175495 ".gnu.version_r"

This is yet another interesting statistic that tells us that what we believed to be just large file hashset is actually a mix of files hashes and hashes of sections of executable files. These are useful in malware analysis, but not so much in forensic time-saving exercise that relies on excluding files with known hashes. In other words, if you blindly use NSRL hashset on your forensic images, you are wasting time testing correlations that by the sole nature of NSRL data set will be a waste of CPU cycles. The only set applied to the file system should be hashes of actual files, not their chunks.

The third lesson is pretty obvious — if you plan on using NSRL hashset it’s best to have different sets for different operating systems and for that, we can leverage OpSystemCode field (use a OS-specific subset for the target image). Oh, but not so fast. There are 1300+ OS versions listed inside NSRLOS.txt file. The file includes Amstrad, MSDOS, Novell Netware, NextStep, AIX and many other kinda derelict platforms. If you plan on using NSRL it’s probably good to simply exclude files belonging to these ancient systems first.

Let’s be honest – there is a lot of value in NSRL hashset, but there is always more than just one “but” and I have listed a few. This is not to discourage using it, but be a bit more choosy how we use it and selectively cherry-pick subsets of its data for the time-sensitive analysis.