CIRCL Services

CIRCL hashlookup

CIRCL hashlookup (hashlookup.circl.lu)

CIRCL hashlookup is a public API for looking up file hashes in known-file databases. It includes the NSRL Reference Data Set (RDS) and several other data sources. The service is available through an HTTP REST API, which is documented using an OpenAPI specification. It is free to use and provided on a best-effort basis.

Sources included in CIRCL hashlookup

  • Common Windows 10 and Windows 11 builds (French, Dutch, German, UK, and US)
  • NIST NSRL - All RDS hash sets, including current, modern, Android, iOS, and legacy sets, with SHA-256 mappings
  • Ubuntu distribution packages
  • CentOS core operating system distribution
  • Fedora Project EPEL repository
  • Kali Linux distribution packages
  • openSUSE distribution packages
  • OpenBSD binary tar.gz archives
  • CDNJS
  • Snap public repository

Is this a database of malicious or non-malicious file hashes?

The CIRCL hashlookup service only provides details about known files that appear in one or more of its source databases. These details provide context for file hashes encountered during investigations or digital forensic analysis. A match does not, by itself, indicate whether a file is malicious.

hashlookup:trust

A trust level is included in every hashlookup response in the hashlookup:trust field.

The trust level ranges from 0 to 100. A value of 50 means that the service has no opinion about the file. A value below 50 indicates less confidence that the file is legitimate. A value above 50 indicates that the file appears in multiple sources and is therefore considered more trustworthy.

API Usage

Get information about the hash lookup database (via ReST)

curl -X 'GET' \
  'https://hashlookup.circl.lu/info' \
  -H 'accept: application/json'

1
2
3
4
5
6
7
8
9

{
  "nsrl-version": "March 2022",
  "stat:hashlookup_total_keys": 5077811007,
  "stat:nsrl_modern_rds": "192677749",
  "stat:nsrl_legacy": "113737918",
  "stat:nsrl_ios": "931242",
  "stat:nsrl_android": "41589780",
  "hashlookup-version": "1.2"
}

Perform an MD5 hash lookup

curl -X 'GET' \
  'https://hashlookup.circl.lu/lookup/md5/8ED4B4ED952526D89899E723F3488DE4' \
  -H 'accept: application/json'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{
  "CRC32": "7A5407CA",
  "FileName": "wow64_microsoft-windows-i..timezones.resources_31bf3856ad364e35_10.0.16299.579_de-de_f24979c73226184d.manifest",
  "FileSize": "2520",
  "MD5": "8ED4B4ED952526D89899E723F3488DE4",
  "OpSystemCode": {
    "MfgCode": "1006",
    "OpSystemCode": "362",
    "OpSystemName": "TBD",
    "OpSystemVersion": "none"
  },
  "ProductCode": {
    "ApplicationType": "Security",
    "Language": "Multilanguage",
    "MfgCode": "608",
    "OpSystemCode": "868",
    "ProductCode": "190742",
    "ProductName": "Cumulative Update for Windows Server 2016 for x64 (KB4338817)",
    "ProductVersion": "1709"
  },
  "SHA-1": "00000079FD7AAC9B2F9C988C50750E1F50B27EB5",
  "SpecialCode": "",
  "db": "nsrl_modern_rds",
  "insert-timestamp": "1630942434.8964827",
  "source": "NSRL"
}

Perform a SHA-1 hash lookup

curl -X 'GET'   'https://hashlookup.circl.lu/lookup/sha1/FFFFFDAC1B1B4C513896C805C2C698D9688BE69F'   -H 'accept: application/json' | jq .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{
  "CRC32": "CBD64CD9",
  "FileName": ".rela.dyn",
  "FileSize": "240",
  "MD5": "131312A96CAD4ACAA7E2631A34A0D47C",
  "OpSystemCode": {
    "MfgCode": "1006",
    "OpSystemCode": "362",
    "OpSystemName": "TBD",
    "OpSystemVersion": "none"
  },
  "ProductCode": {
    "ApplicationType": "Operating System",
    "Language": "English",
    "MfgCode": "1722",
    "OpSystemCode": "599",
    "ProductCode": "163709",
    "ProductName": "BlackArch Linux",
    "ProductVersion": "2017.03.01"
  },
  "SHA-1": "FFFFFDAC1B1B4C513896C805C2C698D9688BE69F",
  "SpecialCode": "",
  "db": "nsrl_modern_rds",
  "insert-timestamp": "1631011386.4436111",
  "source": "NSRL"
}

Perform a SHA-256 hash lookup

curl -s -X 'GET'   'https://hashlookup.circl.lu/lookup/sha256/301c9ec7a9aadee4d745e8fd4fa659dafbbcc6b75b9ff491d14cbbdd840814e9'   -H 'accept: application/json' | jq

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

{
  "FileName": "./usr/bin/openssl",
  "FileSize": "723944",
  "MD5": "34D827A288FA51B93297EF2A8A43B769",
  "SHA-1": "72F104BF11A12511154267328F069FE0541E841E",
  "SHA-256": "301C9EC7A9AADEE4D745E8FD4FA659DAFBBCC6B75B9FF491D14CBBDD840814E9",
  "SHA-512": "2533D682DB224F0D3BEA043A8A986DC1D341FBEFFD158CB97CD360190BE091F43CC6DBF07E6E985CC0DCE17ADC207A61AC9831BE91099202093ACFED584602D1",
  "SSDEEP": "12288:g7LKf6QceJ83r69SOPdxouwUnSysbLY+YR2L7b+3l7E71rb/t:gsceJ83rESOlxJwUZsbLY+YR2Xa3l7E7",
  "TLSH": "T150F4281AE64719BDC8B2C230455B50327A31B945F332BF6B26C196311E42B1EA73FBE5",
  "insert-timestamp": "1636385379.0646722",
  "source": "snap:BbsqA1how7wjAmzvZEBaOXf5L7I9NBHe_31",
  "hashlookup:parent-total": 124,
  "parents": [
    {
      "SHA-1": "0006E05A9FC1F165A94713131592E4269DCB0B5D"
    },
    {
      "SHA-1": "027EC67FDB1BCB3CA236FEAC0A47334ECE3F5BB0"
    },
    {
      "FileSize": "613848",
      "MD5": "124A707963928961F17F873921B0DF13",
      "PackageDescription": "Secure Sockets Layer toolkit - cryptographic utility\n This package is part of the OpenSSL project's implementation of the SSL\n and TLS cryptographic protocols for secure communication over the\n Internet.\n .\n It contains the general-purpose command line binary /usr/bin/openssl,\n useful for cryptographic operations such as:\n  * creating RSA, DH, and DSA key parameters;\n  * creating X.509 certificates, CSRs, and CRLs;\n  * calculating message digests;\n  * encrypting and decrypting with ciphers;\n  * testing SSL/TLS clients and servers;\n  * handling S/MIME signed or encrypted mail.",
      "PackageMaintainer": "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>",
      "PackageName": "openssl",
      "PackageSection": "utils",
      "PackageVersion": "1.1.1-1ubuntu2.1~18.04.13",
      "SHA-1": "02ADDB9985B9F21F42072CEA4A3C1A97448C67AC",
      "SHA-256": "E8E123812167819F0D1AD572C85094F13369413A6E3D1127E4A786CC0A31FD0D"
    },
    {
      "SHA-1": "05EAE0930E00C981FB9EE08BBA153CA6C310CB62"
    },
    {
      "SHA-1": "06DFA4B0BA4E3E6A9CD72455A5F4B0D5F6D579C4"
    },
    {
      "SHA-1": "0721FF5DB7675EEF9627EC9D664F6494A4DB651A"
    },
    {
      "SHA-1": "08797034F4F2681C861EB210B7A0CFE1BE608E00"
    },
    {
      "SHA-1": "088A0984F19981D1B3523C1B11752D19907C61D0"
    },
    {
      "SHA-1": "0A879A1E2214A51D3101FA3406F885C93F0269CD"
    },
    {
      "SHA-1": "0A8F5BBF8826329A0F4C7A062204B7F4BC901414"
    }
  ]
}

Bulk search of MD5 hashes

curl -X 'POST'   'https://hashlookup.circl.lu/bulk/md5' -H "Content-Type: application/json"  -d "{\"hashes\": [\"6E2F8616A01725DCB37BED0A2495AEB2\", \"8ED4B4ED952526D89899E723F3488DE4\", \"344428FA4BA313712E4CA9B16D089AC4\"]}" | jq .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

[
  {
    "CRC32": "E774FD92",
    "FileName": "network",
    "FileSize": "7279",
    "MD5": "6E2F8616A01725DCB37BED0A2495AEB2",
    "OpSystemCode": "362",
    "ProductCode": "8321",
    "SHA-1": "00000903319A8CE18A03DFA22C07C6CA43602061",
    "SpecialCode": "",
    "db": "nsrl_legacy",
    "insert-timestamp": "1631050497.0385447",
    "source": "NSRL"
  },
  {
    "CRC32": "7A5407CA",
    "FileName": "wow64_microsoft-windows-i..timezones.resources_31bf3856ad364e35_10.0.16299.579_de-de_f24979c73226184d.manifest",
    "FileSize": "2520",
    "MD5": "8ED4B4ED952526D89899E723F3488DE4",
    "OpSystemCode": "362",
    "ProductCode": "190742",
    "SHA-1": "00000079FD7AAC9B2F9C988C50750E1F50B27EB5",
    "SpecialCode": "",
    "db": "nsrl_modern_rds",
    "insert-timestamp": "1630942434.8964827",
    "source": "NSRL"
  },
  {
    "CRC32": "7516A25F",
    "FileName": ".text._ZNSt14overflow_errorC1ERKSs",
    "FileSize": "33",
    "MD5": "344428FA4BA313712E4CA9B16D089AC4",
    "OpSystemCode": "362",
    "ProductCode": "219181",
    "SHA-1": "0000001FFEF4BE312BAB534ECA7AEAA3E4684D85",
    "SpecialCode": "",
    "db": "nsrl_modern_rds",
    "insert-timestamp": "1630942434.8922813",
    "source": "NSRL"
  }
]

Bulk search of SHA-1 hashes

curl -X 'POST'   'https://hashlookup.circl.lu/bulk/sha1' -H "Content-Type: application/json"  -d "{\"hashes\": [\"FFFFFDAC1B1B4C513896C805C2C698D9688BE69F\", \"FFFFFF4DB8282D002893A9BAF00E9E9D4BA45E65\", \"FFFFFE4C92E3F7282C7502F1734B243FA52326FB\"]}" | jq .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

[
  {
    "CRC32": "CBD64CD9",
    "FileName": ".rela.dyn",
    "FileSize": "240",
    "MD5": "131312A96CAD4ACAA7E2631A34A0D47C",
    "OpSystemCode": "362",
    "ProductCode": "163709",
    "SHA-1": "FFFFFDAC1B1B4C513896C805C2C698D9688BE69F",
    "SpecialCode": "",
    "db": "nsrl_modern_rds",
    "insert-timestamp": "1631011386.4436111",
    "source": "NSRL"
  },
  {
    "CRC32": "8654F11A",
    "FileName": "s_copypix.c",
    "FileSize": "19541",
    "MD5": "559D049F44942683093A91BA19D0AF54",
    "OpSystemCode": "362",
    "ProductCode": "223222",
    "SHA-1": "FFFFFF4DB8282D002893A9BAF00E9E9D4BA45E65",
    "SpecialCode": "",
    "db": "nsrl_modern_rds",
    "insert-timestamp": "1631011386.4556186",
    "source": "NSRL"
  },
  {
    "CRC32": "8E51A269",
    "FileName": "358.git2-msvstfs.dll",
    "FileSize": "65",
    "MD5": "9E4C165089CBA3653484C3F23F1CBC67",
    "OpSystemCode": "362",
    "ProductCode": "201317",
    "SHA-1": "FFFFFE4C92E3F7282C7502F1734B243FA52326FB",
    "SpecialCode": "",
    "db": "nsrl_modern_rds",
    "insert-timestamp": "1631011386.44553",
    "source": "NSRL"
  }
]

API and HTTP return codes

HTTP return code Description and Interpretation
200 The requested hash is present in at least one database.
404 The requested hash is not present in any database.
400 The supplied hash is incorrectly formatted.

Querying the hashlookup database via DNS

The domain to query is <query>.dns.hashlookup.circl.lu. The query can be info or an MD5 or SHA-1 value.

Information about the hashlookup database

dig +short -t TXT info.dns.hashlookup.circl.lu | jq -r . | jq .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "nsrl-version": "RDS Verion 2.73.1 - July 2021",
  "nsrl-NSRL-items": "165968856",
  "nsrl-Android-items": "33419323",
  "nsrl-iOS-items": "46447082",
  "nsrl-NSRLMfg": "543004",
  "nsrl-NSRLOS": "6414",
  "nsrl-NSRLProd": "333546",
  "hashlookup-version": "0.1"
}

Query a hash

dig +short -t TXT 931606baaa7a2b4ef61198406f8fc3f4.dns.hashlookup.circl.lu | jq -r . | jq .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "CRC32": "13C49389",
  "FileName": "ls",
  "FileSize": "133792",
  "MD5": "931606BAAA7A2B4EF61198406F8FC3F4",
  "OpSystemCode": "362",
  "ProductCode": "217853",
  "SHA-1": "D3A21675A8F19518D8B8F3CEF0F6A21DE1DA6CC7",
  "SpecialCode": ""
}

Sample use-cases

How can I quickly check a set of files in a local directory?

sha1sum * | cut -f1 -d" " | parallel 'curl  -s https://hashlookup.circl.lu/lookup/sha1/{}' | jq .

Negative results (hashes that do not exist in the database) can be excluded with the -f option in curl.

sha1sum * | cut -f1 -d" " | parallel 'curl -f -s https://hashlookup.circl.lu/lookup/sha1/{}' | jq .

Querying hashlookup without online queries

If you do not want to send your lookup queries to CIRCL, you can download and query the hashlookup Bloom filter locally.

A Bloom filter (a compact representation of the dataset) containing all SHA-1 values known to hashlookup is available at https://cra.circl.lu/hashlookup/hashlookup-full.bloom (~700 MB). It uses the format supported by the DCSO bloom library and CLI and is updated monthly.

To use the Bloom filter locally, first install the DCSO bloom CLI, and then run:

find /usr/bin/ -type f -print0 | xargs -0 sha1sum | awk '{ print $1 }'  | tr a-f A-F | bloom c /home/adulau/hashlookup-full.bloom

The Bloom filter does not contain metadata; it contains only the SHA-1 hash values stored in CIRCL hashlookup. You can inspect the Bloom filter file with the bloom CLI:

adulau@kolmogorov ~/hashlookup $ bloom s hashlookup-full.bloom
File:			/home/adulau/hashlookup/hashlookup-full.bloom
Capacity:		296893697
Elements present:	296890922
FP probability:		1.00e-04
Bits:			5691486835
Hash functions:		14

The hashlookup forensic analyser supports the Bloom filter and can also be used locally instead of sending online queries.

python3 bin/hashlookup-analyser.py --bloomfilter /home/adulau/hashlookup/hashlookup-full.bloom --include-stats -d /bin

Libraries and software for using CIRCL hashlookup

Top