Thumbnail Image
Item

CRATOR a CRAwler for TOR: Turning Dark Web Pages into Open Source INTelligence

De Pascale,Daniel
; 
Cascavilla,Giuseppe
; 
Tamburri,Damian A.
; 
Van Den Heuvel,Willem Jan
Abstract
Dark web crawling is a complex process that involves specific methodologies and techniques to navigate the Tor network and extract data from hidden services. This study proposes a dark web crawler designed to extract pages handling security protocols, such as CAPTCHAs, efficiently. Our approach uses a combination of seed URL lists, link analysis, and scanning to discover new content. We also incorporate methods for user-agent rotation and proxy usage to maintain anonymity and avoid detection. We evaluate the effectiveness of our crawler using metrics such as coverage, performance, and robustness. Our results demonstrate that our crawler effectively extracts pages handling security protocols while preserving anonymity and avoiding detection. Our proposed dark web crawler can be used for several applications, including threat intelligence, cybersecurity, and online investigations.
Description
Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Date
2024
Journal Title
Journal ISSN
Volume Title
Publisher
Springer Science and Business Media Deutschland GmbH
Collections
Tilburg University Research Output Collection
Show all metadata
Files
Thumbnail Image
crator_a_crawler_for_tor.pdf
Adobe PDF, 1.17 MB
Research Projects
Organizational Units
Journal Issue
Keywords
crawler, Dark Web, Law Enforcement Agency, Open Source Intelligence, TOR
Citation
De Pascale, D, Cascavilla, G, Tamburri, D A & Van Den Heuvel, W J 2024, CRATOR a CRAwler for TOR : Turning Dark Web Pages into Open Source INTelligence. in J Garcia-Alfaro, R Kozik, M Choraś & S Katsikas (eds), Computer Security – ESORICS 2024 . Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14983 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 144-161, 29th European Symposium on Research in Computer Security, ESORICS 2024, Bydgoszcz, Poland, 16/09/24. https://doi.org/10.1007/978-3-031-70890-9_8
URI
https://hdl.handle.net/20.500.14602/25987
License
info:eu-repo/semantics/openAccess
Embedded videos