DRKSpiderJava
DRKSpider is an open source website crawler (sitemap generator) and link checker.
Features: Generates a website tree matching site's navigation hierarchy. Sitemap exporter for search engine optimization: generate sitemaps for Google, Yahoo and other engines. Detailed information about every link found. Several scan limit options. Site scan results window. Export command allows to export to CSV format for further processing. Preferences command in the File menu for saving DRKSpider settings. Exclude word list for URL excluding. Proxy server support. Internet system cache control. Referrer information field control. Binary install distribution (NSIS Setup) HTML Validator by World Wide Web Consortium added in the link properties dialog. SWFRIP add-in for parsing SWF content (SWFRIP source code). Web lookup for any tree item. Generates sitemap file for search engine optimization (SEO).
