Invisible Web


Def.: everything a search engine does not see is invisible
  1. non linked web servers and pages

  2. "dangerous" pages/URLs (?URLs: http://...?... - crawler traps)

  3. database content (accessible by textboxes etc. only)


  1. a) servers: detectable by domain registry

    b) pages: important amount detectable by path-crawling

  2. "dangerous"/dynamic pages: robust crawler software

  3. databases (textbox only etc.):

    a) theoretically by automated (dictionary) input

    b) practically by meta searching the databases


How big is all that invisible web??


start         (C) W.Sander-Beuermann, University of Hannover, RRZN, SearchEngineLab