Established evaluation methodologies for information retrieval are not well suited to the task of comparing systems in many real settings. We have implemented a comparison tool which support this task. By offering a working search interface the tool permits study of real information needs as they occur, uses the documents actually available at the time of the search, and records judgements taking into account the instantaneous needs of the searcher. We report experiments, to validate the tool and explore potential biases, and ongoing and future applications.