This paper considers the underlying assumptions related to the use of random assignment in educational research and evaluation; more specifically, the ability of random assignment to create similar comparison groups for the purpose of determining the effectiveness of educational programs. In theory, randomly assigning individuals to comparison groups is considered to be the best method available to maximize the likelihood that groups used in this type of research will be similar; however, in educational research designed to identify proven best practices, random assignment of individuals is rarely possible; other methods including random assignment of intact units and non-random selection techniques are often used. Using a database simulation, this study set out to determine the degree to which various selection methods might be effective at creating comparable groups. Given the complex dynamics of the teaching and learning process and the abundance of potentially confounding variables, it seems likely that comparison groups will always be dissimilar to some degree. While random assignment of individuals performed as expected when controlling for a single extraneous factor, the likelihood that comparison groups created in this manner will differ on one or more potentially confounding variables is extremely likely. Based on the results of this study, random assignment of intact units is not an acceptable alternative to random assignment of individuals. In fact, when using intact units, non-random selection techniques were considerably more effective at controlling for potentially confounding influences than randomly assigning existing classrooms to treatment and control groups.