This paper is concerned with model selection based on penalized maximized log likelihood functions. Its main emphasis is on how these penalties might be chosen in small samples to give good statistical properties. We explore how some of the more successful principles and practices in hypothesis testing can be used to improve the properties of these model selection procedures. This leads to choosing the penalties in order to control probabilities of different models being selected. Various ways this can be achieved using simulation methods are discussed and a computer algorithm is outlined. Some illustrative Monte Carlo simulations are also reported.