One of the goals of humanoid research is the development of a humanoid capable of performing useful tasks in unknown or unpredictable environments. To address the complexities of this task, the robot must continually accumulate and utilize new control and perceptual knowledge. In this paper, we present a control framework for accomplishing this. Robot control policies can be learned at different levels of abstraction. We show how task-relevant perceptual features can be discovered that make better control policies possible. We also explore how trajectories of closed-loop policies can provide uniquely relevant state information. The approach presented in this paper is illustrated with several case studies on actual robot systems.