Most prior investigations of visual working memory (VWM) presented the to-be-remembered items simultaneously in a static configuration (e.g., Luck & Vogel, 1997). However, in everyday situations, such as driving on a busy multilane highway, items (e.g., cars) are presented sequentially and must be retained to support later actions (e.g., knowing if it’s safe to change lanes). In a simultaneous presentation, the relative positions of items are apparent but for sequential presentation, relative positions must be inferred in relation to the background structure (e.g., highway lane markings). To examine sequential encoding in VWM, we developed a novel task in which dots were presented slowly, one at a time, with each dot appearing in one of six boxes (Experiment 1), or in invisible boxes within a visible encompassing outer frame (Experiment 2). Experiment 1 found strong recency effects for judgments of color at the end of the sequence but not for the location of dots. In contrast, without dividing lines, Experiment 2 found strong recency effects for both color and location judgments. These results held true for accuracy, reaction time, and an integrated measure of speed and accuracy. We hypothesize that background structure allows the updating of VWM, slotting each new item into that structure to provide a new configuration that retains both old and new items, whereas in the absence of structure, VWM suffers from severe retroactive interference.