Although ubiquitous, color pickers have remained largely unchanged for 25 years. Based on contextual interviews with artists and designers, we created the Color Portraits design space to characterize five key color manipulation activities: sampling and tweaking individual colors, manipulating color relationships, combining colors with other elements, revisiting previous color choices, and revealing a design process through color. We found similar color manipulation requirements with scientists and engineers. We designed novel color interaction tools inspired by the design space, and used them as probes to identify specific design requirements, including: interactive palettes for sampling colors and exploring relationships; color composites for blending and decomposing colors with other elements; interactive histories to enable reuse of previous color choices; and providing color as a way to reveal underlying processes. We argue that color tools should allow users to interact with colors, not just pick or sample them.