Consider for a moment that many colonial householders were lucky to have plaster walls at all. Then thank your stars and garters that today, paints and palettes that replicate or simulate early colors are widely available for different authentic effects.
Well into the late 1700s, most of the colors that went into paint in America were imported here from England as blocks of pigment, which were then mixed with a binding medium like linseed oil. Since both pigments and binders were scarce and expensive, especially in rural locales, paint was applied sparingly: on door and window surrounds, or wainscots (sometimes only as dots or commas of color, as they are most famously at the Peter Wentz farmstead in Pennsylvania). Traveling painters sought out and ground colors from such alternative sources as local earth pigments, plants like indigo, and even dried animal blood.
For a binder, many of these itinerant painters and householders turned to something produced on site: buttermilk. Mixed with the available pigments, milk paint produced a palette of crayon-soft colors in shades from warm to cool: pale pink, ochre, dark red, red browns, cane yellows, and occasionally a copper green. All were country approximations of the high-fashion colors imported to the wealthy in cities: Prussian blue, red lead, vermilion, and verdigris.
Only in the early 19th century, as paint became more widely available, did white become the signature color of the Greek Revival.