I would never wear a down mid or outer layer when cross country skiing. For Lift Serve Alpine, I hate freezing my kester off on the slow chairlift up where synthetics often fail me.Until I sweat into the down and it becomes useless as an insulator. I love down for low-intensity activities -- digging a snow pit, hanging out to/from the lodge, après, transitions. But down doesn't have much of a place in my systems where I'm generating a good deal of heat.
YMMV. But I've never found down that breathes like the Polartec Grid fleece in the R1 or anything like the Nano Air jacket.