By far the most challenging, and in the same breath most rewarding has to be the Milkyway. The scouting, timing the moon phase, planning the composition, again the weather, setting up in complete darkness, tripping over everything in front of me and sacrificing my arms and legs to save my camera. Light pollution, clouds, other photographers and their flashlights, ...  trying to focus on tiny little dot's in the view finder for manual focus.  And oddly enough it is worth everything I go through when that image is captured.  It is amazing what the eye can not see.