For the record, Gil, my response was pretty much the same as yours-- I told 'em that, first, if it's good VFR where I'm at, it wouldn't occur to me to check the weather at a non-towered airport six or seven miles away. But even if I did for some reason, and found out it was zero-zero there, it wouldn't prompt me to request any sort of ATC clearance, so long as I was confident I could remain VFR after departure.
When I got to work this morning I asked my boss. He was positive that you need a Special VFR. He based it on his experience at his father's private airport a few miles northwest of Rhinelander, WI, which is Class E.
Of course... he is basing it on what he was told and what is commonly done there, not on any FAR interpretation.