It sees to me (as a lawyer) that lurking behind the trolley problem is the duty to rescue. The law generally does not recognize such a duty. The law does not compel you to throw the switch, because there is no end to an abstract duty to rescue. Someone always needs rescuing, so why am I not rescuing them?
I think we sense this problem in our bones so to speak, i.e., we are programmed to regard inaction with respect to strangers' danger as more likely than action to be in our own interest. So long as the people on the track have no special meaning to us, our instinct is to let the cookie crumble.
We may try to rationalize our decision, but any such effort seems futile. We're not going to feel good about the decision either way, but one reason may be that we are not being asked abstractly to choose who dies. We are being asked to rescue the one(s) saved, and we rebel at that demand.