"Fans" is way too general to have any meaning. There are "fans" on both sides of the "pulling the starter" issue to be sure.
To assign one position to "the fans" is unfair and wildly inaccurate. You're just inventing a position to "fans" and then arguing against it.
I'll state MY general belief about starting pitching: "starter loses game early, it's on him. Loses game late, it's on the manager." I strongly believe it's much, much more favorable to remove a starter too early than too late. But that's a general philosophy. Everything needs to be placed in context, and a manager needs to have a "feel" for the situation. Last night, for example, that's not on Baldelli.
People can have more complicated views on pitching than you're giving them credit for.