I'm pretty sure the distaste comes from his almost 5 ERA over his last 17 starts and his averaging of 5.2 IP per start. He always seemed to be a liability to the bullpen and would just get demolished some games. I think he is capable of being a decent 5th starter... but he did not exhibit that after June of last year.
He had a 4.12 ERA on July 5, then he had 2 terrible starts, then he posted a 4.13 ERA the rest of the way with the team going 5-4 in his starts, that's far from some huge drop off. You can parse the numbers any way you want, but all of his starts counted the same and he wasn't good overall. However prior to last year he was pretty good. His career ERA sits at 3.87, and he's still cheap and on a one year deal. A team's 6th starter is a virtual guarantee to make plenty of starts, and for a 6th starter you could do a lot, lot worse than Mike Fiers. I never understood how Fiers became the rotation's punching bag. By years end his ERA was basically the same as all of the other guys in the rotation save for LMJ, yet Fiers was constantly viewed as the black hole starter that needed to be replaced.
I think it is because he was always letting teams back into the game, especially in the 5th or 6th inning. He definitely seemed like the weak link (until Fister fell of a cliff, but everyone should have seen that coming).
