Usuthu are bottom of the Premiership table and remain the only winless side after seven games of the season.
Rosslee said: “A coward resigns”, when asked whether he had considered quitting his post.
“You’re asking me to go into battle and then when the firing starts I must run. Of course there’s pressure and that’s because I’m in the public eye,” Rosslee said on Tuesday.
He believes the team can turn things around and said he would not cave into the mounting pressure.
“If I resign, what does it say about me that when the goings get tough, I must run away? You can’t do that.”
AmaZulu have lost four out of their seven games so far in the season, while their other three fixtures ended in draws.
Usuthu face Bloemfontein Celtic in the Telkom Knockout first round on Saturday at the Princess Magogo Stadium.
Rosslee said the tumultuous start is a test of mettle for the entire team.
“In a time of adversity your true character and the test of it is shown. Whether it’s now or later, this is the time when the character is shown.
The 44-year-old insisted he did not fear being sacked either, despite his bosses openly stating at the start of the year that a top four league finish and cup win were among their targets.
“What they (the owners) have reiterated is that they believe I’ve done the job over the past two years. They’ve said that they made the mistake in the past of changing coaches and this time they want a bit of continuity.”
The former Ajax Cape Town coach felt it was difficult for him alone to shoulder all the blame.
“At the end of the day, I’m the coach, but I don’t make the mistakes on the field, the players do. But none of them get the boot.”