Hundreds of people who signed up to volunteer at the Tokyo Olympic Games have resigned in protest against derogatory remarks about women made by the Games’ organising committee president, Yoshiro Mori.

Mori is under increasing pressure to resign over his comments, which have prompted widespread anger and frustrated Tokyo’s attempts to convince the world it can host the Games safely amid a global pandemic.

The organising committee will hold an emergency meeting this week where Mori could face criticism, though his resignation is not up for discussion.

The 83-year-old former prime minister sparked anger last week when he complained that meetings tended to drag on because “competitive” women in attendance “talked too much”.

Recalling his time as chairman of the Japan Rugby Football Union, Mori told an online meeting of the Japan Olympic Committee: “Women have a strong sense of rivalry. If one raises her hand to speak, all the others feel the need to speak, too. Everyone ends up saying something.”

He later apologised and the International Olympic Committee said it considered the matter closed, but anger over what many say is a widely held view of women in Japan has intensified.

More than 400 of the 80,000 people who signed up to volunteer at the Tokyo Games have resigned, according to local media, while an online petition calling for action to be taken against Mori had attracted more than 140,000 signatures by Monday. Olympic organisers said they had received more than 5,500 complaints.

“We are taking this very seriously,” the Olympic minister, Seiko Hashimoto, said on Tuesday morning when asked about the resignation of the volunteers.

Daichi Oyama, 28, who withdrew as a volunteer over coronavirus concerns, said Mori’s comments were proving “very embarrassing for Japan”. He added: “If every time he says something things get worse, he should quit.”

A poll conducted over the weekend found that almost 60% of people thought Mori was “unfit” to lead the organising committee, with only 6.8% saying he was qualified for the role.

The controversy has added to the myriad problems facing the IOC and Japanese organisers as speculation mounts over the future of this summer’s Games, which have already been delayed by a year due to the pandemic.

In a poll conducted shortly after the controversy erupted, 82% said they should be cancelled or postponed again, with 14.5% saying they should go ahead as planned on 23 July.

The tennis player Naomi Osaka said Mori’s comments were “ignorant” but did not call on him to step down. “I did look at the comments. I didn’t think they were good,” she said before the start of the Australian Open. “I think if you’re in a position like that, you really should think before you say anything. I don’t know in what situation he said those things, but I think it’s really uninformed and a bit ignorant.”

Yuko Inazawa, a member of the JRFU when Mori was president, said she believed the “talkative women” remarks had been directed at her.

“I think conferences dragged on as I was asking questions from my standpoint as an amateur,” Inazawa, who became the first female board member of the JRFU in 2013, told Kyodo. “But that is absolutely not the same thing as saying women make conferences drag on.”

It is possible that Mori will weather the storm given the support he has among influential figures in the ruling Liberal Democratic party and the IOC. He was a veteran LDP lawmaker when he became prime minister in 2000, but his approval ratings plummeted to 7% after a series of blunders and scandals involving cabinet ministers.

One senior LDP official said sacking Mori would mean losing the wealth of experience he has acquired since becoming head of the organising committee in 2014. “He is a walking dictionary of the Olympics and he knows everything about the last eight years,” the official said.

There are concerns that his resignation could hamper preparations for the Games at a crucial time, with less than six months to go before the opening ceremony on 23 July.

“We had told him beforehand, ‘You can’t say anything about resigning’,” an organising committee source told Reuters, referring to Mori’s testy encounter with reporters last week. “Nothing good will come from replacing him,” the source added.

Kaori Yamaguchi, a judo champion and member of the JOC board, said many believed Mori was “untouchable”.

“I think it’s much easier for the IOC, it’s faster to work with a dictator,” said Yamaguchi, who was present at the JOC meeting where Mori made the remarks. “If Mori says OK, it means OK.”

Dick Pound, a veteran IOC member who has cast doubt on Tokyo 2020’s future, noted that Mori had apologised and added: “I think we should all get on with our lives.”

But organising committee employees voiced anger that Mori had not spoken to them directly. One woman who for the committee, who asked not to be named, said: “Mori’s remarks [are] insulting to the women who are working to make the Games a success, and I am confident that the female staff do not accept his apology.”