Senior positions in football remain a "bastion of white male domination", the head of the game's anti-discrimination group Kick It Out has warned.
Lord Herman Ouseley, in comments to be delivered to the 20th anniversary dinner of Kick It Out on Tuesday, says there is little interest among clubs in introducing equality or diversity in board rooms.
Ouseley's speech comes with interest focused as a Premier League committee meets to discuss sexist emails sent by chief executive Richard Scudamore.
Ouseley's speech will say: "In effect and in reality the top of the football pyramid is still a bastion of white male domination and finds difficulty in having to face up to the challenge of change which, if tackled seriously, would mean giving up some space to allow the participation of women, black, Asian and disabled people as equals within its decision-making structures and as employees at all levels.
"Some in leadership positions are trying to move policies and practices forward positively but progress, while notable and helpful, remains painfully slow.
"More often than not there is little or no interest whatsoever in the boardrooms or the senior management teams for any high-profile equality and diversity activity, and most clubs and some of the over-arching bodies would be much more comfortable for it to go away as an issue.
"In fact, many leaders in football hold the view that there is no longer a race or discrimination problem and it is time for Kick It Out to pack its bags and move on."
Meanwhile, the Premier League has defended its working environment after the woman who blew the whistle on the sexist emails said she was "humiliated" and "demeaned".
Speaking on Good Morning Britain, Rani Abraham said: "They were sexist - very, very degrading to women."
Asked what she thought should happen to Scudamore, she said: "I feel he should resign" - and if he did not, then "he should be asked to leave".
The Football Association's independent board member Heather Rabbatts, who will chair a meeting of its inclusion advisory board to discuss the case on Tuesday, has said Scudamore should consider his position in light of "growing evidence of a closed culture of sexism" at the Premier League.
A Premier League statement said a review of the matter was underway ahead of the audit and remuneration committee meeting.
The league statement said: "We do not recognise this characterisation of the working environment at the Premier League, nor do we believe that it can be supported by the facts.
"The chief executive has already apologised for any offence caused and a proper review of all the evidence is now underway within the Premier League's established and rigorous procedures.
"This process is not yet concluded and it is therefore not possible to offer comments in detail at this stage. However we will make a further statement in due course
"The Premier League continues to be fully committed to treating all staff fairly and on merit, regardless of gender."
The emails referred to women in a derogatory terms, contained sexual innuendoes, and made jokes about "female irrationality".
A Downing Street spokesman said Prime Minister David Cameron shared the view expressed by sports minister Helen Grant that Scudamore's comments were "unacceptable" but that it was for the Premier League to decide on his future.
The Prime Minister's official spokesman said: "Helen Grant ... was right to say what she did, I have clearly associated the Prime Minister directly with those remarks.
"But in terms of staffing decisions at the Premier League, it's for the Premier League."
The Prime Minister admitted he had not personally read the messages sent by Mr Scudamore but said everyone had to treat others with respect.
Asked if a minister would survive in their job if they admitted sexist behaviour, he told BBC Radio 5 Live: "I don't think they would.
"I have to be careful what I say because I haven't seen these specific emails but...we have to set and keep high standards in politics.
"I have tried to enforce that in my own party."
He added: "I haven't actually seen the emails myself but obviously people should treat everybody else with respect."