One of the residents whose investigation found staff and visitors were being put at risk has called on Martin Esom to step down.