The dilapidated state of a heritage-listed building has frustrated both the developer and neighbours, leading to accusations of “demolition by neglect”.