Diamond’s father reveals he’s never met his grandchildren