I'd ring ... tell him you can't make it and you'll make another time with him later ... and just don't bother making that call. You don't have to see him if you don't want to but it's entirely your choice.