He knew you loved him, no matter how many times you did or didn't get to say it. You have happy memories of him, and that's what's important.

I'm so sorry, baby. You know I'm always here. (and can provide bad jokes too if you need them.)