I've added tests to prove this and created a PR:
https://github.com/joomla-framework/di/pull/10
Sven, I think the issue you are experiencing is that the callable you are passing ( [$this, 'getResponse'] ) returns a different object. If you register your key with a callback, that exact callback is used when fetching the item from the container, not the value returned by the callback. So you are getting, each time you get it from the container, the value returned by your $this->getResponse() method (as in, that method is called every time you request $c->get('jresponse');
If you want to use the value, simply register it like this:
$c->set('MyApp\\Interfaces\\App\\AppResponseInterface', $this->getResponse(), true, true)
->alias('jresponse', 'MyApp\\Interfaces\\App\AppResponseInterface');