Hi This has been a really good read for me.
I don't think re-building the CMS from scratch is a good idea. As a coder its always tempting, but ultimately its always a tricky path to navigate and one which is hard to sell to stakeholders be they 3rd party devs, site maintainers etc. That's why I think Andrew's idea of slowly implementing small sections of the framework in "stealth" mode is a really good idea. If I understand correctly that has already started to happen with some of the vendor/ code that is currently in the cms.
What we need to do is get developers enthused about writing for the Joomla project.
The is a massive base of people with varying degrees of competency in writing PHP for Joomla, the project needs to engage those developers and show them that they have a place within the project and produce strong guidelines on what to write when. With that in mind I like Andrew's concept of sprints on certain areas of code. You can have a couple of architects draw up an iterative process that advances the CMS/FW and then launch the code monkeys along that path.
This is massively more productive that the 'write it and perhaps we'll use it' attitude that is currently used, and which is a disaster. As a dev I want to know that what I'm writing is relevant and that I'm writing code in a manner that will be accepted into the code base.
If you mentor new devs supplying them with the best practices to ensure their code is correct (writing tests, code style etc). The iterative process will produce a positive feedback loop where by devs see the fruits of their labour in a short time, are encourage and enthused by the results and then want to repeat the process, the complete opposite of the current negative feedback loop when you bust a gut for a month on something you personally deem important only to have someone block it based on some arbitrary requirement you weren't even aware existed before you started.
This is also why I'm against say "lets start again from scratch". It reduces confidence in the merit of incrementally contributing to the existing code, and leaves those dev's not capable or willing to work out an entire new architecture watching from the sidelines.
95% of those developers will be coming from the CMS, and that is where their interest lies. When I do a custom project realistically I am not going to use the Joomla framework, I'd sooner work on Express or Laraval, or pick individual composer projects few of which would be those available in the framework. So for me the Joomla Framework is only relevant IF its directly feeding into the CMS. I'd love to see that change and for the framework to become my defacto choice, but it will only ever get there if there are a lot more people contributing to it, and to get those contributors we need to set up a short positive feedback loop in the contribution process.
With regards to Andrew's idea of version'd classes. To my mind there are definitely certain classes which you would not want multiple versions of. JSession, JApplication etc. Basically any class which is required to be initiated before the call stack arrives at a component/plugin entry point should be a known quantity for a 3rd party developer. However MVC etc could have different versions, indeed we already have that in the fact that JModel and JModelLegacy are already in the CMS. I'm currently refactoring Fabrik to use namespaces and the new MCV classes. What I hadn't really appreciated before starting was that it really does allow you to easily swap out class definitions. So you can start with :
use \JRegistry as JRegistry
and at a later stage I imagine I could replace that with
use \vendor\Joomla\Registry\Registry2 as JRegistry, and as long as the two classes implement the same interface all is going to work as before.
So that could enable rapid iteration within the framework and deployment to the CMS (at least for those modules which are not umbrella requirements for the CMS) with little backwards compatibilty issues.
perhaps some of the short term sprints we could look at would be related to namespacing the current core components? I've always learnt how to code in Joomla by taking those components as a reference for how things should be done. If 3rd party dev's start seeing that type of change then they are naturally going to start to mimic that within their own code. This would have the double impact of helping them update their PHP practices and also increase their potential usefulness to the project