I'm interested in possible solutions to this problem, but it's far harder than it looks.
Exactly.
Limited extension points where we accept runtime errors and confine the extension points can be OK; e.g. SOME STATEMENT ... WITH (THINGY, OTHER_THINGY) where we allow any series of identifier|keyword|literal|bareword, accumulate it and pass it as a List of Node to something else to deal with. Bison can cater to that and similar structures where the boundaries of the generic/extensible region can be clearly defined and limited.
The other area where there's room for extension without throwing out the whole thing and rebuilding is handling of new top-level statements. We can probably dispatch the statement text to a sub-parser provided by an extension that registers interest in that statement name when we attempt to parse it and fail. Even then I'm pretty sure it won't be possible to do so while still allowing multi-statements. I wish we didn't support multi-statements, but we're fairly stuck with them.
