Novel UI systems have a lot of the same problems as other novel pieces of media.
Novel UI systems have a lot of the same problems as other novel pieces of media. The existing widgets, even when they are objectively terrible, represent a familiar visual language to users — in the same way that samey blockbusters combine familiar franchises with familiar special effects, familiar story structures, familiar characterization cliches, and familiar forms of cinematography for an ultimately conservative work that is unlikely to lose money, and in the same way that bubblegum pop combines familiar artists with familiar rhythms and familiar harmonies to produce minor variations on a familiar style that is unlikely to be rejected for being too extreme. When a UI is big business and lots of money rides on it being usable for a wide audience, that UI will be conservative, and in the rare exceptions to that rule often we end up with something universally reviled (Windows 8, the MS Word ‘ribbon’ interface, Microsoft Bob, the Macintosh finder’s ‘galaxy mode’ from the early 90s, Google Wave, literally every UI change on Facebook or Tumblr). Because UI is a form of creative media that people have to use every day — an experimental UI for an important and widespread product is like having Steve Reich and Daphne Oram compose the soundtrack for a large supermarket chain.
However, since the situation is comparable to more traditional forms of media, we can borrow a bit of media’s solution. The solution to Hollywood sequelitis and horrible blockbusters is the slow and careful importing of ideas, techniques, and talents from experimental film, just as visually striking import giallo and the rise of auteur directors in the 60s and 70s revived the 50s slump in creative moviemaking (along with big names like Hitchcock grabbing talent and ideas from other fields like experimental animation — compare North By Northwest’s cinematography with that of Vertigo), and just as the constrained compositional environment of punk revived stagnant arena-rock (itself descended from blues- and folk-derived attempts to use experimental electronic hardware to inject vitality into overly commercial 50s rock/r&b, itself an attempt to bring in new influences to revive a stagnant crooner-centric ecosystem, etc., going back long before Mozart). And, we certainly used to do this: PARC was so influential because they were an isolated group of geniuses quickly iterating on UI design. To some extent, we still do this: video games often have unusual UI ideas, most of which don’t work, and Alan Kay is still playing with UI innovation with Squeak. The thing is, for the most part, fringe UI designs haven’t gotten pulled into the mainstream since the early 80s, when “GUI” became synonymous with “Alto clone UI” and anything that didn’t use a mouse began to be considered as not ultimately related to UI.
Some UI experimentation continues, here and there. The biggest example of UI experimentation being pulled into the mainstream in the past five years or so is, shockingly, in text-based UIs, wherein conversational user interfaces based around messaging systems have begun approximating the kinds of features and nuances that command line interfaces in the UNIX world have shipped with since the late 70s. SIRI is a cross between Eliza and Autocorrect, trying blindly to approximate a cross between ZSH and Google while remaining fixated upon being accessible to new users, and it’s an interesting enough experiment; special-purpose slack and twitter bots are more innovative because the cost of failure is low.
My recommendation: release smart people in the UX field into a situation where they can iterate quickly on interesting projects of their choosing, none of which will ever be marketed. Then, get a couple canny con-artist types like Steve Jobs to drop in, take a look at the end result of five or ten years of iteration, and distribute a simplified version to the mainstream.