That's effectively making the blocksize limit completely uncapped and only preventing spikes, and even in the case of spikes it doesn't differentiate between 'real' traffic and low value spam attacks. It suffers from the same fundamental problems as bitcoin unlimited: There are in the end no transaction fees, and inevitably some miners will want to impose some cap on block size for practical purposes, resulting in a fork.

Difficulty adjustment works because there's a clear goal of having a certain rate of making new blocks. Without a target to attempt automatic adjustment makes no sense.