Shouldn’t the dictatorship of the proletariat have been disbanded after the revolution was successful?

This question doesn’t even make sense. The dictatorship of the proletariat doesn’t start until the revolution is successful. When the revolution succeeds it replaces the pre-existing dictatorship of the bourgeoisie with the new dictatorship of the proletariat. These are definitional. The fact that you asked the question means you are missing some critical pieces of information that will make it quite literally impossible for you to analyze anything about history, communist theory, revolutionary politics, and left organizing.

Why were the people not free to self organize into communes of their own design that best reflects their values?

You are describing libertarianism. Under current global conditions, people self-organizing into collectives creates warlords which reproduces feudalism which reproduces capitalism. People self-organizing into communes that best reflect their values is quite literally how we got to where we are today. Prehistoric human communities formed around shared values and splintered along values misalignment. They formed and disbanded and reformed. And eventually the technologies for hoarding became available (generally agriculture) and then conquest became a viable strategy for survival. Those conditions haven’t really changed yet. The point of a socialist transition to communism is actually to collectively organizing human activity to bring about the conditions whereby conquest is no longer a viable strategy for survival. That requires significant reorganization of production and distribution. So far, we’ve seen it takes longer than a century to pull that off.