The colour for each large cube is computed by averaging the colours in the small cubes, with some weighting applied according to how many faces of the small cubes were actually visible. It doesn’t check from which directions they were visible though it might be possible to add this.

There’s a bit more work needed to handle thin structures effectively and also to reduce the amount of blurring which takes place, but I think these issues won’t be visible in the distance anyway. I need to increase the volume size and do some proper tests here.

Also a suggestion on the colors: Instead of trying to average all of the neighboring colors, why not only average the nearby surface colors? That way you wouldn’t have those weird artefacts near the boundary between the ground and the water. You might be able to get even better results if you took into account per-face colors by only averaging over faces which are oriented similarly.