In theory this could work since as you said, there is no possible 'real' calculation that could produce that value.
In practice, unless the hardware is specifically designed to produce that value for a modulo-by-zero operation, then every modulo operation by any value that might be zero would require the compiler to emit code to specifically check if the divisor is zero and produce that value if so, to override the default behavior that the CPU divide instruction would otherwise have done (such as crash the program). This could negatively impact performance.
I doubt this would provide much real-world benefit because in most cases that the sentinel value would be useful:
if (x % y == UINT_MAX) { // y is 0}You could have just checked if y is 0 yourself:
if (y == 0) { // ...}But if you are simply making a compiler yourself, for C or C++, you are free to implement your proposed behavior. Modulo by zero is undefined, and permissible undefined behavior includes the behavior you propose.