Octal notation is simply a method of representing values, because binary is inefficient for humans. A computer would not actually use octal internally, because computers know only 0 and 1. Octal would be used by a person to make the code more readable (hex is preferred though), in the case of a transponder, the number on the front of the box is actually in octal representation, not decimal. If the dials were marked with what the transponder is actually using, you would see groups of 1s and 0s, not 0-7.
Take a transponder code of 4076. In (standard) binary, this would be 100000111110, and this is what the transponder understands. The decimal equivilant would be 2110.
The VFR transponder code of 1200 is actually 640 in decimal notation. A transponder could be set using decimal notation (valid codes would then range from 0 to 4095), but it would make the internal mechanism more complicated since it would have to convert from decimal to binary, which is more involved than from octal to binary.
The alternative encoding that I provided is Gray code. The advantage of Gray code is that, when increasing the value by one, only one digit ever changes at a time. This reduces the error potential and makes the system more efficient. Like I said, I've never researched transponders, so I don't know if they actually use Gray code or not, I just guess that they do.
Actually, to get super technical, computers don't even know 1 and 0, they only know high voltage (commonly referred to as 1), and low voltage (commonly referred to as 0).