system of numeration

(noun) numeration system

number system
number representation system
mathematical notation (generic term)