The oldest known document in the world is the Rgveda, which is otherwise known simply as the Vedas (plural because it is split into four parts). It was orally passed down until around 5 BC, when writing either started or became more commonplace in India. It came from the Vedic people, who lived between the Ganga and Sindhu rivers. Unlike other civilizations, the Vedas are the only remaining evidence left from this society -- so far no archeological discoveries have been made that shed more light on them. This means that there are no brick altars or similar remains that testify to their level of science and technology.
The Vedas are not mathematical texts; they are merely hymns to the
Vedic gods. However, the word Veda means "knowledge", and when
analyzed closely it actually contains many mathematical references,
especially in the section on Jyotisa, or "the constellations".
Unfortunately, almost all of these references are implied, so much of
the interpretation is largely guesswork. Another reason that the
Vedas are hard to interpret is that because it was an oral document,
there are no symbols for numbers or operations -- only words. It is
highly likely, however, that they did use symbols, because without
them math becomes very tedious. For example, consider doing a
multiplication problem using "four thousand six hundred and
thirty-seven times two hundred and eighty-eight." You would most
likely convert the words to symbols, do the math on a piece of paper,
and then probably only take the time to convert the answer back into
words.
And what makes it even harder is that the Vedas were written in verse1, probably for ease in memorization (because it was passed down orally). This means that not only were ancient mathematicians poets as well, but more importantly the math they were working with had to be written to fit in verse. Most people consider it hard to do long division using symbols and well worked out methods. Imagine trying to complete this process with words and in poetry! And then, consider the difficulty of decoding your work 4000 years later!
First of all, a scholar has to determine the name of every number, i.e. "three" or "forty-seven". But besides this, there are words meaning "some", "many", "both", "few", etc., and ordinal numbers ("first", "second", "third", etc.) These also have to be deciphered to even begin to get into the math. And yet another oddity of the Sanskrit language involves what happens with compound numbers, numbers with more than one "digit" (like "thirty-four"). In normal Sanskrit, compound words (like "servant of the king") came from left to right in order of prevalence (so our example would be "king-servant"; "servant-king" would mean a servant who was treated well). However, compound numbers are written the opposite way, with the higher digits on the right. (Our number 529 would be written "nine-two-five".) It is always like this, and there is even a rule included: ankanam vamato gatih, which literally means "the understanding of the numbers in the reverse way."2
And then, the whole thing is in the form of stories and myths,
which have to be closely analyzed for mathematical content. A good
example of a story from which we can extract mathematics is one about
a man named Manu.3 Manu had ten wives, who had one, two, three,
four, etc. sons each (the first wife had one son, the second wife
two, etc.). The one son allied with the nine sons, and the two sons
allied with the eight, and so on until the five sons were left by
themselves. They asked Manu for help, and so he gave them each a
samidh or "oblation-stick". The five sons then used these sticks to
defeat all of the other sons.
On the surface, this is just a silly fable, but it shows several things about Vedic mathematics. Because the ten sons did not ally with anyone, and the nine did with the one, eight with two, and so on, the mathematicians must have been thinking that nine plus one, and eight plus two equal ten. This obviously shows that they practiced addition, and it also implies that they used a base 10, or decimal, system. For the second part of the story, the authors probably added the tens up to find that there were 50 allied sons. When the five remaining sons asked their father for help, it is likely that he gave them just enough mathematical power to defeat the others. This would mean that each stick equaled the strength of 10 men, for a total of 50. With the five sons added to that, they were able to defeat the 50. (Or maybe the father gave them 50 men worth of sticks, thinking it would be an equal battle, but not realizing that the five sons would throw off the balance.) But this 50 business implies both multiplication and division as well, because there were five groups of ten sons allied, or five times ten. Then, when the father went to decide the power of the sticks, he would have had to divide that 50 by five, to split the power equally among the five sons.4
Going even further, the story can be shown to symbolize the idea of positional notation -- the idea of place values in numerals. (For an example of positional notation, 218 is the same as 200 + 10 + 8, or 2 x 102 plus 1 x 101 plus 8 x 100. In summary, the order of numerals tells how big the numbers are.) The "oblation-sticks" are obviously thought of as very powerful, just as 10 might be thought of as more "powerful" than a lowly 1. So when the 5 "lowly" sons were "added" to the 5 "powerful" sticks, this could have symbolized the 50 and 5 making 55, which is a bigger number (and therefore more powerful) than 50. This view of things also gives further evidence to the fact that they used base 10.
So this simple story shows examples of addition, multiplication, division, base 10, and even positional notation. The Vedas are full of these stories, and many more examples are given throughout of all these concepts, along with subtraction, fractions, and squares.5 There are even instances of arithmetic and geometric sequences, which are series of numbers that increase by adding or multiplying a certain number (arithmetic sequence: 2 (+3 =) 5, 8, 11, 14, ... ; geometric sequence: 2 ( x 3 =) 6, 18, 54, ...).
The Vedas give us many examples of arithmetic in ancient India, but to find out about the geometry of the time we turn to a collection of Hindu religious documents called the sulva-sutras.
2 Pandit, M. D. Mathematics as Known to the Vedic Samhitas. Dehli, India: Sri Satguru Publications, 1993, p 153. [bibliography entry]
3 From Maitrayani Samhita (part of the Vedas), section 1.5.8, trans. M. D. Pandit. [bibliography entry]
4 Most of the ideas in this paragraph come from Pandit, p 102.
5 See Pandit for more details.