Let's say your income before taxes and contributions was $1,111, and your tax rate is 10%. Ordinarily, you would pay $111.10 in taxes, and have $999.90 left over, which would be eligible for maaser. So you would need to pay $99.99 in maaser.
But if you paid the $99.99 in maaser, you would then be able to deduct $99.99 from your taxable income, making your taxes only $101.101 (assuming you had stayed in the same bracket). This means you would have $1009.899 to pay maaser on, and would owe $100.9899 in maaser.
So, in addition to the possible almost $10 deduction in your net maaser contribution from getting a tax break (as suggested in this question), could there possibly be a further "deduction" in the sense that you now have more income to pay maaser on? And what about after you pay that maaser and get another tax break? How far does it go?
(I'm more interested in whether this is addressed in halacha than whether it is mathematically or practically feasible, since I'm pretty sure it is both of those.)
Related: Should tax deductions be deducted from total maaser giving?
Is maaser calculated pre-tax or after-tax?
Does Ma'aser count if you have an ulterior motive?
I don't know whether my answer describes real situation — it is an idealized model. Firstly, you get income. Based on this, let us define:
Of course,
Therefore, you need to calculate solution to equation
.
The analytical solution doesn't have to necessarily exist. You can use any iterative method of root calculation to get the numerical result for , for instance Newton's method. When you finally have it, you pay Ma'aser and of tax. This equation is analytically solvable in special cases. For instance, let us assume that
.
It is a linear progression --- if you pay 17% of tax, . In such case
.
Solution to this equation for is
.
Analogously,
.
The answer (how much should you pay) in case of linear progression is thus
of state tax,
of Ma'aser, where
EDIT: In answer to OP question in comments --- checking the convergence that is --- let us generalize variables to have explicit time dependence. Thus,
So that in one particular (—th) month one owes Ma'aser and pays of tax (in linear model, I assume constant as well). We can write equations analogous to the previous ones, but now what matters in —th month is how much one paid in previous one:
and
.
Reindexing ( and ) the first equation and putting second into first yields
.
This iterative equation can be solved the following way: we split into constant and variable part:
.
The choice is of course not unique. We want to choose such that equation for has simple form. Combining the two previous equations yields
The equation for is really simple when
,
i.e., the constant terms are canceling each other. Then,
,
and since , when . It means that in long time , which is
.
This is exactly the answer for calculated in previous section. Therefore, regardless of initial conditions, if one only follows the rules that tax payed in —th month determines amount of Ma'aser in —th month (and vice versa), he/she eventually reaches the proper (`equilibrium') amounts of Ma'aser and tax.