I created a variable of type counter. The scope is global and when out of values it's set to stop the test. The starting value is 0 and the max value is how many errors we want to stop the test at. Increment value is 1.
In my catches, I increment this counter. Once the max value is reached the test should be stopped.
This seems to work in my smoke tests but I've yet to have real world test of it. I'll report back once I do.