Is there a more elegant way to handle determing fail status?
 

Ok, so I have a diagnostic tool, written by someone else. That tool
runs a series of small tests defined by the user and can simplified
summary output that can be one of the following:

FAILED_CRITICAL
FAILED_HIGH
FAILED_MEDIUM
FAILED_LOW
PASSED

I also have a wrapper script I wrote to run these tests, summarize the
results of all tests aggregated and then fail based on a particular
fail level.

The idea is that if I run 3 tests with the diagnostic tool and it
tells me the following:

testA: PASSED
testB: FAILED_MEDIUM
testC: PASSED

AND I told the wrapper to only fail on HIGH or above, the wrapper will
tell me that I had a medium failure, but the wrapper will still exit
with a 0 (success)

if I get the same results as above, but tell the wrapper to fail on
LOW, then it will tell me I had that medium failure, but the wrapper
will exit with a 1 (failure).

The problem is that my exit determination looks like this:

if fail_priority == fail_levels['FAILED_CRITICAL']:
if critical_fails:
return 1
if fail_priority == fail_levels['FAILED_HIGH']:
if critical_fails or high_fails:
return 1
if fail_priority == fail_levels['FAILED_MEDIUM']:
if critical_fails or high_fails or medium_fails:
return 1
if fail_priority == fail_levels['FAILED_LOW']:
if critical_fails or high_fails or medium_fails or low_fails:
return 1

return 0

So, to explain the above... the fail level can be set by the user when
running the wrapper using -f (or it defaults to 'high')
the wrapper assigns a number to each level using this:

# Set correct fail level
args.fail_level = 'FAILED_%s' % args.fail_level.upper()

# Get our failure priority and create the priority values
fail_levels = {'FAILED_CRITICAL':4,
'FAILED_HIGH':3,
'FAILED_MEDIUM':2,
'FAILED_LOW':1}
fail_priority = fail_levels[args.fail_level]

the variables critical_fails, high_fails, medium_fails, low_fails are
all counters that are etiher None, or the number of tests that were
failed.

So using this output from the diagnostic tool:

testA: PASSED
testB: FAILED_HIGH
testC: PASSED
testD: FAILED_MEDIUM
testE: PASSED

critical_fails would be None
high_fails would be 1
medium_fails would be 1
low_fails would be None.

The exit code determination above works, but it just feels inelegant.
It feels like there's a better way of implementing that, but I can't
come up with one that still honors the fail level properly (e.g. other
solutions will fail on medium, but won't fail properly on medium OR
higher).

I can provide the full script if necessary, if the above isn't enough
to point me in a direction that has a better way of doing this...

Thanks for looking,

Jeff

Re: Is there a more elegant way to handle determing fail status?
 

On Tuesday, January 15, 2013 3:24:44 PM UTC-8, J wrote:
> Ok, so I have a diagnostic tool, written by someone else. That tool
>
> runs a series of small tests defined by the user and can simplified
>
> summary output that can be one of the following:
>
>
>
> FAILED_CRITICAL
>
> FAILED_HIGH
>
> FAILED_MEDIUM
>
> FAILED_LOW
>
> PASSED
>
>
>
> I also have a wrapper script I wrote to run these tests, summarize the
>
> results of all tests aggregated and then fail based on a particular
>
> fail level.
>
>
>
> The idea is that if I run 3 tests with the diagnostic tool and it
>
> tells me the following:
>
>
>
> testA: PASSED
>
> testB: FAILED_MEDIUM
>
> testC: PASSED
>
>
>
> AND I told the wrapper to only fail on HIGH or above, the wrapper will
>
> tell me that I had a medium failure, but the wrapper will still exit
>
> with a 0 (success)
>
>
>
> if I get the same results as above, but tell the wrapper to fail on
>
> LOW, then it will tell me I had that medium failure, but the wrapper
>
> will exit with a 1 (failure).
>
>
>
> The problem is that my exit determination looks like this:
>
>
>
> if fail_priority == fail_levels['FAILED_CRITICAL']:
>
> if critical_fails:
>
> return 1
>
> if fail_priority == fail_levels['FAILED_HIGH']:
>
> if critical_fails or high_fails:
>
> return 1
>
> if fail_priority == fail_levels['FAILED_MEDIUM']:
>
> if critical_fails or high_fails or medium_fails:
>
> return 1
>
> if fail_priority == fail_levels['FAILED_LOW']:
>
> if critical_fails or high_fails or medium_fails or low_fails:
>
> return 1
>
>
>
> return 0
>
>
>
> So, to explain the above... the fail level can be set by the user when
>
> running the wrapper using -f (or it defaults to 'high')
>
> the wrapper assigns a number to each level using this:
>
>
>
> # Set correct fail level
>
> args.fail_level = 'FAILED_%s' % args.fail_level.upper()
>
>
>
> # Get our failure priority and create the priority values
>
> fail_levels = {'FAILED_CRITICAL':4,
>
> 'FAILED_HIGH':3,
>
> 'FAILED_MEDIUM':2,
>
> 'FAILED_LOW':1}
>
> fail_priority = fail_levels[args.fail_level]
>
>
>
> the variables critical_fails, high_fails, medium_fails, low_fails are
>
> all counters that are etiher None, or the number of tests that were
>
> failed.
>
>
>
> So using this output from the diagnostic tool:
>
>
>
> testA: PASSED
>
> testB: FAILED_HIGH
>
> testC: PASSED
>
> testD: FAILED_MEDIUM
>
> testE: PASSED
>
>
>
> critical_fails would be None
>
> high_fails would be 1
>
> medium_fails would be 1
>
> low_fails would be None.
>
>
>
> The exit code determination above works, but it just feels inelegant.
>
> It feels like there's a better way of implementing that, but I can't
>
> come up with one that still honors the fail level properly (e.g. other
>
> solutions will fail on medium, but won't fail properly on medium OR
>
> higher).
>
>
>
> I can provide the full script if necessary, if the above isn't enough
>
> to point me in a direction that has a better way of doing this...
>
>
>
> Thanks for looking,
>
>
>
> Jeff

My back of the envelope coding would do it this way. Use an array of fail_counters, with PASSED as the first element all the way up to FAILED_CRITICALas the last element. Then use a simple loop starting from index fail_priority to the end of the list looking for errors. Like this:

# Array of fail counters
fail_counters = [
0, # PASSED
0, # LOW
0, # MEDIUM
0, # HIGH
0 # CRITICAL
]

.... run tests, accumulate error counts in fail_counters

for i in range(fail_priority, len(fail_counters)):
if fail_counters[i]:
return 1
return 0

Re: Is there a more elegant way to handle determing fail status?
 

On Tuesday, January 15, 2013 3:24:44 PM UTC-8, J wrote:
> Ok, so I have a diagnostic tool, written by someone else. That tool
>
> runs a series of small tests defined by the user and can simplified
>
> summary output that can be one of the following:
>
>
>
> FAILED_CRITICAL
>
> FAILED_HIGH
>
> FAILED_MEDIUM
>
> FAILED_LOW
>
> PASSED
>
>
>
> I also have a wrapper script I wrote to run these tests, summarize the
>
> results of all tests aggregated and then fail based on a particular
>
> fail level.
>
>
>
> The idea is that if I run 3 tests with the diagnostic tool and it
>
> tells me the following:
>
>
>
> testA: PASSED
>
> testB: FAILED_MEDIUM
>
> testC: PASSED
>
>
>
> AND I told the wrapper to only fail on HIGH or above, the wrapper will
>
> tell me that I had a medium failure, but the wrapper will still exit
>
> with a 0 (success)
>
>
>
> if I get the same results as above, but tell the wrapper to fail on
>
> LOW, then it will tell me I had that medium failure, but the wrapper
>
> will exit with a 1 (failure).
>
>
>
> The problem is that my exit determination looks like this:
>
>
>
> if fail_priority == fail_levels['FAILED_CRITICAL']:
>
> if critical_fails:
>
> return 1
>
> if fail_priority == fail_levels['FAILED_HIGH']:
>
> if critical_fails or high_fails:
>
> return 1
>
> if fail_priority == fail_levels['FAILED_MEDIUM']:
>
> if critical_fails or high_fails or medium_fails:
>
> return 1
>
> if fail_priority == fail_levels['FAILED_LOW']:
>
> if critical_fails or high_fails or medium_fails or low_fails:
>
> return 1
>
>
>
> return 0
>
>
>
> So, to explain the above... the fail level can be set by the user when
>
> running the wrapper using -f (or it defaults to 'high')
>
> the wrapper assigns a number to each level using this:
>
>
>
> # Set correct fail level
>
> args.fail_level = 'FAILED_%s' % args.fail_level.upper()
>
>
>
> # Get our failure priority and create the priority values
>
> fail_levels = {'FAILED_CRITICAL':4,
>
> 'FAILED_HIGH':3,
>
> 'FAILED_MEDIUM':2,
>
> 'FAILED_LOW':1}
>
> fail_priority = fail_levels[args.fail_level]
>
>
>
> the variables critical_fails, high_fails, medium_fails, low_fails are
>
> all counters that are etiher None, or the number of tests that were
>
> failed.
>
>
>
> So using this output from the diagnostic tool:
>
>
>
> testA: PASSED
>
> testB: FAILED_HIGH
>
> testC: PASSED
>
> testD: FAILED_MEDIUM
>
> testE: PASSED
>
>
>
> critical_fails would be None
>
> high_fails would be 1
>
> medium_fails would be 1
>
> low_fails would be None.
>
>
>
> The exit code determination above works, but it just feels inelegant.
>
> It feels like there's a better way of implementing that, but I can't
>
> come up with one that still honors the fail level properly (e.g. other
>
> solutions will fail on medium, but won't fail properly on medium OR
>
> higher).
>
>
>
> I can provide the full script if necessary, if the above isn't enough
>
> to point me in a direction that has a better way of doing this...
>
>
>
> Thanks for looking,
>
>
>
> Jeff

My back of the envelope coding would do it this way. Use an array of fail_counters, with PASSED as the first element all the way up to FAILED_CRITICALas the last element. Then use a simple loop starting from index fail_priority to the end of the list looking for errors. Like this:

# Array of fail counters
fail_counters = [
0, # PASSED
0, # LOW
0, # MEDIUM
0, # HIGH
0 # CRITICAL
]

.... run tests, accumulate error counts in fail_counters

for i in range(fail_priority, len(fail_counters)):
if fail_counters[i]:
return 1
return 0

Re: Is there a more elegant way to handle determing fail status?
 

On Tue, 15 Jan 2013 18:24:44 -0500, J wrote:


> The problem is that my exit determination looks like this:
>
> if fail_priority == fail_levels['FAILED_CRITICAL']:
> if critical_fails:
> return 1
> if fail_priority == fail_levels['FAILED_HIGH']:
> if critical_fails or high_fails:
> return 1
> if fail_priority == fail_levels['FAILED_MEDIUM']:
> if critical_fails or high_fails or medium_fails:
> return 1
> if fail_priority == fail_levels['FAILED_LOW']:
> if critical_fails or high_fails or medium_fails or low_fails:
> return 1
> return 0


Here's a general solution that doesn't require the failure levels to be
numbers, or listed in order, or to support any operation except equality
testing.

Start with an ordered list of failure severity levels, and an equivalent
ordered list of failure counts. Find the offset of the requested failure
level in the FAILURES list. Then inspect the failure counts, up to and
including that offset, ignoring any failure outside of that range:


failures = [FAILED_CRITICAL, FAILED_HIGH, FAILED_MEDIUM, FAILED_LOW]
counts = [critical_fails, high_fails, medium_fails, low_fails]
i = failures.index(fail_priority)
if any(counts[:i+1]):
print "Failed"
else:
print "No failures that we care about"


The actual values for FAILED_CRITICAL etc. can be arbitrary values:
integers *in any order*, strings, complicated records, anything so long
as they support equality and are distinct. fail_priority must be set to
one of those values.


> the variables critical_fails, high_fails, medium_fails, low_fails are
> all counters that are etiher None, or the number of tests that were
> failed.

Why set them to None when no tests failed? If no tests failed, why not
just use 0?

That is, instead of the counters being "the number of failures, or None",
they could be "the number of failures".



--
Steven