Add modest memory size limits on some tests

This should help insure that jobs are not prevented from running
at the same time on the same node due to memory limits.

bug 6675