Cause excessively large malloc()s to fail rather than assert.

GCC's exception frame searching will malloc() larger than a page;
returning NULL forces it to fall back on slower, in-place searching.

Signed-off-by: Scott Wood <scott@thor.buserror.net>