NEON optimization alignment problem

I built ffmpeg on Cortex-A8 on CodeSourcery 2007q3-51 with:
--arch=arm --cpu=cortex-a8 --extra-cflags="-mfpu=neon -mfloat-abi=softfp"

It's causing thousands of alignment trap in /var/log/messages.  However if
set /proc/cpu/alignment to be 0 (ignored),  ffmpeg just works correctly.

It might be related to FAST_UNALIGNED,  but even I turned it off,  the
alignment is still happening from couple places of NEON code.  For example,
alignment trap is caught in ff_h264_h_loop_filter_luma_neon where "VLD1.64"
is loading from address not aligned to 64bits.

Is the alignment have to be set to ignored? or I didn't compile it
correctly?