Stick something like this in your .bashrc file:

# set PS1, last exit code if != 0, and PWD, with colors
PRE='\[\033['
POST='m\]'
COL_LASTEXIT="${PRE}01;41;37${POST}" # last command exit value
COL_PWD="${PRE}00;01;36${POST}" # current directory
COL_DEFAULT="${PRE}00${POST}" # back to default colors
export PS1="$COL_LASTEXIT\${?#0}$COL_PWD \w$COL_DEFAULT\\n\\\$ "

Command exit values other than zero are displayed in the subsequent prompt:

Check out the little red '1'

It’s the

${?#0}

bit which includes the last command exit value in the prompt, unless the value is zero, in which case nothing is displayed.