Eric Blake | 4 Apr 21:50 2007
Picon

Re: Escape colour codes

Thorsten Kampe <thorsten <at> thorstenkampe.de> writes:

> > I need a simple test case.
> 
> It's easy to show that one Cygwin application (lftp) has a problem 
> with its prompt but it's rather lenghty to show that not lftp is the 
> culprit but readline making output to Windows Terminals (Cmd, 4NT, 
> Console, FAR manager, Poderosa)
> 
> 1. Install lftp, start cygwin.bat
> 
> 2. Start lftp. Type
> set cmd:prompt "\[\e[1;36m\]>\[\e[m\] "

Yep, I reproduced that.  Thanks for the better details.

> Conclusion: there is something wrong with lftp or readline or the 
> Terminal.

The bug is in lftp.  Read on.

> 6. Insert
> prompt1 "%{\e[1;36m%}>%{\e[m%} "
> into your yafcrc and start yafc

The bug is in yafc.

> 9. Install Python and IPython (http://ipython.scipy.org/moin/)
> set
> prompt_in1 '\C_White[\#\C_White]\C_LightCyan>>> '
> prompt_out '\C_White[\N\C_White]    '
> in your ipythonrc and start IPython. Type 1 [Enter]

The bug is in IPython.

> 
> 12. Remove the \001/\002 constructs in line 88 and 89 from 
> ColorANSI.py in Cygwin IPython 
> 
> from
>         Normal = '\001\033[0m\002'   # Reset normal coloring
>         _base  = '\001\033[%sm\002'  # Template for all other colors
> 
> to
> 
>         Normal = '\033[0m'   # Reset normal coloring
>         _base  = '\033[%sm'  # Template for all other colors

I'm not sure if this is correct.  If you pass invisible characters to readline 
without marking them as such, using \1 and \2, then readline messes up the 
display width.  In other words, I wonder if IPython is adding extra \1 
somewhere else in the sequence of things, which is then resulting in the 
spurious \1 to the cmd terminal.

> 13. Recompile yafc, comment out lines 167 and 172 in prompt.c
> 
> 			case '{': /* begin non-printable character string */
> #ifdef HAVE_LIBREADLINE
> //				ins = "\001\001"; /* \001 + 

Actually, that should be:

ins = "\001"; /* RL_PROMPT_START_IGNORE */

> RL_PROMPT_START_IGNORE */
> #endif
> 				break;
> 			case '}': /* end non-printable character string */
> #ifdef HAVE_LIBREADLINE
> //			ins = "\001\002"; /* \001 + RL_PROMPT_END_IGNORE */

And that should be:

ins = "\002"; /* RL_PROMPT_END_IGNORE */

In other words, the extra \001 is what is resulting in the smiley faces.

> I tried to modify the lftp source, too, but my C knowledge was not 
> sufficient
> 
>    char StartIgn[3], EndIgn[3];
>    /* bash adds the extra \001 too, don't know why */
>    StartIgn[0] = '\001';
>    StartIgn[1] = RL_PROMPT_START_IGNORE;
>    StartIgn[2] = 0;
>    EndIgn[0] = '\001';
>    EndIgn[1] = RL_PROMPT_END_IGNORE;
>    EndIgn[2] = 0;

Rewrite that as:

char StartIgn[2], EndIgn[2];
StartIgn[0] = RL_PROMPT_START_IGNORE;
StartIgn[1] = '\0';
EndIgn[0] = RL_PROMPT_END_IGNORE;
EndIgn[1] = '\0';

The bug in all three of these programs is that they are adding spurious \1 into 
the string passed to readline.  When you call readline("\001\001invisible\001
\002plain"), then readline assumes that anything between the FIRST \001 and the 
\002 is invisible (ie. special to the terminal instead of literal output).  So 
readline thinks that it should PRINT the invisible string "\001invisible\001" 
special to the terminal, followed by the visible string "plain".  However, as 
you noticed, \001 is NOT special to the cmd.com terminal, and results in a 
smiley face, and readline is now thoroughly confused (it thinks it is waiting 
for input on position 6, but in reality it is waiting for input on position 8, 
because you printed literal characters while claiming they were invisible).

Bash, on the other hand, DOES map \[ to the sequence '\001\001' inside of 
parse.y's decode_prompt_string(), BECAUSE it later calls expand_prompt_string() 
to get rid of the extra \001.  It needs to do this so that it can support 
PS1='$(foo)' (the prompt is the expansion of command foo), and needed a way to 
tell \[ and \] in PS1 apart from literal \001 and \002 resulting from the 
expansion of other elements in the prompt string.  When the prompt is finally 
expanded and ready to hand to readline, the extra \001 _used by bash_ is gone, 
leaving only the SINGLE \001 _used by readline_.  In other words, the common 
bug in all three programs you mentioned is that they copied bash's escape 
sequences, but NOT bash's round of internal expansion, prior to calling 
readline.  The comment in the lftp sources was rather revealing - if the coder 
didn't know why bash used an extra \001, they shouldn't have copied that.

Meanwhile, I'm asking the upstream readline maintainer if there is any way to 
output a literal printing \001 (cmd.com hollow smiley) without having it be 
claimed as invisible (in case you really _wanted_ a smiley in your prompt), and 
the converse question of if there is any way to output a literal \002 (cmd.com 
solid smiley) as part of an invisible sequence (in case it is possible that 
your terminal can change its title bar to include a smiley, for example).

Just because most other terminals (rxvt, xterm, ...) are tolerant of unknown 
control characters, and treat spurious invisible \001 as non-printing 
characters, doesn't mean that lftp, yafc, or IPython should assume that all 
terminals behave that way.

--

-- 
Eric Blake
volunteer cygwin readline maintainer


Gmane