English
Etymology
new +
line — the term
was coined at Bell Laboratories while developing the
C
programming language and
UNIX operating
systems.
Noun
- The character or
character sequence that
indicates the end of a line
of text and transition to
the next line; or, a control code
or escape
sequence used in a programming
language to denote this character.
-
- Microsoft Windows uses CRLF for newline.
- 1959, Association for Computing Machinery, Communications of
the ACM (ACM Digital Library)
Page 599
- The ugly part is the quote marks on two adjacent lines that
mean a newline character.
- 1987, T. D. Brown, C for Basic Programmers
Page 13
- The calculator program starts off by printing the string "O\n",
that is, it prints a zero and then moves to a newline.
- 2002, Laura Lemay, Rafe Colburn, Sams Teach Yourself Perl in 21
Days,
Page 69
- Unlike in C, you don't have to loop through the output and
watch every character to make sure it's a newline; Perl will keep
track of that for you.
- 2002, Allen B. Downey, Jeffrey Elkner and Chris Meyers,
How to Think Like a Computer Scientist: Learning with Python
- It contains only a single statement, which outputs a newline
character. (That's what happens when you use a print command
without any arguments.)
- 2006, Jeffrey E.F. Friedl, Mastering Regular Expressions
Page 111
- The original Unix regex tools worked on a line-by-line basis,
so the thought of matching a newline wasn't an issue until the
advent of sed and lex.
In
computing, a newline (also
known as a line break or end-of-line / EOL character) is a special
character
or sequence of characters signifying the end of a line of text. The
name comes from the fact that the next character after the newline
will appear on a new line—that is, on the next line below the text,
immediately proceeding the newline. The actual codes representing a
newline vary across hardware platforms and operating systems, which
can be a problem when exchanging data between systems with
different representations.
There is also some confusion as to whether
newlines terminate or separate lines. If a newline is considered a
separator, there will be no newline after the last line of a file.
The general convention on most systems is to add a newline even
after the last line, i.e., to treat newline as a line terminator.
Some programs have problems processing the last line of a file if
it isn't newline terminated. Conversely, programs that expect
newline to be used as a separator will interpret a final newline as
starting a new (empty) line. This can result in a different line
count being reported for the file, but is otherwise generally
harmless.
In text intended primarily to be read by humans
using software which implements the
word wrap
feature, a newline character typically only needs to be stored if a
line break is required independent of whether the next word would
fit on the same line, such as between paragraphs and in vertical
lists. See
hard return
and
soft
return.
Representations
Software applications and
operating
systems usually represent a newline with one or two
control
characters:
- Systems based on ASCII or a compatible
character set use either LF (Line feed, 0Ah) or CR (Carriage
Return, 0Dh) individually, or CR followed by LF (CR+LF, 0Dh
0Ah); see below for the historical reason for the CR+LF convention.
These characters are based on printer commands: The line feed
indicated that one line of paper should feed out of the printer,
and a carriage return indicated that the printer carriage should
return to the beginning of the current line.
- LF: Multics, Unix and Unix-like systems
(GNU/Linux, AIX,
Xenix,
Mac OS
X, etc.), BeOS, Amiga, RISC OS, and
others
- CR+LF: DEC
RT-11 and
most other early non-Unix, non-IBM OSes, CP/M, MP/M, DOS, OS/2, Microsoft
Windows
- CR: Commodore
machines, Apple II
family, Mac OS
up to version 9 and
OS-9
- EBCDIC
systems—mainly
IBM mainframe systems, including z/OS (OS/390) and i5/OS (OS/400)—use NEL
(Next Line, 0x15) as the newline character. Note that EBCDIC also
has control characters called CR and LF, but the numerical value of
LF (25) differs from the one used by ASCII (0A). Additionally,
there are some EBCDIC variants that also use NEL but assign a
different numeric code to the character.
- Operating systems for the CDC 6000
series defined a newline as two or more zero-valued six-bit
characters at the end of a 60-bit word. Some configurations also
defined a zero-valued character as a colon
character, with the result that multiple colons could be
interpreted as a newline depending on position.
- OpenVMS
uses a record-based
file system, which stores text files as one record per line. In
most file formats, no line terminators are actually stored, but the
Record
Management Services facility can transparently add a terminator
to each line when it is retrieved by an application.
- Fixed line length was used by some early mainframe operating systems.
In such a system, an implicit end-of-line was assumed every, say,
80 characters. No newline character was stored. If a file was
imported from the outside world, lines shorter than the line length
had to be padded with spaces, while lines longer than the line
length had to be truncated. This mimicked the use of punched
cards, on which each line was stored on a separate card,
usually with 80 columns on each card.
Most textual
Internet protocols
(including
HTTP,
SMTP,
FTP,
IRC
and many others) mandate the use of ASCII CR+LF (0Dh 0Ah) on the
protocol level, but recommend that tolerant applications recognize
lone LF as well. In practice, there are many applications that
erroneously use the
C
newline character '\n' instead (see section
Newline in programming languages below). This leads to problems
when trying to communicate with systems adhering to a stricter
interpretation of the standards; one such system is the
qmail MTA
that actively refuses to accept messages from systems that send
bare LF instead of the required CR+LF.
http://cr.yp.to/docs/smtplf.html
Unicode
The
Unicode standard
addresses the problem by defining a large number of characters that
conforming applications should recognize as line terminators:
LF: Line
Feed, U+000A CR:
Carriage
Return, U+000D CR+LF: CR followed by LF, U+000D
followed by U+000A NEL: Next Line,
U+0085 FF: Form Feed,
U+000C LS: Line Separator,
U+2028 PS: Paragraph
Separator, U+2029
This may seem overly complicated compared to an
approach such as converting all line terminators to a single
character, for example LF. The simple approach breaks down,
however, when trying to convert a text file from an encoding like
EBCDIC to
Unicode and back. When converting to Unicode, NEL would have to be
replaced by LF, but when converting back it would be impossible to
decide if an LF should be mapped to an EBCDIC LF or NEL. The
approach taken in the Unicode standard allows round-trip
transformation to be information-preserving while still enabling
applications to recognize all possible types of line
terminators.
History
ASCII was developed simultaneously by the
ISO and the ASA, the predecessor organization to
ANSI. During the period of
1963–
1968,
the ISO draft standards supported the use of either CR+LF or LF
alone as a newline, while the ASA drafts supported only CR+LF. The
Multics
operating system began development in
1964 and used LF alone
as its newline.
Unix followed the
Multics practice, and later systems followed Unix.
The sequence CR+LF was in common use on many
early computer systems that had adopted
teletype machines, typically an
ASR33, as a
console device, because this sequence was required to position
those printers at the start of a new line. On these systems, text
was often routinely composed to be compatible with these printers,
since the concept of
device
drivers hiding such hardware details from the application was
not yet well developed; applications had to talk directly to the
teletype machine and follow its conventions. The separation of the
two functions concealed the fact that the print head could not
return from the far right to the beginning of the next line in
one-character time. That is why the sequence was always sent with
the CR first. In fact, it was often necessary to send extra
characters (extraneous CRs or NULs, which are ignored) to give the
print head time to move to the left margin. Even after teletypes
were replaced by
computer
terminals with higher
baud rates, many operating systems
still supported automatic sending of these fill characters, for
compatibility with cheaper terminals that required multiple
character times to scroll the display.
MS-DOS, built upon a
CP/M clone
called
86-DOS (which
Microsoft
purchased and renamed), adopted CP/M's CR+LF; CP/M's use of CR+LF
made sense for using computer terminals via serial lines. This
convention was inherited by Microsoft's later
Windows
operating system.
Newline in programming languages
To facilitate the creation
of
portable
programs, programming languages provide some abstractions to deal
with the different types of newline sequences used in different
environments.
The
C
programming language provides the
escape
sequences '\n' (newline) and '\r' (carriage return). However,
contrary to popular belief, these are in fact not required to be
equivalent to the ASCII LF and CR control characters. The C
standard only guarantees two things:
- Each of these escape sequences maps to a unique
implementation-defined number that can be stored in a single char
value.
- When writing a file in text mode, '\n' is transparently
translated to the native newline sequence used by the system, which
may be longer than one character. (Note that a C implementation is
allowed to not store newline characters in files. For example, the
lines of a text file could be stored as rows of a SQL table or as
fixed-length records.) When reading in text mode, the native
newline sequence is translated back to '\n'. In binary mode, the
second mode of I/O supported
by the C library, no translation is performed, and the internal
representation of any escape sequence is output
directly.
On Unix platforms, where C originated, the native
newline sequence is ASCII LF (0x0A), so '\n' was simply defined to
be that value. With the internal and external representation being
identical, the translation performed in text mode effectively turns
into a
no-op,
making text mode and binary mode behave the same. This has caused
many programmers who developed their software on Unix systems to
simply ignore the distinction completely, resulting in code that is
not portable to different platforms.
Another common problem is the use of '\n' when
communicating using an Internet protocol that mandates the use of
ASCII CR+LF for ending lines. Writing '\n' to a text mode stream
works correctly on Windows systems, but produces only LF on Unix,
and something completely different on more exotic systems. Using
"\r\n" in binary mode is slightly better, as it works on many
ASCII-compatible systems, but still fails in the general case. One
approach is to use binary mode and specify the numeric values of
the control sequence directly, "\x0D\x0A".
Many languages, such as
C++,
Perlhttp://perldoc.perl.org/functions/binmode.html
and
Python provide the same interpretation of '\n' as C.
Java also provides '\n' and '\r' escape sequences. In contrast
to C, these are guaranteed to represent the values U+000A and
U+000D, respectively. The Java I/O libraries do not transparently
translate these into platform-dependent newline sequences on input
or output. Instead, they provide functions for writing a full line
that automatically add the native newline sequence, and functions
for reading lines that accept any of CR, LF, or CR+LF as a line
terminator (see
BufferedReader.readLine()). The
System.getProperties method can be used to retrieve the
underlying line separator.
Some languages have created special
variables,
constants and
subroutines to
facilitate newlines during program execution. One example is the
PHP constant
PHP_EOL, which will produce either '\r\n' or '\n' appropriate to
the operating system the program is executed on.
http://www.php.net/reserved.constants
Though special newline handling facilities can aid execution during
runtime, they do not ensure the validity of newlines for the
source
code itself.
Common problems
The different newline conventions often
cause text files that have been transferred between systems of
different types to be displayed incorrectly. For example, files
originating on
Unix or
Apple
Macintosh systems may appear as a single long line on a
Windows
system. Conversely, when viewing a file from a Windows computer on
a Unix system, the extra CR may be displayed as ^M at the end of
each line or as a second line break.
The problem can be hard to spot if some programs
handle the foreign newlines properly while others don't. For
example, a
compiler may
fail with obscure syntax errors even though the source file looks
correct when displayed on the
console
or in an
editor.
Modern text editors generally recognize all flavours of CR / LF
newlines and allow the user to convert between the different
standards.
Web browsers
are usually also capable of displaying text files of different
types.
The
File
Transfer Protocol can automatically convert newlines in files
being transferred between
systems
with different newline representations when the transfer is done in
"ASCII mode". However, transferring binary files in this mode
usually has disastrous results: Any occurrence of the newline byte
sequence—which does not have line terminator semantics in
this context, but is just part of a normal sequence of
bytes—will be translated to whatever newline
representation the other system uses, effectively corrupting the
file. FTP clients often employ some
heuristics (for example, inspection of
filename
extensions) to automatically select either binary or ASCII
mode, but in the end it is up to the user to make sure his or her
files are transferred in the correct mode. If there is any doubt as
to the correct mode, binary mode should be used, as then no files
will be altered by FTP, though they may display incorrectly.
Conversion utilities
Generally, using a text editor is the
simplest and most convenient way of converting a text file between
different newline formats; most modern editors can read and write
files using at least the different ASCII CR/LF conventions.
Unfortunately, the standard
Windows
editor
Notepad is not one
of them (though
Wordpad is).
On Windows systems without a better editor, the
old MS-DOS editor EDIT that still ships with modern Windows
versions can be used to convert a Unix text file to DOS/Windows
newlines. A simple way of doing this is by creating a shortcut to
EDIT on the desktop (context menu / New / Shortcut / "edit" / Next
/ Finish), dragging the text file in question onto it, and then
saving the file again (File / Save).
Editors are often unsuitable for converting
larger files. For larger files (on Windows NT/2000/XP) you can use
the following command: TYPE unix_file | FIND "" /V >
dos_file
On many
Unix systems, the
dos2unix (sometimes named fromdos) and
unix2dos
(sometimes named todos) utilities can be used to translate between
ASCII CR+LF (DOS/Windows) and LF (Unix) newlines. Different
versions of these commands vary slightly in their syntax. However,
the
tr
command is available on virtually every
Unix-like system
and can be used to perform arbitrary replacement operations on
single characters. A DOS/Windows text file can be converted to Unix
format by simply removing all ASCII CR characters with tr -d '\r'
< inputfile > outputfile or, if the text has only
CRs, by converting CRs to LFs with tr '\r' '\n' < inputfile
> outputfile
The same tasks can be performed with
sed, or in
Perl if the platform
has a Perl interpreter: sed -e 's/$/\r/' inputfile >
outputfile # UNIX to DOS (adding CRs) sed -e 's/\r$//' inputfile
> outputfile # DOS to UNIX (removing CRs) perl -pe
's/\r\n|\n|\r/\r\n/g' inputfile > outputfile # Convert to
DOS perl -pe 's/\r\n|\n|\r/\n/g' inputfile > outputfile #
Convert to UNIX perl -pe 's/\r\n|\n|\r/\r/g' inputfile >
outputfile # Convert to old Mac
To identify what type of line breaks a text file
contains one can use
grep.
The following grep commands will echo the filename (in this case
myfile.txt) to the command line if the file is of the specified
style:
grep -PL '\r\n' myfile.txt # show UNIX style file
(LF terminated) grep -Pl '\r\n' myfile.txt # show DOS style file
(CRLF terminated)
For Debian-based systems, you can use these
commands:
egrep -L '\r\n' myfile.txt # show UNIX style file
(LF terminated) egrep -l '\r\n' myfile.txt # show DOS style file
(CRLF terminated)
The above grep commands will work under
Unix systems or in
Cygwin under
Windows. Note that these commands make some assumptions about the
kinds of files that exist on the system (specifically it's assuming
only UNIX and DOS-style files—no Mac OS 9-style files).
Check the -P, -L, and -l options to understand how it works.
This technique can be further combined with
find to list files
recursively. For instance, the following will check all "regular
files" (e.g. it will exclude directories, symbolic links, etc.) to
find all UNIX-style files in a directory tree, starting from the
current directory (.), and save the results in file unix_files.txt,
overwriting it if the file already exists:
find . -type f -exec grep -PL '\r\n' \; >
unix_files.txt
dos2unix, unix2dos, mac2unix, unix2mac, mac2dos,
dos2mac can perform conversions. The flip
http://ccrma-www.stanford.edu/~craig/utility/flip/
command may also be useful.
newline in Czech: Nový řádek
newline in German: Zeilenvorschub
newline in Spanish: Nueva línea
newline in French: CRLF
newline in Japanese: 改行コード
newline in Polish: End-of-line
newline in Portuguese: Newline