Unix Command Pipelines for Packet Analysis

 

Graphical packet analyzers provide powerful capabilities for protocol dissection, packet inspection, and visual correlation of network events. However, their operational efficiency can decline as capture datasets increase in size. When dealing with multi-gigabyte or terabyte-scale PCAP/PCAPNG files, loading, filtering, indexing, and navigating packet records through a graphical interface can impose significant memory and processing demands. Although performance depends on the underlying hardware, capture characteristics, and analyzer configuration, large-scale investigations often benefit from command-line processing. One of the principal advantages of this approach is the ability to construct processing pipelines, where the standard output (stdout) of one utility is connected to the standard input (stdin) of another. This allows investigators to combine specialized, single-purpose tools into repeatable analysis workflows without necessarily generating intermediate files on disk.

Pipelines originated in UNIX environments but are not exclusive to them. MS-DOS 2.0 introduced the pipe operator (|) through COMMAND.COM, initially implementing command-to-command data transfer using temporary files. Modern UNIX-like systems generally implement pipelines through interconnected standard input and output streams, enabling concurrent processing. Windows also supports command pipelines through cmd.exe while PowerShell implements a more sophisticated pipeline model that passes structured .NET objects between commands rather than relying exclusively on textual streams. The distinction is particularly relevant to forensic workflows because many UNIX-based network analysis utilities are designed around text-oriented processing. Tools such as grep, awk, sort, uniq, and cut can therefore be combined to filter, normalize, aggregate, and extract investigative indicators from command output.

A fundamental example of pipeline usage is directing extensive command output into a terminal pager. Rather than allowing a directory listing generated by ls -al to scroll continuously beyond the visible terminal area, an investigator can pipe the output into less or more for controlled navigation. The same principle applies to packet inspection with tcpdump. Although the -c option limits the number of packets processed, even a small packet sample can generate substantial terminal output when hexadecimal and ASCII representations are requested. The -X option displays packet data in hexadecimal and ASCII without the link-layer header, whereas -XX includes the link-layer header in the representation.

For a standard untagged Ethernet frame carrying a 1,500-byte IP packet, the captured frame is typically 1,514 bytes, excluding the four-byte Ethernet Frame Check Sequence (FCS), which is commonly absent from packet captures. When the complete captured packet data is displayed, the resulting hexadecimal and ASCII representation can occupy dozens of terminal lines per packet. Consequently, examining even a handful of packets can quickly overwhelm the terminal display. Piping the output through less provides controlled pagination, backward navigation, and text-search capabilities, making packet inspection more manageable:

tcpdump -nn -r evidence.pcap -c 5 -X | less

Both more and less are terminal paging utilities designed to present lengthy command output in manageable screen-sized segments. Rather than allowing packet-analysis output to scroll continuously beyond the terminal's visible buffer, these utilities provide controlled navigation through captured output. The more utility originated in the 3BSD UNIX distribution in 1978. Its successor, less, was developed by Mark Nudelman and first released in 1985 to address several limitations associated with traditional paging utilities. Unlike more, which primarily supports forward navigation, less provides more flexible movement through displayed content, including backward navigation when examining buffered or piped input. It is distributed independently as open-source software and is available across numerous UNIX-like operating systems and Linux distributions.

Both utilities support forward text searching using the / character. However, less provides additional navigation and search capabilities, including backward searching with ?, repeating searches with n and N, regular-expression-based pattern matching, and search-result highlighting. These capabilities are particularly useful during forensic examination, where investigators may need to locate specific IP addresses, port numbers, protocol identifiers, timestamps, or other investigative artefacts within extensive command output.

An important operational distinction concerns their handling of piped input. When command output is redirected into a pager through a pipeline, less can buffer incoming data and permit backward navigation through previously displayed content. Traditional implementations of more generally provide forward-only navigation when processing non-seekable input from a pipe. This makes less particularly suitable for examining lengthy packet dumps generated by tcpdump, where an investigator may need to revisit previously displayed packet information without rerunning the analysis command. For example:

tcpdump -n -r evidence.pcap -X | less

This command reads packets from the specified capture file, displays their hexadecimal and ASCII representations, and passes the resulting output to less for interactive examination. Investigators can navigate through the output, search for relevant strings, and revisit previously displayed packet records. Another useful option is -S, which disables line wrapping in less. This is particularly beneficial when examining hexadecimal packet representations because automatic line wrapping can disrupt the visual alignment of hexadecimal values and their corresponding ASCII representations. With line wrapping disabled, long lines remain intact, and investigators can navigate horizontally to inspect content extending beyond the terminal's visible width.

It is important to recognize that pager behaviour varies across operating systems and distributions. Some systems implement more as a compatibility interface backed by less, while others provide separate implementations with different navigation capabilities. Microsoft Windows also includes more.com, although its functionality and interaction with command pipelines differ from those of commonly used UNIX-like implementations. For network forensic investigations involving extensive packet output, less is generally the more appropriate choice because its buffering, backward navigation, search functionality, and horizontal scrolling capabilities support more controlled and efficient examination of packet-level evidence.

The cut utility extracts selected portions of text from each input line based on character positions (-c), byte positions (-b), or delimited fields (-f). By default, field extraction uses a tab character as the delimiter, although an alternative delimiter can be specified using the -d option.

Despite its simplicity, cut has an important limitation in forensic data processing: it does not understand the semantic structure of the input. It extracts content strictly according to positional boundaries or explicitly defined delimiters. Consequently, using fixed character positions against command output containing variable-length fields can produce incomplete or misleading results. Consider the default packet-summary output generated by tcpdump. A typical line contains a timestamp, a network-layer protocol identifier, a source address and port, a direction indicator (>), and a destination address and port. Although the timestamp generally occupies a predictable number of characters under the default formatting, the lengths of the remaining fields vary depending on the IP address, port number, protocol, and packet representation. For example, consider the following command in the figure above. This command extracts characters 17 through 40 from every line of tcpdump output. It may appear to produce meaningful results when examining packets originating from 192.168.99.52.52709 because the combined source-address representation happens to align with the selected character range, placing the packet-direction indicator (>) at the end of the extracted output. However, this alignment is coincidental rather than structurally reliable. Consider a packet originating from 68.183.138.51.80. Because the source-address and port representation is shorter, the destination-address information begins earlier within the selected character range and becomes partially included in the extracted output. Conversely, a longer source representation can cause the extraction to truncate the source information before reaching the direction indicator. The problem becomes more pronounced when tcpdump output varies because of hostname or service-name resolution, alternative timestamp formats introduced by options such as -t and -tttt, or the use of IPv6 addresses. These variations change the position and length of fields, making fixed-character extraction unreliable across heterogeneous packet records.

From a network forensic perspective, this demonstrates why positional extraction should be used cautiously when processing packet-summary output. Although cut is useful for consistently structured text, it is not a protocol-aware parser and cannot reliably identify network fields whose positions vary between records. Investigators should therefore prefer structured field extraction, protocol-aware tools, or more flexible text-processing utilities such as awk when working with variable-width packet metadata.

Although character-position extraction is vulnerable to variations in field length, the cut utility also supports field-based extraction. Instead of selecting fixed character ranges, investigators can specify fields using a delimiter. By default, cut uses a tab character as its field delimiter. Since tcpdump's packet-summary output is generally space-separated, the -d option can be used to specify a space as the delimiter. Quoting the space ensures that the shell passes it to cut as a single argument. Fields are numbered sequentially, beginning with 1. In a typical default IPv4 tcpdump summary line containing single-space-separated fields, the structure is approximately as follows:

Field	Content
1	Timestamp
2	Network-layer protocol identifier
3	Source address and port
4	Packet direction indicator (>)
5	Destination address and port, including trailing punctuation

Based on this representation, the following command selects the source and destination fields:

Unlike character-position extraction, selecting fields avoids dependence on the width of the source and destination address representations. A shorter or longer address does not, by itself, shift the selected field positions. The -f option also accepts comma-separated field lists and ranges. For example, -f3,5 selects the third and fifth fields, while -f3-5 selects all fields from the third through the fifth. An important characteristic of cut is that selected fields are emitted in their original input order, regardless of the order in which they are specified. Consequently, -f5,3 produces the same field ordering as -f3,5.

However, investigators should exercise caution when applying this technique to tcpdump output. The delimiter-based approach assumes a consistent single-space-separated representation. Multiple consecutive spaces can create empty fields, and changes in output formatting can invalidate the assumed field positions. Therefore, this method is suitable for controlled and verified output formats but should not be treated as a universally reliable way of extracting packet metadata. For more robust processing of variable-whitespace output, utilities such as awk, or structured packet-extraction tools are preferable.

The real strength of UNIX pipelines becomes apparent when multiple text-processing utilities are chained into a single analysis workflow. Rather than relying on one utility to perform every extraction operation, investigators can combine several small, specialized commands, with each stage transforming the output produced by the preceding stage. Consider the following example, which extracts source port numbers from tcpdump packet-summary output:

The pipeline performs the extraction in successive stages:

1. Packet summarization: tcpdump -n -r magnitude_1hr.pcap reads the capture file and generates packet-summary output. The -n option prevents hostname.

2. Source endpoint extraction: The first cut operation uses a space delimiter (-d' ') to select field 3 (-f3), which, in the assumed output format, contains the source IPv4 address and port.

3. Port extraction: The second cut operation changes the delimiter to a period (-d.) and selects field 5 (-f5). For an IPv4 endpoint represented as 192.168.99.52.52709, the first four fields correspond to the IPv4 octets, while the fifth contains the source port.

Although this pipeline illustrates the composability of UNIX text-processing utilities, its reliability depends on the packet-summary format and the protocols represented in the capture. It is not a protocol-aware extraction method. Several conditions can produce unexpected results. Protocols such as ICMP do not use TCP or UDP port numbers, so their packet-summary lines may produce empty output during the fifth-field extraction. ARP records do not follow the assumed IPv4 endpoint representation and may pass through the second cut operation unchanged because they contain no period delimiter. IPv6 addresses introduce additional complications because their address notation uses colons rather than periods, and their endpoint representation does not follow the five-field IPv4 structure.

The cut -s option can suppress input lines that do not contain the specified delimiter, but it does not establish whether the remaining fields represent valid IP addresses or transport-layer ports. Consequently, it should not be treated as a substitute for protocol validation. One way to reduce these inconsistencies is to restrict the packet records being processed using a Berkeley Packet Filter (BPF) expression:

The BPF expression limits the output to TCP and UDP packets, excluding protocols such as ARP and ICMP that do not provide the expected transport-port representation. However, this restriction does not eliminate the underlying formatting assumptions or make the extraction method suitable for IPv6 without modification. From a forensic perspective, this example demonstrates both the efficiency and the limitations of composable command-line processing. Pipelines allow investigators to progressively isolate information of interest from large packet datasets, but each transformation must be evaluated against the structure and characteristics of the underlying evidence. Where reliable extraction of network endpoints is required, protocol-aware tools and structured field extraction should be preferred over positional parsing of human-readable packet summaries.

awk is a pattern-action text-processing language named after its original authors: Alfred Aho, Peter Weinberger, and Brian Kernighan. It provides more flexible text extraction and transformation capabilities than utilities such as cut, making it particularly useful when processing variable-width packet-summary output. Unlike cut, which relies primarily on fixed character positions or delimiters, awk automatically divides input lines into fields. By default, it treats consecutive spaces and tabs as a single field separator, avoiding the empty-field problems that can arise when using cut with a single-space delimiter. An alternative field separator can be specified using the -F option.

One of awk's principal advantages is its pattern-action processing model. A pattern determines which input records should be processed, while the associated action specifies what should happen to matching records. This allows investigators to combine filtering and field extraction within a single command rather than chaining separate utilities such as grep and cut. Consider the following example:

The command operates as follows:

1. tcpdump -nn -r magnitude_1hr.pcap reads the capture file and generates packet-summary records without performing hostname or service-name resolution.

2. $2=="IP" tests whether the second field contains the exact network-layer identifier IP, restricting processing to matching IPv4 packet-summary records.

3. {print $3} extracts the third field, which contains the source IPv4 address and, where applicable, its associated transport-layer port.

The distinction between exact field matching and general regular-expression matching is important. An expression such as / IP / searches the entire input line for the specified character sequence. When tcpdump is configured to display packet payloads using options such as -A or -X, similar text may occur within the displayed payload. Testing $2=="IP" instead restricts the comparison to the expected protocol field, reducing the possibility of matching unrelated text.

Beyond extraction, awk supports associative arrays and arithmetic operations, allowing investigators to aggregate packet records without requiring a separate scripting environment. For example, the following pipeline counts IPv4 packets by source host and sorts the results in descending numerical order:

tcpdump -nn -r magnitude_1hr.pcap | awk '
$2=="IP" {
    n=split($3,a,".")
    h=(n==5) ? a[1]"."a[2]"."a[3]"."a[4] : $3
    c[h]++
}
END {
    for (h in c)
        print c[h], h
}' | sort -nr

This pipeline performs several operations:

1. Protocol selection: The condition $2=="IP" selects IPv4 packet-summary records.

2. Endpoint decomposition: split($3,a,".") divides the source endpoint into components using the period character as the delimiter.

3. Source-host normalization: When five components are identified, the first four are recombined to produce the IPv4 address without its appended port number. When the expected five-component representation is absent, the original source field is retained.

4. Packet aggregation: The associative array c[h]++ maintains a running count for each normalized source-host value.

5. Result generation: The END block executes after all input records have been processed, printing the accumulated packet counts and corresponding source hosts.

6. Numerical sorting: The final sort -nr operation arranges the results in descending numerical order based on packet count.

The resulting output provides a simple source-host traffic distribution, helping investigators identify addresses responsible for the highest number of observed IPv4 packets within the examined capture.

It is important to distinguish packet counts from connection counts or unique-flow counts. This pipeline counts matching packet-summary records; it does not identify distinct TCP sessions, UDP conversations, or application-layer transactions. Additionally, its endpoint-normalization logic assumes the conventional IPv4 address-and-port representation produced by tcpdump. Different output formats or endpoint representations may require adjustments. For evidentiary completeness, diagnostic output from tcpdump should not routinely be discarded using 2>/dev/null. Investigators should retain relevant warnings and errors, preferably in a separate examination log, so that problems encountered while reading or processing the capture remain documented.

This example demonstrates how awk extends command-line network analysis beyond simple extraction. Its pattern matching, field manipulation, conditional expressions, associative arrays, and aggregation capabilities make it a valuable component of scalable and reproducible network forensic workflows.

The utility of UNIX pipelines becomes particularly apparent when several text-processing commands are combined to produce frequency distributions. A common analysis pattern involves extracting a field from packet-summary output, grouping identical values, counting their occurrences, and ranking the results by frequency. Consider the following command, which identifies the most frequently observed source ports in a packet capture:

Each stage performs a distinct operation:

1. Packet summarization: tcpdump -nn -r magnitude_1hr.pcap reads the capture file and produces a textual summary for each packet.

2. Source endpoint extraction: The first cut selects field 3, which, in the assumed IPv4 output format, contains the source address and port.

3. Port extraction: The second cut uses the period character as a delimiter and selects field 5, corresponding to the source port in an endpoint such as 192.168.99.52.52709.

4. Grouping: sort arranges identical port values together. This is necessary because uniq only identifies and processes consecutive identical lines.

5. Frequency counting: uniq -c collapses adjacent identical values and prefixes each unique value with its occurrence count.

6. Ranking: sort -rn sorts the resulting records numerically in descending order, placing the most frequently observed ports first.

7. Result limitation: head displays the first ten records, providing a concise ranking of the most frequently observed source ports.

The initial sort is essential because packet records are encountered in capture order rather than grouped by source port. Without sorting, identical port values appearing in different parts of the input would be counted separately by uniq -c, producing an incomplete frequency distribution.

The resulting output represents a ranking of source ports by packet frequency. However, the appearance of a blank entry in the results requires attention. It indicates that the extraction process encountered records without the expected fifth field. Such records may originate from protocols without transport-layer ports or from packet representations that do not conform to the assumed IPv4 endpoint format. Consequently, the initial pipeline should not be treated as a reliable port-extraction method across heterogeneous packet captures. A more flexible approach uses awk to extract the source port while accounting for the conventional IPv4 and IPv6 endpoint representations produced by tcpdump:

tcpdump -nn -r magnitude_1hr.pcap 'tcp or udp' | awk '
{
    if ($2=="IP") {
        n=split($3,a,".")
        if (n==5) print a[5]
    }
    else if ($2=="IP6") {
        n=split($3,a,".")
        if (n==2) print a[2]
    }
}' | sort | uniq -c | sort -rn | head

This version introduces two improvements. First, the Berkeley Packet Filter expression (tcp or udp) restricts processing to TCP and UDP traffic, excluding protocols such as ICMP and ARP that do not use TCP or UDP ports. Second, awk distinguishes between IPv4 and IPv6 packet-summary records and applies the corresponding endpoint decomposition. For conventional IPv4 output, the source endpoint contains four address octets followed by the transport port, producing five period-separated components. For conventional IPv6 output, the port follows the colon-separated IPv6 address after a period, producing two period-separated components. The conditional checks ensure that only records matching these expected representations contribute a port value. An additional consideration is IP fragmentation. Non-initial fragments may be identified as TCP or UDP traffic by the BPF filter but may not contain an accessible transport-layer header. Such records do not necessarily expose a source port in tcpdump's packet-summary output. The component-count checks help prevent these records from being incorrectly interpreted as valid port values. The extracted port values are then passed through the same frequency-counting pipeline: sort groups identical values, uniq -c calculates their occurrence counts, and sort -rn ranks the results in descending numerical order.

The resulting distribution provides an overview of source-port activity within the examined capture. It can help investigators identify frequently observed transport ports and prioritize subsequent examination of associated network communications.

It is important to interpret these results correctly: the counts represent packets, not unique connections, sessions, or communicating hosts. A high-frequency source port does not, by itself, establish malicious activity or identify an application. Further examination of destination ports, endpoint pairs, TCP session characteristics, packet timing, and application-layer evidence is necessary to establish investigative significance.

After extracting, filtering, and grouping packet information using utilities such as cut, sort, and uniq, investigators can use wc (word count) to quantify the resulting dataset. When invoked without options, wc reports three measurements: the number of newline characters, the number of whitespace-separated words, and the number of bytes in the input, respectively. Its individual options allow investigators to obtain specific measurements:

• -l: Counts newline characters.

• -w: Counts whitespace-separated words.

• -c: Counts bytes.

• -m: Counts characters, with the interpretation depending on the active character encoding and locale.

For network forensic analysis, wc -l is particularly useful when the preceding pipeline produces one newline-terminated record per extracted value. Under these conditions, the reported count corresponds to the number of records in the output. Consider the following command:

This pipeline extracts source-port values from the assumed IPv4 packet-summary representation, groups identical values, calculates their frequencies, and finally counts the number of resulting frequency records. The important distinction is that wc -l is counting the output records generated by uniq -c, rather than the original packets. Therefore, the result represents the number of distinct extracted source-port values, subject to the correctness of the preceding extraction operations.

Investigators should also account for blank or malformed records. An empty line is still counted by wc -l if it contains a terminating newline. Consequently, unexpected records can inflate the reported total. Input validation and filtering should therefore precede the counting operation. A more concise approach is to use sort -u, which sorts the extracted values while retaining only unique entries, eliminating the need for a separate uniq operation:

tcpdump -nn -r magnitude_1hr.pcap 'tcp or udp' 2>/dev/null \
  | awk '
    $2=="IP" {
        n=split($3,a,".")
        if (n==5) print a[5]
    }
    $2=="IP6" {
        n=split($3,a,".")
        if (n==2) print a[2]
    }
  ' \
  | sort -u \
  | wc -l

This version explicitly handles the conventional IPv4 and IPv6 source-endpoint representations produced by tcpdump. The component-count checks prevent an address component from being mistakenly interpreted as a transport port when the expected port representation is absent. They also exclude records that do not match the assumed endpoint formats.

The BPF expression tcp or udp restricts processing to TCP and UDP traffic. However, investigators should remember that non-initial IP fragments can match this filter without containing a transport-layer header. Such fragments do not provide a source port for extraction, which is why validating the endpoint representation remains necessary.

Another important consideration is the meaning of distinctness. The preceding command counts distinct numeric source-port values across the entire capture, irrespective of the originating host. For example, if five different servers use source port 80, that value contributes only one entry to the distinct-port count. If the investigative objective is instead to determine the number of distinct source endpoints, the complete source address-and-port representation must be retained during extraction. In the conventional tcpdump summary format, this can be achieved using:

tcpdump -nn -r magnitude_1hr.pcap 'tcp or udp' \
  | awk '$2=="IP" || $2=="IP6" {print $3}' \
  | sort -u \
  | wc -l

This counts unique source endpoint strings rather than unique port numbers. It is therefore a different measurement and can provide a more meaningful view of the number of distinct communicating endpoints represented in the capture.

From a forensic perspective, the distinction between total packet counts, distinct port counts, and distinct endpoint counts is important. Each measurement answers a different investigative question, and the interpretation of the result must reflect the exact field and aggregation method used.

grep is a command-line pattern-matching utility used to search input files or streams for lines matching a specified regular expression. Its name originates from the ed text editor command g/re/p, meaning global search for a regular expression and print matching lines. In network forensic investigations, grep is particularly useful for filtering packet summaries, extracting indicators from log files, locating suspicious strings, and identifying records associated with specific network activities.

By default, grep interprets patterns using POSIX Basic Regular Expressions (BRE). In this syntax, several characters commonly associated with regular-expression operators, including +, ?, |, (, ), {, and }, do not have their usual operator meanings unless appropriately escaped. This can make complex patterns less readable and introduce errors when analysts assume that all regular-expression implementations follow the same conventions. The -E option enables Extended Regular Expressions (ERE), which provides a more intuitive syntax by allowing these operators to be used without escaping. This is particularly useful when constructing patterns involving alternation, grouping, or repetition. For example, the following command searches for HTTP GET and POST request methods:

grep -E 'GET|POST' access.log

The vertical bar (|) represents alternation, instructing grep to match either GET or POST. Under the default BRE interpretation, the same expression would require escaped alternation operators:

grep 'GET\|POST' access.log

Both BRE and ERE support POSIX character classes, which provide portable ways to match groups of characters. For example, [[:digit:]] matches decimal digits, while [[:alpha:]] matches alphabetic characters. These classes are useful when constructing patterns for identifying numeric values, addresses, timestamps, and other structured text within forensic artefacts.

Regular expressions are widely used throughout the command-line analysis toolkit, but their syntax and implementation vary between utilities. For example, grep defaults to BRE, sed also defaults to BRE, and awk uses ERE for its regular-expression operations. The sed -E option enables extended regular-expression syntax where supported. Network security monitoring and intrusion detection platforms introduce further differences. Snort and Suricata support PCRE-based pattern matching through their respective rule engines, while Zeek employs its own regular-expression implementation and matching syntax. Consequently, a pattern developed for one utility or detection platform should not automatically be assumed to behave identically in another.

Forensic analysts should therefore validate regular expressions in the actual utility or detection engine in which they will be executed. A pattern that successfully identifies an artefact in GNU grep may produce different results when transferred to an IDS rule or another text-processing environment. For reproducible examination workflows, explicitly specify the intended regular-expression mode rather than relying on defaults or shell aliases. For example, using grep -E makes the expected ERE interpretation clear to anyone reviewing or reproducing the command.

egrep is the historical command name associated with extended regular-expression searching. Although it remains available on many systems for backward compatibility, it has been deprecated in favour of the explicit grep -E form. Using grep -E in forensic scripts, examination notes, and documented command sequences improves clarity and portability.

sed, short for stream editor, is a command-line utility designed to perform text transformations as data passes through a processing pipeline. It is particularly useful in network forensic workflows where extracted information requires minor modifications, normalization, or formatting before being passed to subsequent analysis utilities. Unlike interactive text editors, sed processes input line by line, applying specified editing instructions to each line's pattern space. This makes it suitable for automated and repeatable text-processing operations without requiring investigators to open or manually modify the source files. One of its most frequently used operations is substitution, which follows the general syntax:

sed 's/regular_expression/replacement/flags'

For example, consider the following command:

sed 's/:/,/g' /etc/passwd

This command performs a character substitution on the contents of /etc/passwd. The colon (:) is the search pattern, the comma (,) is the replacement character, and the g flag instructs sed to replace every matching occurrence within each input line rather than stopping after the first match. The forward slash (/) characters serve as delimiters separating the search pattern, replacement expression, and optional flags. However, the delimiter is not fixed. An alternative character can be used, provided it does not conflict with the intended expression. For example:

sed 's|/var/log/|/evidence/logs/|g' filelist.txt

Using an alternative delimiter is particularly convenient when working with filesystem paths because it avoids having to escape every forward slash within the expression. By default, sed interprets patterns using POSIX Basic Regular Expressions (BRE). Extended Regular Expression (ERE) syntax can be enabled using the -E option, where supported. This allows investigators to construct more readable expressions involving grouping, alternation, and repetition. An important consideration for forensic examination is how sed handles its input and output. By default, it writes transformed content to standard output (stdout) without modifying the original input file. This makes it suitable for incorporating non-destructive transformations into command-line pipelines. For example:

tcpdump -nn -r magnitude_1hr.pcap | sed 's/:/,/g'

This demonstrates how sed can be placed downstream of another utility to transform its textual output before further processing. However, investigators must understand the scope of the substitution: it modifies every matching colon in each line, not just delimiters associated with a particular network field. Such a transformation should therefore only be used when the intended text structure is known.

The -i option requires particular caution. It enables in-place editing, modifying the input file rather than simply writing transformed content to standard output. Using this option directly against original evidence files can compromise evidence integrity and undermine the reproducibility of an examination. Investigators should preserve original evidence, conduct any necessary transformations on verified working copies, and document the commands and resulting outputs. For forensic workflows, the default standard-output behaviour of sed is generally preferable because it supports controlled, non-destructive text transformation and seamless integration with other UNIX utilities.

Regular expressions use special characters, known as metacharacters, to describe search patterns and identify specific sequences of text. Understanding their behaviour is important when constructing filtering, extraction, and transformation commands for network forensic investigations.

One common source of confusion is the difference between shell wildcard expressions and regular-expression syntax. Although both are used for pattern matching, they follow different rules. In shell filename expansion (globbing), the asterisk (*) represents any sequence of characters. In regular expressions, however, the asterisk is a quantifier that modifies the preceding expression, meaning zero or more occurrences of that element. For example, the shell glob *.txt, which matches filenames ending in .txt, can be represented by the regular expression .*\.txt$. Here, the period (.) matches any character, the escaped period (\.) represents a literal full stop, and the dollar sign ($) anchors the match to the end of the input line.

Regular-expression quantifiers such as *, +, ?, and {n,m} control how many times an expression can occur. In backtracking-based engines such as PCRE, these quantifiers are greedy by default. A greedy quantifier attempts to consume as much input as possible while still allowing the complete pattern to match. For example, consider the following expression:

<.*>

Given the input:

<title>Network Forensics</title>

A greedy match can consume the entire string, from the first opening angle bracket to the final closing angle bracket. This occurs because .* can match any sequence of characters, including additional angle brackets, and the engine attempts to extend the match as far as possible while satisfying the remaining pattern.

An important distinction exists between regular-expression matching models. POSIX regular-expression implementations, such as those used by traditional grep, sed, and awk, follow a leftmost-longest matching rule. They select the match beginning at the earliest possible position and, at that position, prefer the longest overall match. PCRE-based engines generally use leftmost-first matching. They select the earliest possible starting position and then follow the order in which alternatives and quantifiers are evaluated, using backtracking when necessary. Consequently, alternative ordering can affect which substring is returned, even when several alternatives could match. These differences are important when transferring regular expressions between command-line utilities and security analysis platforms. A pattern that produces a particular extraction result in one engine may behave differently in another.

Greedy matching does not ordinarily change whether grep identifies a line as matching a pattern. However, it can significantly affect the extracted substring or replacement result when regular expressions are used for text processing with tools such as sed, awk, or PCRE-based utilities. One way to control the extent of a match is to use a negated character class. For example:

<[^>]*>

Here, [^>] matches any character except a closing angle bracket. The expression therefore stops at the first closing bracket rather than consuming subsequent bracketed content. In PCRE, another option is to use a non-greedy quantifier:

<.*?>

The additional question mark modifies the * quantifier, causing it to match as few characters as possible while still satisfying the complete expression. For network forensic analysts, understanding these matching behaviours is essential when constructing reliable extraction patterns for logs, protocol metadata, HTTP headers, and other textual artefacts. Regular expressions should always be tested in the actual engine that will execute them, particularly when the resulting matches are used to support investigative findings.

Regular expressions describe patterns of text using special characters known as metacharacters. These characters define matching conditions that allow investigators to identify, filter, extract, and transform textual artefacts during forensic examination. The following is not an exhaustive list of regular-expression metacharacters. Their interpretation depends on their position within an expression and the regular-expression flavor implemented by the processing utility. The examples below primarily follow POSIX Extended Regular Expression (ERE) syntax, as supported by grep -E, sed -E, and awk.

Common regular-expression metacharacters include:

• Period (.): Matches any single character other than a newline. For example, 192.168.1.1 matches periods as wildcard characters rather than necessarily representing literal dots. To match an actual period, escape it as \..

• Square brackets ([ ]): Define a bracket expression that matches any single character from a specified set. For example, [0-9] matches a decimal digit, while [aeiou] matches any lowercase vowel. A caret (^) placed immediately after the opening bracket negates the character set. For example, [^>] matches any single character except a closing angle bracket. Note that characters such as -, ^, and ] have special positional rules within bracket expressions.

• Caret (^): Outside a bracket expression, anchors a match to the beginning of a line. For example, ^GET matches lines beginning with the string GET.

• Dollar sign ($): Anchors a match to the end of a line. For example, 404$ matches lines ending with the value 404.

• Asterisk (*): Matches zero or more occurrences of the immediately preceding regular-expression element. For example, ab*c matches ac, abc, abbc, and other equivalent sequences.

• Plus sign (+): Matches one or more occurrences of the preceding element. For example, ab+c matches abc, abbc, and other sequences containing at least one b.

• Question mark (?): Matches zero or one occurrence of the preceding element. For example, colou?r matches both color and colour.

The quantifiers *, +, and ? apply to the immediately preceding element. When the intention is to repeat an entire sequence rather than a single character, parentheses can be used for grouping. For example, (ab)+ matches one or more consecutive occurrences of the sequence ab.

An important distinction exists between Extended and Basic Regular Expression syntax. In POSIX ERE, operators such as +, ?, and parentheses are used without escaping. In POSIX Basic Regular Expressions (BRE), as used by default in traditional grep and sed, their corresponding operator forms differ. In particular, \+ and \? are extensions supported by GNU implementations but are not portable POSIX BRE operators. A quantifier should also not be confused with the shell's filename-expansion wildcard. Unlike shell globbing, where * can represent an arbitrary sequence of characters, a regular-expression quantifier must follow an element that it modifies. When combined with the period metacharacter, these quantifiers provide two particularly useful expressions:

• .* matches any sequence of zero or more characters.

• .+ matches any sequence of one or more characters.

These expressions are frequently used in text extraction and pattern matching. However, their behaviour should be considered carefully, particularly when greedy matching can cause an expression to consume more text than intended. For forensic analysts, understanding the position, scope, and interpretation of regular-expression metacharacters is essential when developing reliable searches across network logs, packet summaries, authentication records, and other textual evidence. Patterns should always be validated against the regular-expression engine in which they will be executed.