Regular expressions and text guides
Deduplicate and sort lines: define equality first
Decide whether case, spaces and leading zeros matter, then deduplicate without silently changing meaning.
Define identical lines
Exact text equality treats Ada, ada and Ada followed by a space as different values. Trimming or folding case changes that rule.
Use raw deduplication when exact bytes or case-sensitive identifiers matter. Apply normalization only if the consumer’s contract calls for it.
Ada
ada
Ada
Exact deduplication preserves: Ada, adaChoose text or numeric order
Lexicographic sorting and numeric sorting differ. As text, 10 comes before 2; as numbers, 2 comes before 10.
IDs with leading zeros may need text order even if they contain only digits. Do not convert opaque identifiers to numbers just to sort them.
Text order: 1, 10, 2
Numeric order: 1, 2, 10Order may carry information
Sorting destroys the original sequence, which may represent log chronology, rank or user preference. Deduplication can also remove repeated events.
For a membership list, stable first-occurrence deduplication may be appropriate. For an event stream, retain duplicates and compare counts instead.
Measure the change
Record input and output line counts, then inspect blank lines and one duplicate group. Use Text Diff to confirm only intended changes occurred.
Nav Tools options have distinct behaviors; choose them explicitly. A shorter output is not automatically a better or more accurate dataset.