A one-value edit should not turn an entire text file into a formatting change. When that happens, the replacement operation may be correct while the read-and-write path is doing more work than intended. Decoding, newline translation, and encoding can each change how the file is represented, even when an editor displays nearly identical text.

Python UTF-8 file editing needs an explicit contract for content, the byte order mark, and line endings. Being able to read a file is not proof that writing it back will preserve its bytes. This guide addresses the informational question of how to make a narrow edit without silently changing unrelated formatting. It is aimed at developers maintaining build settings, text fixtures, and localization assets.

The evidence has three layers: a local engineering note identifies the failure pattern, Python documentation defines the API behavior, and a small executable lab checks synthetic examples. None of those layers should be presented as a substitute for the others.

Define whether the contract protects text or bytes

Text equivalence and byte equivalence are different acceptance criteria. If an edit is allowed to change only one designated fragment, treat everything else, including the opening signature and each line ending, as an invariant. Read the original bytes before choosing a decoding and writing strategy.

An editor can display LF and CRLF as identical line breaks. It can also hide an opening UTF-8 signature. A script that operates only on the displayed meaning may therefore satisfy a human reading test while violating the file's formatting contract. The first engineering decision is not which function to call, but which differences are permitted.

Write down three boundaries before implementing the edit: the exact fragment that may change, the surrounding representation that must remain stable, and the source file that must remain untouched during review. The lab below uses small synthetic files with known UTF-8 content. It does not claim to detect arbitrary encodings or preserve filesystem metadata.

If a project actually wants to normalize formatting, that is a separate change with a separate review. Combining newline normalization with a functional edit makes the result harder to inspect and leaves ambiguous expectations for the next automation run. Preservation and normalization are both valid goals, but they should not happen accidentally in the same operation.

Understand what utf-8-sig changes during a round trip

The ordinary UTF-8 decoder preserves an opening signature as a character, while the utf-8-sig decoder skips the signature when it appears at the beginning. Encoding with utf-8-sig adds a signature. Consequently, accepting both input forms does not imply preserving their original signature state on output.

Python's codecs documentation describes the UTF-8 byte order mark as a signature rather than a mechanism for choosing byte order. UTF-8 does not need an endian marker. The important distinction for an editing tool is between decoding behavior and encoding behavior, not simply whether both operations use the same codec name.

Consider two fixtures. The first begins with the signature and the second does not. In this lab, decoding the first with utf-8-sig and encoding it with the same codec reproduces the original bytes. Applying that sequence to the second fixture adds bytes that were not there before. Testing only the first case would conceal the second outcome.

Do not solve this by deleting every invisible character. The opening signature and characters elsewhere in the document are different concerns. For a narrow replacement of a known ASCII fragment inside a known UTF-8 file, retaining the original bytes avoids an unnecessary representation change. For a text-aware transformation, record whether the original had a signature and make the output policy explicit.

Control newline translation independently of encoding

Setting the encoding to UTF-8 does not disable text-mode newline processing. With the default reading policy, Python normalizes several line-ending forms into a newline character. To preserve the original separators through a text round trip, specify the newline policy on both the read and write operations.

The Python 3.11 open() documentation defines those operations separately. On reading, newline=None enables universal newline handling and translates recognized endings into \n. With newline='', universal newline recognition still occurs, but the returned endings are untranslated. On writing, newline=None translates \n into the system's default line separator, whereas newline='' performs no newline translation.

This is why the statement “text mode always turns CRLF files into LF files” is too broad. It confuses the string returned by a read with the bytes eventually written to disk. The write parameters and platform still matter. The lab uses a mixed-ending fixture and compares its actual bytes rather than drawing conclusions from how the terminal displays it.

Once the reading stage has collapsed different endings into one character, choosing CRLF on output cannot reconstruct which lines originally used LF or a standalone CR. Preservation must begin before that information is lost. A related failure happens when a replacement searches for a CRLF fragment in a file that contains LF instead. No exception may occur, but nothing has been edited. Match-count checks are essential for detecting that outcome.

Guard narrow replacements before writing a candidate

Byte replacement can preserve unrelated data in a small, well-understood UTF-8 fixture, but only when the target is unambiguous. Reject empty targets, missing targets, and repeated targets before creating output. Validate the encoding contract and compare the candidate bytes rather than assuming a successful write proves correctness.

The patch_one helper below deliberately does very little. It requires exactly one occurrence of a nonempty old fragment and uses strict UTF-8 decoding before and after replacement. Its purpose is to enforce the constraints of this lab, not to become a universal configuration editor. It does not understand nested JSON fields, XML elements, or comments.

A missing match can indicate an unexpected file revision, a different line ending, or the wrong fixture. Multiple matches mean the requested target is insufficiently specific. Both conditions should stop the operation rather than trigger a broader replacement. The malformed-input test also makes sure invalid UTF-8 is rejected instead of quietly losing data through an ignore policy.

The writing boundary is equally narrow. The lab creates a separate candidate with xb, which fails if the destination already exists. It then reads that candidate back and checks that the source bytes are unchanged. This is not a general concurrency protocol or a durability guarantee. It is simply an inspectable, isolated editing exercise that does not overwrite an existing project file.

Verification: run the fixed-version local lab

Three isolated tests make the important behaviors observable: signature handling, newline translation, and guarded replacement. Each test uses in-memory data or files created inside a temporary directory. The pass conditions are exact byte comparisons and expected exceptions, so no external service or pre-existing dataset is required.

Prerequisites: CPython 3.11.15, its standard library, and permission to create temporary files in a local test environment. Save the complete program below as encoding_lab.py. No third-party packages are required. The script checks the patch version to hold this reproduction constant; that check is not a claim that these APIs are unavailable in other releases.

  1. Run python --version, then python encoding_lab.py EncodingTests.test_bom. Pass means the marked fixture round-trips exactly and the initially unmarked fixture does not remain identical after encoding with utf-8-sig. An assertion failure is a failed check, even if the displayed text looks correct.
  2. Run python encoding_lab.py EncodingTests.test_newlines. Pass means the default read normalizes the mixed endings, while explicitly untranslated reading and writing preserve the fixture bytes. A line count or screenshot is not an adequate substitute for the byte comparison.
  3. Run python encoding_lab.py EncodingTests.test_guarded_patch, then python encoding_lab.py for the whole suite. Pass requires the expected candidate, an unchanged source, rejection of zero and multiple matches, and rejection of malformed UTF-8. Test failures and a mismatched interpreter version terminate with a nonzero exit code.
import codecs
import sys
import tempfile
import unittest
from pathlib import Path


def patch_one(raw, old, new):
    raw.decode('utf-8', errors='strict')
    if not old or raw.count(old) != 1:
        raise ValueError('expected exactly one match')
    result = raw.replace(old, new, 1)
    result.decode('utf-8', errors='strict')
    return result


class EncodingTests(unittest.TestCase):
    def test_bom(self):
        raw = codecs.BOM_UTF8 + 'name=\u6e2c\u8a66\r\n'.encode('utf-8')
        self.assertTrue(raw.decode('utf-8').startswith('\ufeff'))
        self.assertFalse(raw.decode('utf-8-sig').startswith('\ufeff'))
        self.assertEqual(raw.decode('utf-8-sig').encode('utf-8-sig'), raw)
        plain = b'name=sample\n'
        self.assertNotEqual(plain.decode('utf-8-sig').encode('utf-8-sig'), plain)

    def test_newlines(self):
        raw = b'a\r\nb\nc\r'
        with tempfile.TemporaryDirectory() as folder:
            source = Path(folder) / 'source.txt'
            target = Path(folder) / 'copy.txt'
            source.write_bytes(raw)
            with open(source, encoding='utf-8', newline=None) as stream:
                self.assertEqual(stream.read(), 'a\nb\nc\n')
            with open(source, encoding='utf-8', newline='') as stream:
                text = stream.read()
            with open(target, 'w', encoding='utf-8', newline='') as stream:
                stream.write(text)
            self.assertEqual(target.read_bytes(), raw)

    def test_guarded_patch(self):
        old, new = b'mode=old', b'mode=new'
        raw = codecs.BOM_UTF8 + b'mode=old\r\nlabel=' + '\u6e2c\u8a66'.encode('utf-8') + b'\n'
        result = patch_one(raw, old, new)
        self.assertEqual(result, codecs.BOM_UTF8 + b'mode=new\r\nlabel=' + '\u6e2c\u8a66'.encode('utf-8') + b'\n')
        for bad in (b'missing\n', old + b'\n' + old):
            with self.assertRaises(ValueError):
                patch_one(bad, old, new)
        with self.assertRaises(UnicodeDecodeError):
            patch_one(b'\xffmode=old', old, new)
        with tempfile.TemporaryDirectory() as folder:
            source = Path(folder) / 'source.txt'
            target = Path(folder) / 'candidate.txt'
            source.write_bytes(raw)
            with open(target, 'xb') as stream:
                stream.write(result)
            self.assertEqual(source.read_bytes(), raw)
            self.assertEqual(target.read_bytes(), result)


if __name__ == '__main__':
    if sys.version_info[:3] != (3, 11, 15):
        raise SystemExit('Use CPython 3.11.15 for this fixed-version lab')
    unittest.main(verbosity=2)

Observed scheduler result: running python encoding_lab.py on CPython 3.11.15 returned ok for all three tests, the test runner reported OK, and the process exit code was 0. This is a correctness result for these synthetic cases. It is not a large-file test, a cross-platform execution matrix, or a performance benchmark.

The references belong to the Python 3.11 documentation branch, while the interpreter used for this execution was specifically 3.11.15. Record those facts separately. A documentation branch can be updated without changing which runtime produced a particular test result.

Turn the checks into an editing acceptance checklist

A maintenance tool should be considered complete only when the intended fragment changed, the protected representation stayed intact, and invalid inputs were refused. Put those conditions into repeatable checks before changing the tool or its execution environment, rather than reconstructing them after a noisy diff appears.

  • Input boundary: establish the encoding and decide whether loading the whole file is appropriate. The lab assumes small UTF-8 fixtures. An unknown or oversized input requires a separate approach rather than silent guessing.
  • Representation evidence: preserve the source bytes and inspect the signature and newline policy. Readable glyphs in a console are useful context, but they are not proof of the underlying representation.
  • Selection boundary: require one clearly defined fragment. If the edit needs structural selection, use an appropriate parser and independently verify whether its serializer preserves the formatting properties you care about.
  • Output evidence: read the candidate back and compare the untouched source. File existence and the return from a writing function demonstrate less than these checks do.
  • Failure coverage: keep the missing-target, repeated-target, and malformed-input cases. Testing only a successful replacement misses both silent non-edits and overly broad edits.

For small files, a full byte comparison gives a direct acceptance condition. A recorded digest is useful for identifying a source version, but it does not explain where a difference occurred. During diagnosis, pair a precise diff with fixtures that separate signature behavior from newline behavior. This keeps the explanation tied to observable changes instead of a general suspicion about encoding.

No speedup or time-saving claim follows from this example. Byte replacement is chosen here because its scope can be constrained and tested, not because it has been shown to outperform a parser. Larger files need their own memory and elapsed-time measurements, and a streaming implementation would need additional boundary tests. Those are separate engineering tasks, not results established by this lab.

Evidence boundaries and the final decision

The source note provides a useful warning about format changes during text editing, but the published conclusion is restricted by official API documentation and synthetic tests. Environment-specific details and unrelated operational topics are excluded, and individual observations are not promoted into universal claims about every platform or tool.

The selected source is 軟體開發/專案開發問題集/Windows環境陷阱/Windows環境陷阱.md under DevelopmentNote. It was read in full. Both language editions use the same source evidence and the same executed lab. The editorial treatment separates the codec's signature behavior from the file wrapper's newline behavior instead of repeating a simplified rule that text mode always changes a particular format.

The following official references were opened successfully and checked against the claims above:

Preserve evidence before changing content. For a small file with a known UTF-8 contract, explicit formatting rules, a guarded replacement, and candidate readback are easier to verify than a guessed encoding repair or reliance on defaults. Run the isolated checks first, then adopt only the acceptance conditions that match the format your own editing tool is meant to preserve.