# Checking multilingual paths in WordPress robots.txt rules

- Author: REBASE-SG
- Published: 2026-10-29 11:45:00
- Modified: 2026-10-29 11:45:00
- Canonical URL: https://rebase-sg.com/robots-txt-multilingual-path-encoding-en/

---

A multilingual URL can look familiar in the browser and become a string of percent signs when copied elsewhere. That leaves a practical question when writing a robots.txt rule: which representation should the site owner use, and what does a difference between them actually mean?



Google’s October 23, 2026 crawling documentation update adds a recommendation about percent-encoding in rule paths. This article examines that clarification for WordPress maintenance. It is not an announcement of a new crawling feature launching today.



Compare representations before changing addresses



The current specification guidance explains that Google accepts raw UTF-8 rule paths and canonicalizes them to percent-encoded forms for comparison. It recommends encoding to make processing easier for third-party tools and services. Finding non-ASCII characters in a rule is therefore insufficient grounds to declare it unreadable by Google.



The illustration uses /café/ and /caf%C3%A9/ as a small example. They represent the same path in this Google parser example. The diagram does not depict two articles, a site migration, or the results of a live crawling test.



Start a WordPress review with the actual public URL. Retyping a path from an article headline creates an unnecessary opportunity for error. A title, an editor field and the final address need not contain identical text. Put the observed address beside the proposed rule so that the comparison has a definite subject.



That distinction can save unrelated work. If a tool presents a path differently, investigate the representation before renaming the page. Moving an existing URL changes an address that visitors may already use. Checking how a rule expresses that address is a separate maintenance task.



Include a page that should remain accessible



Suppose the intended change concerns one document directory. Choose a target page that the rule should cover and a nearby page that should remain available for crawling. These give the review two questions: does the rule cover its intended target, and does it leave the neighboring resource alone?



A tool accepting the syntax does not answer both questions. Record the exact URLs, the rule being assessed and the expected result for each. The record should make it possible for another maintainer to repeat the check.



Also record which tool produced the result. Reading a file, parsing a directive and evaluating a crawler-specific rule set are different checks. A successful result needs a clear description of what was checked. It cannot establish that every AI crawler implements the same behavior.



When two tools disagree about an encoded path, trace the value through copying, saving and exporting before broadening the rule. Preserve the observed differences. Editing several settings until the outputs look similar may remove the evidence needed to understand the original problem.



Verify what the site actually serves



The editing interface may belong to a WordPress plugin, while the delivered file may depend on the hosting setup. After a necessary change, open the public robots.txt and inspect the final output. A saved field in an administration screen is only one step in that process. Keep the previous version and the specific change so that an unintended scope can be corrected.





The scope of the source matters, too. Google’s explanation describes its path matching and a recommendation for interoperability with tools. Consult another service’s own documentation before assuming identical handling. This clarification gives no assurance of indexing, AI citation or additional traffic.



If the current rules have no problem to solve, no edit may be necessary. Where a multilingual path is under investigation, keep its public URL, rule, expected scope and tool result together. That record gives the next maintainer something more useful than a note saying “encoding fixed.”



中文版：把中文路径写进 robots.txt：一次编码核对怎么做。