pw_tokenizer: Migrate the Base64 docs (SEED-0102)
Change-Id: I0c0ef8a7fc013e6b5ac6529850584934ce8f7e96
Reviewed-on: https://pigweed-review.googlesource.com/c/pigweed/pigweed/+/156531
Reviewed-by: Carlos Chinchilla <cachinchilla@google.com>
Reviewed-by: Kayce Basques <kayce@google.com>
Pigweed-Auto-Submit: Kayce Basques <kayce@google.com>
Commit-Queue: Auto-Submit <auto-submit@pigweed-service-accounts.iam.gserviceaccount.com>
Presubmit-Verified: CQ Bot Account <pigweed-scoped@luci-project-accounts.iam.gserviceaccount.com>
diff --git a/pw_tokenizer/design.rst b/pw_tokenizer/design.rst
index e5cf706..24dbedb 100644
--- a/pw_tokenizer/design.rst
+++ b/pw_tokenizer/design.rst
@@ -91,6 +91,35 @@
| When viewed | ``"Battery state: CHARGING; battery voltage: 3989 mV"`` | |
+------------------+-----------------------------------------------------------+---------+
+.. _module-pw_tokenizer-base64-format:
+
+-------------
+Base64 format
+-------------
+The tokenizer encodes messages to a compact binary representation. Applications
+may desire a textual representation of tokenized strings. This makes it easy to
+use tokenized messages alongside plain text messages, but comes at a small
+efficiency cost: encoded Base64 messages occupy about 4/3 (133%) as much memory
+as binary messages.
+
+The Base64 format is comprised of a ``$`` character followed by the
+Base64-encoded contents of the tokenized message. For example, consider
+tokenizing the string ``This is an example: %d!`` with the argument -1. The
+string's token is 0x4b016e66.
+
+.. code-block:: text
+
+ Source code: PW_LOG("This is an example: %d!", -1);
+
+ Plain text: This is an example: -1! [23 bytes]
+
+ Binary: 66 6e 01 4b 01 [ 5 bytes]
+
+ Base64: $Zm4BSwE= [ 9 bytes]
+
+See :ref:`module-pw_tokenizer-base64-guides` for guidance on encoding and
+decoding Base64 messages.
+
.. _module-pw_tokenizer-token-databases:
---------------
diff --git a/pw_tokenizer/docs.rst b/pw_tokenizer/docs.rst
index 6c8d968..be3898b 100644
--- a/pw_tokenizer/docs.rst
+++ b/pw_tokenizer/docs.rst
@@ -157,112 +157,11 @@
detokenization and :ref:`module-pw_tokenizer-detokenization-guides` for detailed
instructions on how to do detokenization in different programming languages.
-.. _module-pw_tokenizer-base64-format:
-
-------------
Base64 format
-------------
-The tokenizer encodes messages to a compact binary representation. Applications
-may desire a textual representation of tokenized strings. This makes it easy to
-use tokenized messages alongside plain text messages, but comes at a small
-efficiency cost: encoded Base64 messages occupy about 4/3 (133%) as much memory
-as binary messages.
-
-The Base64 format is comprised of a ``$`` character followed by the
-Base64-encoded contents of the tokenized message. For example, consider
-tokenizing the string ``This is an example: %d!`` with the argument -1. The
-string's token is 0x4b016e66.
-
-.. code-block:: text
-
- Source code: PW_LOG("This is an example: %d!", -1);
-
- Plain text: This is an example: -1! [23 bytes]
-
- Binary: 66 6e 01 4b 01 [ 5 bytes]
-
- Base64: $Zm4BSwE= [ 9 bytes]
-
-Encoding
-========
-To encode with the Base64 format, add a call to
-``pw::tokenizer::PrefixedBase64Encode`` or ``pw_tokenizer_PrefixedBase64Encode``
-in the tokenizer handler function. For example,
-
-.. code-block:: cpp
-
- void TokenizedMessageHandler(const uint8_t encoded_message[],
- size_t size_bytes) {
- pw::InlineBasicString base64 = pw::tokenizer::PrefixedBase64Encode(
- pw::span(encoded_message, size_bytes));
-
- TransmitLogMessage(base64.data(), base64.size());
- }
-
-Decoding
-========
-The Python ``Detokenizer`` class supports decoding and detokenizing prefixed
-Base64 messages with ``detokenize_base64`` and related methods.
-
-.. tip::
- The Python detokenization tools support recursive detokenization for prefixed
- Base64 text. Tokenized strings found in detokenized text are detokenized, so
- prefixed Base64 messages can be passed as ``%s`` arguments.
-
- For example, the tokenized string for "Wow!" is ``$RhYjmQ==``. This could be
- passed as an argument to the printf-style string ``Nested message: %s``, which
- encodes to ``$pEVTYQkkUmhZam1RPT0=``. The detokenizer would decode the message
- as follows:
-
- ::
-
- "$pEVTYQkkUmhZam1RPT0=" → "Nested message: $RhYjmQ==" → "Nested message: Wow!"
-
-Base64 decoding is supported in C++ or C with the
-``pw::tokenizer::PrefixedBase64Decode`` or ``pw_tokenizer_PrefixedBase64Decode``
-functions.
-
-Investigating undecoded messages
-================================
-Tokenized messages cannot be decoded if the token is not recognized. The Python
-package includes the ``parse_message`` tool, which parses tokenized Base64
-messages without looking up the token in a database. This tool attempts to guess
-the types of the arguments and displays potential ways to decode them.
-
-This tool can be used to extract argument information from an otherwise unusable
-message. It could help identify which statement in the code produced the
-message. This tool is not particularly helpful for tokenized messages without
-arguments, since all it can do is show the value of the unknown token.
-
-The tool is executed by passing Base64 tokenized messages, with or without the
-``$`` prefix, to ``pw_tokenizer.parse_message``. Pass ``-h`` or ``--help`` to
-see full usage information.
-
-Example
--------
-.. code-block::
-
- $ python -m pw_tokenizer.parse_message '$329JMwA=' koSl524TRkFJTEVEX1BSRUNPTkRJVElPTgJPSw== --specs %s %d
-
- INF Decoding arguments for '$329JMwA='
- INF Binary: b'\xdfoI3\x00' [df 6f 49 33 00] (5 bytes)
- INF Token: 0x33496fdf
- INF Args: b'\x00' [00] (1 bytes)
- INF Decoding with up to 8 %s or %d arguments
- INF Attempt 1: [%s]
- INF Attempt 2: [%d] 0
-
- INF Decoding arguments for '$koSl524TRkFJTEVEX1BSRUNPTkRJVElPTgJPSw=='
- INF Binary: b'\x92\x84\xa5\xe7n\x13FAILED_PRECONDITION\x02OK' [92 84 a5 e7 6e 13 46 41 49 4c 45 44 5f 50 52 45 43 4f 4e 44 49 54 49 4f 4e 02 4f 4b] (28 bytes)
- INF Token: 0xe7a58492
- INF Args: b'n\x13FAILED_PRECONDITION\x02OK' [6e 13 46 41 49 4c 45 44 5f 50 52 45 43 4f 4e 44 49 54 49 4f 4e 02 4f 4b] (24 bytes)
- INF Decoding with up to 8 %s or %d arguments
- INF Attempt 1: [%d %s %d %d %d] 55 FAILED_PRECONDITION 1 -40 -38
- INF Attempt 2: [%d %s %s] 55 FAILED_PRECONDITION OK
-
-Detokenizing command line utilities
------------------------------------
-See :ref:`module-pw_tokenizer-cli-detokenizing`.
+See :ref:`module-pw_tokenizer-base64-format` for a conceptual overview and
+:ref:`module-pw_tokenizer-base64-guides` for usage.
.. toctree::
:hidden:
diff --git a/pw_tokenizer/guides.rst b/pw_tokenizer/guides.rst
index 465aa21..0c461c0 100644
--- a/pw_tokenizer/guides.rst
+++ b/pw_tokenizer/guides.rst
@@ -60,6 +60,95 @@
``pw_tokenizer_zephyr.ld`` which is added to the end of the linker file
via a call to ``zephyr_linker_sources(SECTIONS ...)``.
+.. _module-pw_tokenizer-base64-guides:
+
+-------------
+Base64 guides
+-------------
+See :ref:`module-pw_tokenizer-base64-format` for a conceptual overview of
+Base64.
+
+Encoding Base64
+===============
+To encode with the Base64 format, add a call to
+``pw::tokenizer::PrefixedBase64Encode`` or ``pw_tokenizer_PrefixedBase64Encode``
+in the tokenizer handler function. For example,
+
+.. code-block:: cpp
+
+ void TokenizedMessageHandler(const uint8_t encoded_message[],
+ size_t size_bytes) {
+ pw::InlineBasicString base64 = pw::tokenizer::PrefixedBase64Encode(
+ pw::span(encoded_message, size_bytes));
+
+ TransmitLogMessage(base64.data(), base64.size());
+ }
+
+Decoding Base64
+===============
+The Python ``Detokenizer`` class supports decoding and detokenizing prefixed
+Base64 messages with ``detokenize_base64`` and related methods.
+
+.. tip::
+ The Python detokenization tools support recursive detokenization for prefixed
+ Base64 text. Tokenized strings found in detokenized text are detokenized, so
+ prefixed Base64 messages can be passed as ``%s`` arguments.
+
+ For example, the tokenized string for "Wow!" is ``$RhYjmQ==``. This could be
+ passed as an argument to the printf-style string ``Nested message: %s``, which
+ encodes to ``$pEVTYQkkUmhZam1RPT0=``. The detokenizer would decode the message
+ as follows:
+
+ ::
+
+ "$pEVTYQkkUmhZam1RPT0=" → "Nested message: $RhYjmQ==" → "Nested message: Wow!"
+
+Base64 decoding is supported in C++ or C with the
+``pw::tokenizer::PrefixedBase64Decode`` or ``pw_tokenizer_PrefixedBase64Decode``
+functions.
+
+Investigating undecoded messages
+================================
+Tokenized messages cannot be decoded if the token is not recognized. The Python
+package includes the ``parse_message`` tool, which parses tokenized Base64
+messages without looking up the token in a database. This tool attempts to guess
+the types of the arguments and displays potential ways to decode them.
+
+This tool can be used to extract argument information from an otherwise unusable
+message. It could help identify which statement in the code produced the
+message. This tool is not particularly helpful for tokenized messages without
+arguments, since all it can do is show the value of the unknown token.
+
+The tool is executed by passing Base64 tokenized messages, with or without the
+``$`` prefix, to ``pw_tokenizer.parse_message``. Pass ``-h`` or ``--help`` to
+see full usage information.
+
+Example
+-------
+.. code-block::
+
+ $ python -m pw_tokenizer.parse_message '$329JMwA=' koSl524TRkFJTEVEX1BSRUNPTkRJVElPTgJPSw== --specs %s %d
+
+ INF Decoding arguments for '$329JMwA='
+ INF Binary: b'\xdfoI3\x00' [df 6f 49 33 00] (5 bytes)
+ INF Token: 0x33496fdf
+ INF Args: b'\x00' [00] (1 bytes)
+ INF Decoding with up to 8 %s or %d arguments
+ INF Attempt 1: [%s]
+ INF Attempt 2: [%d] 0
+
+ INF Decoding arguments for '$koSl524TRkFJTEVEX1BSRUNPTkRJVElPTgJPSw=='
+ INF Binary: b'\x92\x84\xa5\xe7n\x13FAILED_PRECONDITION\x02OK' [92 84 a5 e7 6e 13 46 41 49 4c 45 44 5f 50 52 45 43 4f 4e 44 49 54 49 4f 4e 02 4f 4b] (28 bytes)
+ INF Token: 0xe7a58492
+ INF Args: b'n\x13FAILED_PRECONDITION\x02OK' [6e 13 46 41 49 4c 45 44 5f 50 52 45 43 4f 4e 44 49 54 49 4f 4e 02 4f 4b] (24 bytes)
+ INF Decoding with up to 8 %s or %d arguments
+ INF Attempt 1: [%d %s %d %d %d] 55 FAILED_PRECONDITION 1 -40 -38
+ INF Attempt 2: [%d %s %s] 55 FAILED_PRECONDITION OK
+
+Detokenizing command line utilities
+-----------------------------------
+See :ref:`module-pw_tokenizer-cli-detokenizing`.
+
.. _module-pw_tokenizer-masks:
---------------------------
