| ==================== | |
| Python on iOS README | |
| ==================== | |
| :Authors: | |
| Russell Keith-Magee (2023-11) | |
| This document provides a quick overview of some iOS specific features in the | |
| Python distribution. | |
| These instructions are only needed if you're planning to compile Python for iOS | |
| yourself. Most users should *not* need to do this. If you're looking to | |
| experiment with writing an iOS app in Python, tools such as `BeeWare's Briefcase | |
| <https://briefcase.readthedocs.io>`__ and `Kivy's Buildozer | |
| <https://buildozer.readthedocs.io>`__ will provide a much more approachable | |
| user experience. | |
| Compilers for building on iOS | |
| ============================= | |
| Building for iOS requires the use of Apple's Xcode tooling. It is strongly | |
| recommended that you use the most recent stable release of Xcode. This will | |
| require the use of the most (or second-most) recently released macOS version, | |
| as Apple does not maintain Xcode for older macOS versions. The Xcode Command | |
| Line Tools are not sufficient for iOS development; you need a *full* Xcode | |
| install. | |
| If you want to run your code on the iOS simulator, you'll also need to install | |
| an iOS Simulator Platform. You should be prompted to select an iOS Simulator | |
| Platform when you first run Xcode. Alternatively, you can add an iOS Simulator | |
| Platform by selecting an open the Platforms tab of the Xcode Settings panel. | |
| iOS specific arguments to configure | |
| =================================== | |
| * ``--enable-framework[=DIR]`` | |
| This argument specifies the location where the Python.framework will be | |
| installed. If ``DIR`` is not specified, the framework will be installed into | |
| a subdirectory of the ``iOS/Frameworks`` folder. | |
| This argument *must* be provided when configuring iOS builds. iOS does not | |
| support non-framework builds. | |
| * ``--with-framework-name=NAME`` | |
| Specify the name for the Python framework; defaults to ``Python``. | |
| .. admonition:: Use this option with care! | |
| Unless you know what you're doing, changing the name of the Python | |
| framework on iOS is not advised. If you use this option, you won't be able | |
| to run the ``make testios`` target without making significant manual | |
| alterations, and you won't be able to use any binary packages unless you | |
| compile them yourself using your own framework name. | |
| Building Python on iOS | |
| ====================== | |
| ABIs and Architectures | |
| ---------------------- | |
| iOS apps can be deployed on physical devices, and on the iOS simulator. Although | |
| the API used on these devices is identical, the ABI is different - you need to | |
| link against different libraries for an iOS device build (``iphoneos``) or an | |
| iOS simulator build (``iphonesimulator``). | |
| Apple uses the ``XCframework`` format to allow specifying a single dependency | |
| that supports multiple ABIs. An ``XCframework`` is a wrapper around multiple | |
| ABI-specific frameworks that share a common API. | |
| iOS can also support different CPU architectures within each ABI. At present, | |
| there is only a single supported architecture on physical devices - ARM64. | |
| However, the *simulator* supports 2 architectures - ARM64 (for running on Apple | |
| Silicon machines), and x86_64 (for running on older Intel-based machines). | |
| To support multiple CPU architectures on a single platform, Apple uses a "fat | |
| binary" format - a single physical file that contains support for multiple | |
| architectures. It is possible to compile and use a "thin" single architecture | |
| version of a binary for testing purposes; however, the "thin" binary will not be | |
| portable to machines using other architectures. | |
| Building a single-architecture framework | |
| ---------------------------------------- | |
| The Python build system will create a ``Python.framework`` that supports a | |
| *single* ABI with a *single* architecture. Unlike macOS, iOS does not allow a | |
| framework to contain non-library content, so the iOS build will produce a | |
| ``bin`` and ``lib`` folder in the same output folder as ``Python.framework``. | |
| The ``lib`` folder will be needed at runtime to support the Python library. | |
| If you want to use Python in a real iOS project, you need to produce multiple | |
| ``Python.framework`` builds, one for each ABI and architecture. iOS builds of | |
| Python *must* be constructed as framework builds. To support this, you must | |
| provide the ``--enable-framework`` flag when configuring the build. The build | |
| also requires the use of cross-compilation. The minimal commands for building | |
| Python for the ARM64 iOS simulator will look something like:: | |
| $ export PATH="$(pwd)/iOS/Resources/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin" | |
| $ ./configure \ | |
| --enable-framework \ | |
| --host=arm64-apple-ios-simulator \ | |
| --build=arm64-apple-darwin \ | |
| --with-build-python=/path/to/python.exe | |
| $ make | |
| $ make install | |
| In this invocation: | |
| * ``iOS/Resources/bin`` has been added to the path, providing some shims for the | |
| compilers and linkers needed by the build. Xcode requires the use of ``xcrun`` | |
| to invoke compiler tooling. However, if ``xcrun`` is pre-evaluated and the | |
| result passed to ``configure``, these results can embed user- and | |
| version-specific paths into the sysconfig data, which limits the portability | |
| of the compiled Python. Alternatively, if ``xcrun`` is used *as* the compiler, | |
| it requires that compiler variables like ``CC`` include spaces, which can | |
| cause significant problems with many C configuration systems which assume that | |
| ``CC`` will be a single executable. | |
| To work around this problem, the ``iOS/Resources/bin`` folder contains some | |
| wrapper scripts that present as simple compilers and linkers, but wrap | |
| underlying calls to ``xcrun``. This allows configure to use a ``CC`` | |
| definition without spaces, and without user- or version-specific paths, while | |
| retaining the ability to adapt to the local Xcode install. These scripts are | |
| included in the ``bin`` directory of an iOS install. | |
| These scripts will, by default, use the currently active Xcode installation. | |
| If you want to use a different Xcode installation, you can use | |
| ``xcode-select`` to set a new default Xcode globally, or you can use the | |
| ``DEVELOPER_DIR`` environment variable to specify an Xcode install. The | |
| scripts will use the default ``iphoneos``/``iphonesimulator`` SDK version for | |
| the select Xcode install; if you want to use a different SDK, you can set the | |
| ``IOS_SDK_VERSION`` environment variable. (e.g, setting | |
| ``IOS_SDK_VERSION=17.1`` would cause the scripts to use the ``iphoneos17.1`` | |
| and ``iphonesimulator17.1`` SDKs, regardless of the Xcode default.) | |
| The path has also been cleared of any user customizations. A common source of | |
| bugs is for tools like Homebrew to accidentally leak macOS binaries into an iOS | |
| build. Resetting the path to a known "bare bones" value is the easiest way to | |
| avoid these problems. | |
| * ``--host`` is the architecture and ABI that you want to build, in GNU compiler | |
| triple format. This will be one of: | |
| - ``arm64-apple-ios`` for ARM64 iOS devices. | |
| - ``arm64-apple-ios-simulator`` for the iOS simulator running on Apple | |
| Silicon devices. | |
| - ``x86_64-apple-ios-simulator`` for the iOS simulator running on Intel | |
| devices. | |
| * ``--build`` is the GNU compiler triple for the machine that will be running | |
| the compiler. This is one of: | |
| - ``arm64-apple-darwin`` for Apple Silicon devices. | |
| - ``x86_64-apple-darwin`` for Intel devices. | |
| * ``/path/to/python.exe`` is the path to a Python binary on the machine that | |
| will be running the compiler. This is needed because the Python compilation | |
| process involves running some Python code. On a normal desktop build of | |
| Python, you can compile a python interpreter and then use that interpreter to | |
| run Python code. However, the binaries produced for iOS won't run on macOS, so | |
| you need to provide an external Python interpreter. This interpreter must be | |
| the same version as the Python that is being compiled. To be completely safe, | |
| this should be the *exact* same commit hash. However, the longer a Python | |
| release has been stable, the more likely it is that this constraint can be | |
| relaxed - the same micro version will often be sufficient. | |
| * The ``install`` target for iOS builds is slightly different to other | |
| platforms. On most platforms, ``make install`` will install the build into | |
| the final runtime location. This won't be the case for iOS, as the final | |
| runtime location will be on a physical device. | |
| However, you still need to run the ``install`` target for iOS builds, as it | |
| performs some final framework assembly steps. The location specified with | |
| ``--enable-framework`` will be the location where ``make install`` will | |
| assemble the complete iOS framework. This completed framework can then | |
| be copied and relocated as required. | |
| For a full CPython build, you also need to specify the paths to iOS builds of | |
| the binary libraries that CPython depends on (XZ, BZip2, LibFFI and OpenSSL). | |
| This can be done by defining the ``LIBLZMA_CFLAGS``, ``LIBLZMA_LIBS``, | |
| ``BZIP2_CFLAGS``, ``BZIP2_LIBS``, ``LIBFFI_CFLAGS``, and ``LIBFFI_LIBS`` | |
| environment variables, and the ``--with-openssl`` configure option. Versions of | |
| these libraries pre-compiled for iOS can be found in `this repository | |
| <https://github.com/beeware/cpython-apple-source-deps/releases>`__. LibFFI is | |
| especially important, as many parts of the standard library (including the | |
| ``platform``, ``sysconfig`` and ``webbrowser`` modules) require the use of the | |
| ``ctypes`` module at runtime. | |
| By default, Python will be compiled with an iOS deployment target (i.e., the | |
| minimum supported iOS version) of 13.0. To specify a different deployment | |
| target, provide the version number as part of the ``--host`` argument - for | |
| example, ``--host=arm64-apple-ios15.4-simulator`` would compile an ARM64 | |
| simulator build with a deployment target of 15.4. | |
| Merge thin frameworks into fat frameworks | |
| ----------------------------------------- | |
| Once you've built a ``Python.framework`` for each ABI and and architecture, you | |
| must produce a "fat" framework for each ABI that contains all the architectures | |
| for that ABI. | |
| The ``iphoneos`` build only needs to support a single architecture, so it can be | |
| used without modification. | |
| If you only want to support a single simulator architecture, (e.g., only support | |
| ARM64 simulators), you can use a single architecture ``Python.framework`` build. | |
| However, if you want to create ``Python.xcframework`` that supports *all* | |
| architectures, you'll need to merge the ``iphonesimulator`` builds for ARM64 and | |
| x86_64 into a single "fat" framework. | |
| The "fat" framework can be constructed by performing a directory merge of the | |
| content of the two "thin" ``Python.framework`` directories, plus the ``bin`` and | |
| ``lib`` folders for each thin framework. When performing this merge: | |
| * The pure Python standard library content is identical for each architecture, | |
| except for a handful of platform-specific files (such as the ``sysconfig`` | |
| module). Ensure that the "fat" framework has the union of all standard library | |
| files. | |
| * Any binary files in the standard library, plus the main | |
| ``libPython3.X.dylib``, can be merged using the ``lipo`` tool, provide by | |
| Xcode:: | |
| $ lipo -create -output module.dylib path/to/x86_64/module.dylib path/to/arm64/module.dylib | |
| * The header files will be identical on both architectures, except for | |
| ``pyconfig.h``. Copy all the headers from one platform (say, arm64), rename | |
| ``pyconfig.h`` to ``pyconfig-arm64.h``, and copy the ``pyconfig.h`` for the | |
| other architecture into the merged header folder as ``pyconfig-x86_64.h``. | |
| Then copy the ``iOS/Resources/pyconfig.h`` file from the CPython sources into | |
| the merged headers folder. This will allow the two Python architectures to | |
| share a common ``pyconfig.h`` header file. | |
| At this point, you should have 2 Python.framework folders - one for ``iphoneos``, | |
| and one for ``iphonesimulator`` that is a merge of x86+64 and ARM64 content. | |
| Merge frameworks into an XCframework | |
| ------------------------------------ | |
| Now that we have 2 (potentially fat) ABI-specific frameworks, we can merge those | |
| frameworks into a single ``XCframework``. | |
| The initial skeleton of an ``XCframework`` is built using:: | |
| xcodebuild -create-xcframework -output Python.xcframework -framework path/to/iphoneos/Python.framework -framework path/to/iphonesimulator/Python.framework | |
| Then, copy the ``bin`` and ``lib`` folders into the architecture-specific slices of | |
| the XCframework:: | |
| cp path/to/iphoneos/bin Python.xcframework/ios-arm64 | |
| cp path/to/iphoneos/lib Python.xcframework/ios-arm64 | |
| cp path/to/iphonesimulator/bin Python.xcframework/ios-arm64_x86_64-simulator | |
| cp path/to/iphonesimulator/lib Python.xcframework/ios-arm64_x86_64-simulator | |
| Note that the name of the architecture-specific slice for the simulator will | |
| depend on the CPU architecture(s) that you build. | |
| You now have a Python.xcframework that can be used in a project. | |
| Testing Python on iOS | |
| ===================== | |
| The ``iOS/testbed`` folder that contains an Xcode project that is able to run | |
| the iOS test suite. This project converts the Python test suite into a single | |
| test case in Xcode's XCTest framework. The single XCTest passes if the test | |
| suite passes. | |
| To run the test suite, configure a Python build for an iOS simulator (i.e., | |
| ``--host=arm64-apple-ios-simulator`` or ``--host=x86_64-apple-ios-simulator`` | |
| ), specifying a framework build (i.e. ``--enable-framework``). Ensure that your | |
| ``PATH`` has been configured to include the ``iOS/Resources/bin`` folder and | |
| exclude any non-iOS tools, then run:: | |
| $ make all | |
| $ make install | |
| $ make testios | |
| This will: | |
| * Build an iOS framework for your chosen architecture; | |
| * Finalize the single-platform framework; | |
| * Make a clean copy of the testbed project; | |
| * Install the Python iOS framework into the copy of the testbed project; and | |
| * Run the test suite on an "iPhone SE (3rd generation)" simulator. | |
| While the test suite is running, Xcode does not display any console output. | |
| After showing some Xcode build commands, the console output will print ``Testing | |
| started``, and then appear to stop. It will remain in this state until the test | |
| suite completes. On a 2022 M1 MacBook Pro, the test suite takes approximately 12 | |
| minutes to run; a couple of extra minutes is required to boot and prepare the | |
| iOS simulator. | |
| On success, the test suite will exit and report successful completion of the | |
| test suite. No output of the Python test suite will be displayed. | |
| On failure, the output of the Python test suite *will* be displayed. This will | |
| show the details of the tests that failed. | |
| Debugging test failures | |
| ----------------------- | |
| The easiest way to diagnose a single test failure is to open the testbed project | |
| in Xcode and run the tests from there using the "Product > Test" menu item. | |
| To test in Xcode, you must ensure the testbed project has a copy of a compiled | |
| framework. If you've configured your build with the default install location of | |
| ``iOS/Frameworks``, you can copy from that location into the test project. To | |
| test on an ARM64 simulator, run:: | |
| $ rm -rf iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator/* | |
| $ cp -r iOS/Frameworks/arm64-iphonesimulator/* iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator | |
| To test on an x86-64 simulator, run:: | |
| $ rm -rf iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator/* | |
| $ cp -r iOS/Frameworks/x86_64-iphonesimulator/* iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator | |
| To test on a physical device:: | |
| $ rm -rf iOS/testbed/Python.xcframework/ios-arm64/* | |
| $ cp -r iOS/Frameworks/arm64-iphoneos/* iOS/testbed/Python.xcframework/ios-arm64 | |
| Alternatively, you can configure your build to install directly into the | |
| testbed project. For a simulator, use:: | |
| --enable-framework=$(pwd)/iOS/testbed/Python.xcframework/ios-arm64_x86_64-simulator | |
| For a physical device, use:: | |
| --enable-framework=$(pwd)/iOS/testbed/Python.xcframework/ios-arm64 | |
| Testing on an iOS device | |
| ^^^^^^^^^^^^^^^^^^^^^^^^ | |
| To test on an iOS device, the app needs to be signed with known developer | |
| credentials. To obtain these credentials, you must have an iOS Developer | |
| account, and your Xcode install will need to be logged into your account (see | |
| the Accounts tab of the Preferences dialog). | |
| Once the project is open, and you're signed into your Apple Developer account, | |
| select the root node of the project tree (labeled "iOSTestbed"), then the | |
| "Signing & Capabilities" tab in the details page. Select a development team | |
| (this will likely be your own name), and plug in a physical device to your | |
| macOS machine with a USB cable. You should then be able to select your physical | |
| device from the list of targets in the pulldown in the Xcode titlebar. | |
| Running specific tests | |
| ^^^^^^^^^^^^^^^^^^^^^^ | |
| As the test suite is being executed on an iOS simulator, it is not possible to | |
| pass in command line arguments to configure test suite operation. To work | |
| around this limitation, the arguments that would normally be passed as command | |
| line arguments are configured as part of the ``iOSTestbed-Info.plist`` file | |
| that is used to configure the iOS testbed app. In this file, the ``TestArgs`` | |
| key is an array containing the arguments that would be passed to ``python -m`` | |
| on the command line (including ``test`` in position 0, the name of the test | |
| module to be executed). | |
| Disabling automated breakpoints | |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| By default, Xcode will inserts an automatic breakpoint whenever a signal is | |
| raised. The Python test suite raises many of these signals as part of normal | |
| operation; unless you are trying to diagnose an issue with signals, the | |
| automatic breakpoints can be inconvenient. However, they can be disabled by | |
| creating a symbolic breakpoint that is triggered at the start of the test run. | |
| Select "Debug > Breakpoints > Create Symbolic Breakpoint" from the Xcode menu, and | |
| populate the new brewpoint with the following details: | |
| * **Name**: IgnoreSignals | |
| * **Symbol**: UIApplicationMain | |
| * **Action**: Add debugger commands for: | |
| - ``process handle SIGINT -n true -p true -s false`` | |
| - ``process handle SIGUSR1 -n true -p true -s false`` | |
| - ``process handle SIGUSR2 -n true -p true -s false`` | |
| - ``process handle SIGXFSZ -n true -p true -s false`` | |
| * Check the "Automatically continue after evaluating" box. | |
| All other details can be left blank. When the process executes the | |
| ``UIApplicationMain`` entry point, the breakpoint will trigger, run the debugger | |
| commands to disable the automatic breakpoints, and automatically resume. |