Tokenize

Tokenize is a cross-platform tool for turning text files containing untokenised BBC BASIC into tokenized ARM BASIC V files. It was written to be used in a cross-platform environment, but the version here is a standard RISC OS executable suitable for using when developing software natively.

Over and above the basic TEXTLOAD command, Tokenize offers a number of advantages:

The version of Tokenize that is provided here is a standard RISC OS executable which can be used when developing software natively. If you’re looking for a version to use on Linux, you should see the Build Tools section.

In addition, the current trunk in the RISC OS.info repository can be built automatically for Linux using the native-tokenize package in the GCCSDK Autobuilder. This is the recommended way of getting a version suitable for use on the Linux platform.

Download Tokenize
Tokenize is released under Version 1.2 of the European Union Public Licence. Its source code can be found on GitHub or RISC OS.info.

In order to use Tokenize, you will require:

  • a minimum of RISC OS 3.10, and
  • the 32-bit Shared C Library (version 5.43 or later), which is a standard part of RISC OS 5 and can be obtained for other systems as part of the System Resources download from RISC OS Open Ltd.