ROFLize an App: Bonus Guide to Features & Troubleshooting (Part 2)

In this 2-part bonus guide on the features and troubleshooting once you ROFLize an app, the first part covered in detail the marketplace, secrets, and persistent storage. Let's continue.

Public Variables

You will remember that we learned about secrets for confidential values. But an app may also consist of containers that need to access non-sensitive values and configurations. API endpoints, contract addresses, feature flags, etc are some examples of information that do not have or need any confidential attributes. This is when we use public variables. They are basically arbitrary key-value pairs that are exposed to containers as environment variables.

You can manage these public variables using the Oasis CLI. Take this example where we create a public variable called API_URL.

echo -n "https://api.example.com" | oasis rofl public-var set API_URL -

You will notice that this command only updates the local app manifest file. The public variable, however, is not yet propagated to the app. As a result, you will be able to easily configure as many public variables as you want without having to constantly update the on-chain app configuration.

Once the required public variables are created, you can update all of them in the on-chain configuration using the usual command.

oasis rofl update

The Oasis CLI documentation is useful if you need to consult comprehensive public variable management commands, including importing from .env files, removing public variables, and other advanced features.

Now, inside the containers, the public variables can be passed via environment variables. This is possible because each public variable is automatically exposed in the Compose environment and can be used in the Compose file.

services:
  test:
    image: docker.io/library/alpine:3.21.2@sha256:f3240395711384fc3c07daa46cbc8d73aa5ba25ad1deb97424992760f8cb2b94
    command: echo "API URL is $API_URL"
    environment:
      - API_URL=${API_URL}

rofl.yaml Manifest File

Before proceeding in this section, let's familiarize ourselves with the metadata in the yaml root, consisting of these valid fields.

• name: A short name for your app that is readable by humans. e.g. my-app
• version: The ROFL version you are using. e.g. 0.1.1
• repository: A path to the git repository. e.g. https://github.com/user/my-app
• author: The author name and the e-mail address. e.g., if you are John Doe, then it will show John Doe <john@doe.com>
• license: The ROFL license in SPDX format. e.g. Apache-2.0
• tee: The Trusted Execution Environment type that is being used. tdx is the default option, while sgx is also valid.
• kind: As outlined in the initialization process of the workflow. Valid options for TDX TEE are containers, which is the default, or raw. If you use SGX TEE, then only raw is the valid option.

App Resources (resources)

Each containerized app running in ROFL needs pre-defined resources such as the number of assigned vCPUs, amount of memory, storage requirements, GPUs, etc, for its execution. In the app manifest file, these will be headed under resources.

resources:
  memory: 512
  cpus: 1
  storage:
    kind: disk-persistent
    size: 512

If you decide to change the requested resources, it will result in the creation of a different enclave identity for the app. Then you will have to update the policy accordingly.

Let's now see what these resources signify.

• Memory (memory)
  The amount of memory is specified in megabytes. It is initialized to 512 by default.

• vCPU Count (cpus)
  The number of vCPUs allocated to the VM. It is initialized to 1 by default.

• Storage (storage)
  You can choose different storage options for your ROFL app based on its utility. Currently, it can be one of four options.

1. disk-persistent: When the disk of the given size is persistent, it is encrypted and authenticated using a key derived by the decentralized on-chain key management system after successful attestation. This is what our example shows.
2. disk-ephemeral: When the disk of the given size is ephemeral, it is encrypted and authenticated using an ephemeral key randomly generated on each boot.
3. ram: Here, an ephemeral filesystem is entirely contained in encrypted memory.
4. none: Here, no storage provision has been made. This option is not valid for containerized apps, so you have to choose one of the previous three.

The size field defines the amount of storage to provision in megabytes.

Artifacts (artifacts)

This configures locations of artifacts used during the ROFL build process with builder, firmware, kernel, stage2, container.runtime, and container.compose as supported fields. If any fields are left unspecified, they will use the default artifacts from the CLI. For containerized apps, container.compose points to the Compose file included in the ROFL bundle.

artifacts:
  container:
    compose: compose.yaml

Deployments (deployments)

This contains ROFL deployments on specific networks.

The deployment you have defined will show as deployment_name.

Deployment artifacts are optional and merged field by field on top of global artifacts.

deployments:
  testnet:
    network: testnet
    paratime: sapphire
    artifacts:
      container:
        compose: compose.testnet.yaml

There are four components to policy under which your app will spin up.

• quotes: Include TEE-specific policy requirements such as the TCB validity period and the minimum TCB-R number. This helps to indicate what security updates must be applied to the given platform.
• enclaves: Include permissioned enclave IDs for running your app.
• endorsements: Include a list of conditions defining who can run the app.
• any: {} indicates any node can run the app.
• node: <node_id> indicates only a specified node ID can run the app.
• provider: <address> indicates nodes belonging to the specified ROFL provider can run the app.
• provider_instance_admin: <address> indicates machines having the specified admin can run the app.

You can choose one or multiple conditions in a nested format by using and and or operators.

policy.yaml

endorsements:
  - and:
    - provider: oasis1qp2ens0hsp7gh23wajxa4hpetkdek3swyyulyrmz
    - or:
      - provider_instance_admin: oasis1qrk58a6j2qn065m6p06jgjyt032f7qucy5wqeqpt
      - provider_instance_admin: oasis1qqcd0qyda6gtwdrfcqawv3s8cr2kupzw9v967au6

This example indicates that the app can be run only on the specified provider, and on machines owned by either of the two admin addresses.

• fees: <fee_policy> specifies who pays for the registration and other fees. It can be either endorsing_node when the node running the app pays, or instance when the app instance pays.

The final piece of this section is machines, where the specific app deployment takes place. If you remember the oasis rofl deploy tutorial, it creates a new default machine if there is no existing machine. If there is one, then the app is redeployed here.

• <machine_name> is the name you choose for the machine.
• provider: <provider_address> is the Oasis native address of the ROFL provider hosting the machine.
• offer: <offer_name> specifies what offer you have chosen.
• id: <machine_id> is the ID of the machine per provider.
• permissions are optional, and when present, indicate ROFL scheduler-specific permissions.
• log.view will list all the Oasis native addresses that can access machine logs.

appd REST API

Each containerized app running in ROFL runs a special daemon called rofl-appd. It exposes additional functions via a simple HTTP REST API. To enable easier access isolation, the API is exposed via a UNIX socket located at /run/rofl-appd.sock.

Let's consider this example where we have used the short syntax for Compose volumes.

services:
  mycontainer:
    # ... other details omitted ...
    volumes:
      - /run/rofl-appd.sock:/run/rofl-appd.sock

ROFL clients

For your ROFL app, it is strongly recommended that you follow the steps to bind the UNIX socket by accessing the ROFL REST API through one of the ROFL clients. You can choose any one of the following languages.

1. oasis-rofl-client for Python
2. @oasisprotocol/rofl-client for TypeScript
3. oasis-rofl-client for Rust

Note: Although the communication with rofl-appd is through UNIX sockets, the REST service still uses the HTTP protocol. In our examples, we will be using the http://localhost/<endpoint_path> format throughout. You are free to provide any name instead of a hostname.

Endpoints

App Identifier is where the endpoint is used to retrieve the app ID.
Endpoint: /rofl/v1/app/id (GET)
Example response:

rofl1qqn9xndja7e2pnxhttktmecvwzz0yqwxsquqyxdf

Key Generation

Here, each registered app automatically gets access to a decentralized on-chain key management system. Now, the keys can only be generated inside properly attested app instances. They remain unchanged even if the app is deployed elsewhere, or even if its state is erased.
Endpoint: /rofl/v1/keys/generate (POST)
Example request:

{
  "key_id": "demo key",
  "kind": "secp256k1"
}

• key_id is used for domain separation of different keys. It is a unique identifier, with every key ID corresponding to a different key.
• kind defines what kind of key should be generated. Options include:

1. raw-256 to generate 256 bits of entropy
2. raw-386 to generate 384 bits of entropy
3. ed25519 to generate an Ed25519 private key
4. secp256k1 to generate a Secp256k1 private key, as used in our example

The generated key is returned as a hexadecimal string.
Example response:

{
  "key": "a54027bff15a8726b6d9f65383bff20db51c6f3ac5497143a8412a7f16dfdda9"
}

Authenticated Transaction Submission

This is important if your app is registered with a different chain instead of Oasis. It enables your ROFL app to submit authenticated transactions to that chain. As these transactions are signed by an endorsed ephemeral key, they get automatically authenticated.

This also helps to easily authenticate the transaction origin in smart contracts by simply invoking an appropriate subcall.

Subcall.roflEnsureAuthorizedOrigin(roflAppID);

Endpoint: /rofl/v1/tx/sign-submit (POST)
Example:

{
  "encrypt": true,
  "tx": {
    "kind": "eth",
    "data": {
      "gas_limit": 200000,
      "to": "1234845aaB7b6CD88c7fAd9E9E1cf07638805b20",
      "value": "0",
      "data": "dae1ee1f00000000000000000000000000000000000000000000000000002695a9e649b2"
    }
  }
}

Let's decipher the fields before proceeding further.

tx describes the transaction content. Different transaction kinds are supported as defined by the kind field.

Ethereum-compatible calls (eth) use standard fields such as gas_limit, to, value, and data.

For gas_limit, you can input a JSON number (as used in the example), a decimal string, or a 0x-prefixed hex string. There should not be any whitespace, and irrespective of the input, it will be interpreted as a non-negative 64-bit integer.

For value, you can input a JSON number up to 2^64 - 1, a decimal string, or a 0x-prefixed hex string. There should not be any whitespace in the string forms, and the value must represent a non-negative integer up to 256 bits.

For hex-encoded fields such as to and data, you can input strings with or without a leading 0x prefix, but there should not be any whitespace or prefix-only input. Empty strings are accepted for contract creation or empty calldata, e.g. to: "" or data: "". If you are providing input for the to field, it must decode to exactly 20 bytes representing an Ethereum address.

Alternately, Oasis SDK calls (std) support CBOR-serialized hex-encoded Transactions to be specified.

encrypt is a boolean flag specifying whether the transaction should be encrypted. This field is true by default. When an ephemeral key is being used, the encryption is handled transparently for the caller, and any response is first decrypted before being passed on.

Now, as the outcome of the example request, the example response inside data is generated as a JSON response containing a CBOR-serialized hex-encoded call result that you will need to deserialize.

If the call result is successful:

{
  "data": "a1626f6b40"
}

It deserializes as {"ok": ''}.

If it is unsuccessful:

{
  "data": "a1646661696ca364636f646508666d6f64756c656365766d676d6573736167657272657665727465643a20614a416f4c773d3d"
}

It deserializes as {"fail": {"code": 8, "module": "evm", "message": "reverted: aJAoLw=="}}.

Replica Metadata

This allows apps to publish arbitrary key-value pairs included in the on-chain ROFL replica registration and automatically namespaced with net.oasis.app.

• Get Metadata: With this, you can retrieve all user-set metadata key-value pairs. Endpoint: /rofl/v1/metadata (GET) Example response:

{
  "key_fingerprint": "a54027bff15a8726",
  "version": "1.0.0"
}

• Set Metadata: With this, you can set metadata key-value pairs to replace all existing app-provided metadata. Endpoint: /rofl/v1/metadata (POST) Example request:

{
  "key_fingerprint": "a54027bff15a8726",
  "version": "1.0.0"
}

The parameters for metadata validation are the number of pairs, key size, and value size.

• Upsert Metadata: With this, you can input or update metadata key-value pairs. However, if you did not specify it in your request but there is existing app metadata, that will not be affected. Endpoint: /rofl/v1/metadata (PUT) Example request:

{
  "version": "1.0.1"
}

• Delete Metadata: With this, you can delete given metadata keys, while keys that no longer exist will be skipped. Endpoint: /rofl/v1/metadata (DELETE) Example request:

["version", "key_fingerprint"]

Whenever you use Set, Upsert, or Delete Metadata, any change in the metadata triggers a registration refresh.

Query

This runs arbitrary query methods defined in the Oasis Runtime SDK module and returns the result.

Endpoint: /rofl/v1/query (POST)
Example request:

{
  "method": "rofl.App",
  "args": "a16269645500694cb01f85408d624ea267f657bf285787a61db3"
}

Here, method refers to the internal name of query methods; in our example, it is rofl.App. You will recognize query methods by the #[handler(query = "...")] annotation in the Oasis Runtime SDK source.
args represent query parameters for the method serialized as CBOR and hex-encoded.

Example response:

{"data":"a76269645500ffe981cceff2d759d19fa926faebb7d90c0a59a56373656b582056c6d4841fa5ad24ad761088cb7053ad312ef13f3c2f405640fdba2bde4c14386561646d696e55007814f3d954f41b6459eb9e4bc8fbc6767ece5aa9657374616b658249056bc75e2d631000004066706f6c696379a56466656573026671756f746573a263696173f663706373a363746478a173616c6c6f7765645f7464785f6d6f64756c657380737463625f76616c69646974795f706572696f64181e781e6d696e5f7463625f6576616c756174696f6e5f646174615f6e756d6265721268656e636c6176657382a2696d725f7369676e6572582000000000000000000000000000000000000000000000000000000000000000006a6d725f656e636c6176655820bd4844a79a12ba365e890ddeaebbc4e4292797c7d956b42c2e25e4aefce3b124a2696d725f7369676e6572582000000000000000000000000000000000000000000000000000000000000000006a6d725f656e636c6176655820412c94a9baa0949f718dbba41ab89c38ea5c320345bba36b07e2ef857ffe2fb96c656e646f7273656d656e747381a163616e79a06e6d61785f65787069726174696f6e036773656372657473a26a50494e4154415f4a5754590327a462706b58207f88546291174f854a8ce2eb4bbfc8e62e40d60cdae2d600920a766d3006bf6c646e616d65581aad0460d0f742ad697b6d7c8462cc2b153deeb051a791553ef52b656e6f6e63654f8cbbe6631910d9a702e49bd30ee8f46576616c75655902c1352d823e54f75c846579066e2c1a750387f0630e3f29dba5524984712883bb33371ff017e507ae432b135312af64bfb85f3b17def05b2ac2744256f5accf1a26b29cd8cd412f08bfc9204f9bfa670b2d65972cfc4a8d4e2074402f21c18dfe554b1f0a8a731c077699f741807b3a4047ef4dc570958b8b46111a445259e93c9b92a5f72a23d32cbbef875efe586a8ddda38f1fa286d19b369a2022c3eae1cdbf6f6de4dc055bbab36a2c4830f8e2c64437f5f878f419e21f3a4c0f13c47b63697668b34722eaa61bdffa12e82be6d0266a41590254c9d70e9237475f6115c2867065c49afa8032acdfe0bc5dc671ef48ebc58d893937659243479e2eaa38815cd8665541e4c7e40349524cda15f2d410cba100ee27f0a59dc63534f7cff2444b57c7b74060cf8c3e21e1590b597384a89a463d6bb4a52fc52a4592889448a8f8e0c02fef1689c1efea58ddc08783f6d22d7a908cdf2a45bc46a79a3b73ff5f13bbbf7219973a7382eee84b3b4d48036b5e87fb24223e6387a3e37f9c1ac9722534adfef0201ec13db2e70fe9cd0336f020e50b59d7d32951378f568a4ef3a8117386ff0f1ba4b7d33dcf913daa696a6a7d64d20220b0e5eec994ea56aa9c01f56f5e12bf7d555b3f4a218ddbd8ae1586d5cb1e76802c90e26472b233686e8449284829378163acd77d1b65d4953a76cc497584580c1a5ac4fe6d97fd818fa53c2eb43c6f1b7a79211ef57f24036b1f8ed37f4c3d36873bbbd876769a5fde17add6926317afc96c9707e10b150735c63877ffd6475b1a27b6dd108940f0097375e422b1d308c6c8a0a83847368f5760778c54f2a70b44f9365c55c4b5d69341cad62d6fdb11aa25a9e26b4977adebd65718042bda1cbbf988298d293547107c2f5899352188179bc4d22febc66e8e351885498e707dee583d052c0b7c13f930e4529b59c36c20ee3ee6d29d1a3e2bb65bd489b7206cd45d2ae046f1e147d6407c166e494e465552415f4150495f4b45595899a462706b582074be2e227be81a5e35943ac1b79e238395b9bc367f7a0d1eed740c259ae23a37646e616d65581ee5190cda87688753f6f221890bfed8fe0e27b3ced4a9bc54537da1f4cd90656e6f6e63654fcf5411193abf4f4711f17179ea4b6f6576616c756558304675a29c8398bc94b5057063b4badeb9937a6e019ef7f8095d6b5a035106d4e8e7d710af9b6883848886481c1d180cbc686d65746164617461a7736e65742e6f617369732e726f666c2e6e616d656f76616c696461746f722d6167656e74756e65742e6f617369732e726f666c2e617574686f72782a4d61746576c5be204a656b6f766563203c6d617465767a406f6173697370726f746f636f6c2e6f72673e766e65742e6f617369732e726f666c2e6c6963656e73656a4170616368652d322e30766e65742e6f617369732e726f666c2e76657273696f6e65302e312e30776e65742e6f617369732e726f666c2e686f6d6570616765784f68747470733a2f2f6769746875622e636f6d2f6f6173697370726f746f636f6c2f6572632d383030342f626c6f622f6d61737465722f524541444d452e6d642376616c696461746f722d6167656e7478196e65742e6f617369732e726f666c2e7265706f7369746f7279782968747470733a2f2f6769746875622e636f6d2f6f6173697370726f746f636f6c2f6572632d38303034781a6e65742e6f617369732e726f666c2e6465736372697074696f6e787f4c697374656e7320746f2056616c69646174696f6e52657175657374206576656e7473206f6620746865204552432d383030342076616c69646174696f6e20726567697374727920616e642076616c696461746573207768657468657220746865206167656e7420697320706f776572656420627920524f464c205445452e"}

Inside data, the JSON response contains the CBOR-serialized method's return value in hex format.

If you want to try other examples, you can check out the relevant section of the ROFL demo repository for querying with curl directly.
There is also a production-ready Python example at hand - in the ROFL-8004 implementation where the query endpoint is used to fetch various app on-chain metadata for registration in the ERC-8004 identity registry.

Port Proxy

When you publish a port in your compose.yaml file, the ROFL proxy automatically makes your services accessible via public URLs. It also ensures the routed traffic is done correctly.

This uses TLS, which is terminated inside your ROFL enclave, maintaining confidentiality and integrity protection. As a result, even the provider cannot see or modify the traffic. Moreover, the default terminate-tls mode generates and configures a Let's Encrypt certificate in ROFL to authenticate your services.

To enable the proxy and expose a port from your container, you need to publish it in your compose.yaml file.

compose.yaml

services:
  frontend:
    image: docker.io/hashicorp/http-echo:latest
    ports:
      - "5678:5678" # Expose container port 5678 on host port 5678

After deploying your app, you can find the generated URL by running the usual command.

oasis rofl machine show

The output generated in this way will show a Proxy section with the public URL for each published port.

Proxy:
  Domain: m602.test-proxy-b.rofl.app
  Ports from compose file:
    5678 (frontend): https://p5678.m602.test-proxy-b.rofl.app

Configuration

You can use the annotations in your compose.yaml file to configure the proxy behavior.

The general format of an annotation is net.oasis.proxy.ports.<published_port>.<setting>: <value>.
Here, <published_port> is the external port exposed in your compose.yaml, and <setting> indicates the specific proxy configuration like mode or custom_domain.

Example:
Here I will configure port 80 to use the default terminate-tls mode with a custom domain and port 8080 to use TCP passthrough.

compose.yaml

services:
  myservice:
    image: docker.io/my/service:latest
    ports:
      - "80:80"
      - "8080:8080"
    annotations:
      net.oasis.proxy.ports.80.custom_domain: mydomain.com
      net.oasis.proxy.ports.8080.mode: passthrough

This shows:

• The application container exposes ports 80 and 8080.
• On port 80, the proxy terminates TLS for mydomain.com and forwards traffic to the application container.
• On port 8080, the proxy forwards the raw TCP connection to your application container (mode: passthrough).

Annotation Reference

net.oasis.proxy.ports.<published_port>.mode defines how the proxy should handle connections for the specified port.

net.oasis.proxy.ports.<published_port>.custom_domain assigns a custom domain name to the published port.

Here, when using the default terminate-tls mode, you need to use special configuration for your custom domain to route through the proxy. Once the app is deployed, you can use Oasis CLI for instructions to configure A and TXT records in your DNS.

oasis rofl machine show

Proxy:
  Domain: m897.opf-testnet-rofl-25.rofl.app
  Ports from compose file:
    5678 (frontend): https://demo.rofl.build
      * Point the A record of your domain to: 131.153.241.25
      * Add a TXT record to your domain:
oasis-rofl-verification=4SKHCn4E2SNDB5tXayQeHZsvH/+kJSNGuQaTAPepYJc=

If you choose to go with passthrough mode, the proxy will not terminate TLS and your app will then need to handle it directly. Also, here the custom_domain setting is not needed, so you can configure the domain directly to the ROFL instance's address.

For the ignore mode, the port isn't published, so the custom_domain setting has no effect.

Troubleshooting

Here I will cover some common errors and the troubleshooting process.

Compilation

Sometimes you will see an error message if the aes and ssse3 compiler flags are not enabled during compilation of your SGX and TDX-raw ROFL.

error: The following target_feature flags must be set: +aes,+ssse3.
--> /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deoxysii-0.2.4/src/lib.rs:26:1
|
26 | compile_error!("The following target_feature flags must be set: +aes,+ssse3.");
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The workaround is to add default flags to your .cargo/config.toml file.

[build]
rustflags = ["-C", "target-feature=+aes,+ssse3"]
rustdocflags = ["-C", "target-feature=+aes,+ssse3"]

[test]
rustflags = ["-C", "target-feature=+aes,+ssse3"]
rustdocflags = ["-C", "target-feature=+aes,+ssse3"]

Compose file

A couple of errors are possible here due to an upstream podman compose bug.

The first is when environment variables defined are not considered.

services:
  oracle:
    platform: linux/amd64
    environment:
      CONTRACT_ADDRESS: 0x5FbDB2315678afecb367f032d93F642f64180aa3
    entrypoint: /bin/sh -c 'python main.py $${CONTRACT_ADDRESS}'

In this type of error, the CONTRACT_ADDRESS field will return as empty in ROFL. You need to inject the variable value directly inside entrypoint as a workaround.

services:
  oracle:
    platform: linux/amd64
    entrypoint: /bin/sh -c 'python main.py 0x5FbDB2315678afecb367f032d93F642f64180aa3'

The other type of error that may occur is when depends_on is ignored.

services:
  contracts:
    image: "ghcr.io/foundry-rs/foundry:latest"
    platform: linux/amd64
    volumes:
      - ./contracts:/contracts
    entrypoint: /bin/sh -c 'cd contracts && forge create'

  oracle:
    platform: linux/amd64
    entrypoint: /bin/sh -c 'python main.py'
    restart: on-failure
    depends_on:
      contracts:
        condition: service_completed_successfully

In this type of error, instead of oracle spinning up once the contracts service successfully deploys the contracts and finishes, they start in parallel by ignoring the depends_on command.

There is no immediate workaround as of now. You can try to implement customized logic in your oracle service to crash it, and then trigger the restart mechanism and try again.

appd

If you encounter the 422 Unprocessable Entity error, when the provided request couldn't be decoded, you need to ensure all the required fields are present and correctly formatted in accordance with the appd REST API section described above.

ROFL Proxy URL is not working

Sometimes the app might be using outdated artifacts, which will result in the proxy URL returned by oasis rofl machine show being inaccessible. This is easily fixed by updating to the latest Oasis CLI version. The next step is to run oasis rofl upgrade in your project directory to update the artifacts in your rofl.yaml file, and finally, rebuild and redeploy your app.

oasis rofl build
oasis rofl update
oasis rofl deploy

This concludes our 2-part bonus guide describing the various features for your ROFL app, and some common troubleshooting hacks. Looking forward to your feedback in the comments section.

For technical specs, APIs, architecture, and integration guides, the Oasis documentation is your starting point.
For direct support on specific issues, the Oasis engineering team is available in the dev-central channel on the official Discord.