Invalid .tflite Flatbuffer issue for llm inference task on Android CPU

[li-sicong]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Android 15

MediaPipe Tasks SDK version

No response

Task name (e.g. Image classification, Gesture recognition etc.)

LLM inference

Programming Language and version (e.g. C++, Python, Java)

C++

Describe the actual behavior

Got error "ERROR: The model is not a valid Flatbuffer buffer" when passing a gemma3-1b-it-q4.tflite file to the llm_inference_engine_cpu_main example

Describe the expected behaviour

Successfully run llm inference

Standalone code/steps you may have used to try to get what you need

Build command: bazel build --cxxopt=-DABSL_FLAGS_STRIP_NAMES=0 -c opt --config=android_arm64 mediapipe/tasks/cc/genai/inference/c:llm_inference_engine_cpu_main

Other info / Complete Logs

Build OS: ubuntu 24.04

Model: gemma-3-1b-it-q4.tflite downloaded from kaggle: https://www.kaggle.com/models/google/gemma-3/tfLite/gemma3-1b-it-int4

Command
caiman:/data/local/tmp $ ./llm_inference_engine_cpu_main --model_path=./gemma3-1B-it-int4.tflite      

Error log:                                                                                                               
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1758213468.982536   27774 llm_inference_engine_cpu_main.cc:119] Prompt: Write an email
ERROR: The model is not a valid Flatbuffer buffer
E0000 00:00:1758213468.984778   27774 llm_inference_engine_cpu_main.cc:127] Failed to create session: Failed to create engine: INTERNAL: ; RET_CHECK failure (mediapipe/tasks/cc/genai/inference/utils/llm_utils/model_data.cc:334) modelError building tflite model.

[kalyan2789g]

Hi @li-sicong,

Thank you for contacting us. We are currently discussing the issue with our internal development team.

To help us reproduce and resolve this, could you please provide the exact steps you took? Your detailed steps would be a great help.

Thanks,
@kalyan2789g

[li-sicong]

Hi Kalyan!

Thanks for the reply! Here are the steps:

Host Environment:
Ubuntu 24.04
bazel 7.4.1
shell: bash
python: python 3.10.18

Target Environment:
google pixel 9 pro. Android 15.

1. Clone the repo to host:

git clone git@github.com:google-ai-edge/mediapipe.git

at main: 45b2c7e

2. Install Java jdk

mkdir -p $HOME/install/java
 
wget https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15%2B6/OpenJDK17U-jdk_x64_linux_hotspot_17.0.15_6.tar.gz
 
# can omit `-C /dir/to/install`, to unzip in current working directory
tar -xzf OpenJDK17U-jdk_x64_linux_hotspot_17.0.15_6.tar.gz -C $HOME/install/java

echo '\
export JAVA_HOME=$HOME/install/java/jdk-17.0.15+6 \
export PATH=$JAVA_HOME/bin:$PATH \
' >> $HOME/.bashrc

source $HOME/.bashrc

3. Install python packages

python3 -m venv ./venv
source ./venv/bin/activate
pip3 install -r requirements.txt

4. Set up android environment:

./setup_android_sdk_and_ndk.sh

The ndk and jdk setup in WORKSPACE:

android_sdk_repository(name = "androidsdk", path = "/home/sicli01/Android/Sdk")
android_ndk_repository(name = "androidndk", api_level = 21, path = "/home/sicli01/Android/Sdk/ndk-bundle/android-ndk-r28b")
bind(name = "android/crosstool", actual = "@androidndk//:toolchain")

5. Build llm_inference_engine_cpu_main example

bazel build --cxxopt=-DABSL_FLAGS_STRIP_NAMES=0  -c opt --config=android_arm64 mediapipe/tasks/cc/genai/inference/c:llm_inference_engine_cpu_main

6. Push the binary to target phone

adb push ./bazel-bin/mediapipe/tasks/cc/genai/inference/c/llm_inference_engine_cpu_main /data/local/tmp

7. Download the model and push to target phone
   I downloaded the gemma-3-1b-it-q4.tflite from kaggle: https://www.kaggle.com/models/google/gemma-3/tfLite/gemma3-1b-it-int4

adb push $HOME/Downloads/gemma-3-1B-it-int4.tflite /data/local/tmp

8. Run the example binary

adb shell
export TEST_SRCDIR="."
./llm_inference_engine_cpu_main --model_path=./gemma3-1B-it-int4.tflite

9. Error log:

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1758274431.071298 21096 llm_inference_engine_cpu_main.cc:119] Prompt: Write an email
ERROR: The model is not a valid Flatbuffer buffer
E0000 00:00:1758274431.073236 21096 llm_inference_engine_cpu_main.cc:127] Failed to create session: Failed to create engine: INTERNAL: ; RET_CHECK failure (mediapipe/tasks/cc/genai/inference/utils/llm_utils/model_data.cc:334) modelError building tflite model.

Note that I also used the .task file but it doesn't use the xnn_utils/llm.h implementation which provides full LLM features so I gave up on that path.