Humpty Promotes > Rfo-Basic > Project Articles >


Android NDK binaries with rfo-Basic!  (Sep 2015)


hello world

What is NDK
Why Use it?
Simple Interface
Make a Binary

Your Basic! project

About The Shell
Process Properties
Conditions for exit
Communication
About  PIE
The 'hello world' moment

'hello world'     /həˈləʊ/  /wɜː(r)ld/   [noun]   [countable]

Desc:  A wonderment that suspends you in time while future possibilites fly around your head like flies.
Occurrence:  rare.


The only other time I remember was the very first output of my very first compiled language when a teenager.
Subsequent events fade into boredom and indifference. And that wonderment gets forgotten, especially as languages and systems become monotonously similar as the years drag on.

Surprisingly, the same feeling came back when executing a native c-binary from a basic! app on my Android phone. 
'The phone is running 'c' made code !'  For a few brief minutes, I was again a teenager.

I don't know wether it was because of my usual bias against Java running android phones, or that I had forgotten that a phone was really just a small sized PC, or that I had mentally placed NDK as just another mass of geeky dependent-hell files designed not to work for 'normal' people. But that the binary actually ran, freaked me out and gave me quite a buzz for the rest of the day.

This guide is simply about how to make a native binary and run it from rfo-basic.

Preliminaries
It is assummed you are familiar with;
a) rfo-basic
b) the c language and compilers
c) OS terminals and scripts (windows or linux) and filenames.

In this guide, we will be using the c language and our host PC is running linux. If you are using windows,
just convert the filepaths/names and script commands accordingly.



What is NDK?

The Android Native Development Kit is primarily a system of building cross compiled binaries of other languages to interface with Android's OS (which runs on Java). It relies heavily on using JNI as an interface to make library calls in-between the running binary and Java. But as you might have noticed, this website (about rfo-basic) avoids Java/XML due to it's complexities. So why use NDK at all?

Why Use It ?

It so happens that you can still compile NDK binaries and have them execute without JNI as long as you don't call the Java libraries. Such needs are typically tasks which are cpu intensive and don't need graphics or any other user interaction. e.g Background or service utilties e.g servers, batch file operations, or self contained utitlities which have already been proven and written in other languages.

Simple Interface

All we need to do is to find a way to execute and perhaps devise a simple way for the binary to talk to the basic! program.

Launching is not a problem, Basic! already has a built-in shell in the form of the system. command, of which we can use to launch the binary.

The next requirement is somewhere to put the binary, where we can give it permissions to execute.
All android apps are entitled a private area of storage called Primary Internal Private Storage. This is usually at location /data/data/<package_name>. It is created when the app is installed. The app has full rights here, so putting the binary here means we can give it the right to execute. Another bonus, is that the directory is wiped when the app is uninstalled.

For communication between the binary and basic!, you can either use a protocol of read/write to temporary files, or alternatively the stdin/stdout of the shell/binary.



Make a Binary


But first things first, let's see how we can build an actual binary with NDK.

The development environment is more simple than you think. There is no special IDE, no drivers to install and you can install to anywhere you want. You don't even have to set any environment variables if you don't want. Configuration It is mostly setting values inside provided scripts.

1. Download the latest NDK distribution.
The NDK distro is a gigantic collection of compilers and toolchains (utilities for compilation) and examples that cater for windows, linux and mac.
Pick the distro that matches your development PC.
Download and run the self-extracter which will extract a tree of files to a directory of your choice.

2. Check for build script ndk-build.

In the main directory there is script file called 'ndk-build'

This is the main script to call for compilation. Note it's location, you will need it for later.

(For windows, you can use 'ndk-build.cmd' otherwise you need to install cygwin to be able to use the linux script)

3. Navigate to the standalone example file.

Open a terminal and cd into tests/standalone/basic-c-compile/jni/
(although we won't use jni, this is the standard sub-directory structure.)

This is where the source files and makefile go, and where we will be operating from.

You will find a file called main.c which is the 'hello world' c example.
#include <stdio.h>

int main()

 printf ("hello world\n");

 return 0;
}

Now we check if a makefile is here..

4. Create Android.mk

Below is the makefile. If one is not provided, create this one (infact, you should probably use this one);

Android.mk
LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)

LOCAL_MODULE := main

#APP_PLATFORM := android-16
#LOCAL_CFLAGS += -fPIE
#LOCAL_LDFLAGS += -fPIE -pie

LOCAL_SRC_FILES:= main.c

include $(BUILD_EXECUTABLE)


This makefile takes one source file main.c, it will compile and link it with android's bionic library (a cut down version of the standard c library for android), and generate a binary called main.

In the middle of the makefile are 3 commented lines. The 3 lines determine wether PIE is turned on or off.
#APP_PLATFORM := android-16
#LOCAL_CFLAGS += -fPIE
#LOCAL_LDFLAGS += -fPIE -pie


#commented = PIE disabled ( off )
un-commented = PIE enabled ( on )

If your Android phone version is JellyBean or later (4.1+) i.e api16+, then un-comment the lines (PIE on).
If your Android phone version is older, i.e is less than 4.1,  then #comment the lines out (PIE Off)

(see later for explanation of PIE)

5. Create a compile script.

In this directory, create a script called go.sh (or go.cmd) and type in the path to the main build script ndk-build. e.g

go.sh
/mnt/sda4/dev/android-ndk-r10d/ndk-build

This means you can run this script instead of setting up environment variables with a path.
You will run 'go.sh' whenever you need to compile in this directory.

6. Compile

Run the compile script ./go.sh
This should generate any executables inside ../libs/armeabi/
i.e ../libs/armeabi/main
(It will also generate some files in ../obj/local/armeabi/ but we aren't interested in those.)

main is the binary executable which you will copy to your phone (see later).



Your Basic! project

For editor mode, the package_name is "com.rfo.basic".
For Apk, the package name is whatever you named your package(with your apk builder),
For both cases, the project directory will be /mnt/sdcard/<package_name>/

1. Copy the binary you compiled to your <>/data directory. e.g /mnt/sdcard/rfo-basic/data/main

2. In your program, initialise the basic shell and go to private storage /data/data/<package_name>
p$="com.rfo.basic"               % package name
path$ = "/data/data/"+p$+"/"
system.open
system.write "cd "+ path$+" 2>&1"
pause 50

3. Copy over the binary to private storage
b$="main"                  % binary name
file.root datapath$        % <base>/data

srce$ = datapath$+"/"+b$
system.write ("cat "+srce$+" >./"+b$+" 2>&1")

note: the binary should first be in <base>/data. If making a package either the apk-builder should ensure this or your basic program should pre-copy this from assets to <base> /data.

4. Make the binary executable
system.write "chmod 777 "+b$+" 2>&1"  % make executable

5. Run It.
system.write "./"+b$+" 2>&1"   % execute binary

6. Get the Output
r$=""                      % assume no reponse
pause 1000
system.READ.READY ready    % if response
while ready
      system.READ.LINE l$
      r$ += l$+"\n"
      system.READ.READY ready
repeat
print r$                   % output

7. Cleanup and close the shell
onBackKey:
system.close
system.open
system.write "cd "+ path$ + ";rm "+b$   % delete the binary
pause 100
system.close
end

When you run this basic program, it should copy over and execute the binary in a shell, then spool any output to the screen. Any shell errors will normally show by the spooling. If all goes well, you will see "hello world" .

That's It !


About the Shell

Basic! can execute binaries because of it's system command. This is a shell by which you can pass commands to. You can only open one shell at a time but even so, it is a powerful tool.

Process Properties
The shell is a truly spawned parallel process.
The executed binary is also a truly spawned parallel process.
Basic, the shell and the binary will all have differing PIDs.

Both the shell and the binary might be seen under 'Basic' in a task manager but the shell process is named "sh" and the binary process is named with it's called command e.g "./main"

Conditions for exit
If Basic exits or ends, it will also close the shell (if still open).
If the shell is closed, any spawned binaries will continue to run.
If Basic crashes, then both the shell and the binaries may continue executing.

Communication
Because the binary is forked from the shell, they will share the same environment. This means stdin/stdout will be shared.

Data from binary/shell to Basic.
Any data sent to stdout will set system.read.ready true after a linefeed (line based).
If Basic does not read the data, any furthur data will be buffered.
The binary may need to flush it's output stream to ensure this.

If the shell or binary has nothing to send, (or any sent data is not yet flushed),
and any previous data has already been read then system.read.ready will be false,

Regardless of any condition of system.read.ready, Basic will always continue to the next instruction (will not be blocked).

Data from Basic to binary/shell.
Any system.write will be sent to stdin.
If the shell or binary is not reading stdin, the data will be buffered and Basic will continue to the next instruction (will not be blocked).

If the binary is reading from stdin and there is no data sent (from Basic), then the binary will be blocked.





About  PIE

In Jun 2012, Google began to enable PIE (Position Independent Executables) in Android as a security measure.
They sneaked this into the OS starting with JellyBean (4.1) and ending with KitKat (4.4). This was the 'grace period' whereby any phone within this range is able to execute both PIE and non-PIE compiled code. Any phone after this range (starting with Lollipop (5.0)) will only execute PIE binaries.

This means older phones (< v4.1) don't work with PIE while newer phones (>4.4) don't work without it.

The only choice left for binary makers then, was either to force the customer to upgrade their phones or make sure the package ships with both types of binaries. For the latter case, the app will need to choose which binary to execute at runtime.

Fortunately in Basic, you can get the OS version like this
a=0
device a
bundle.get a, "OS", s$
With a bit of manipulation, you can convert this to an integer value to decide which binary to use.
(For apk making, you will need permission READ_PHONE_STATE for the 'device' command)

  Copy over the binary to private storage matching the OS pie or no-pie version. (modify the program above like this;)
b$="main"                  % binary name
file.root datapath$        % <base>/data

a=0
device a
bundle.get a, "OS", s$
v= val(word$(s$,1,"\\."))*10+val(word$(s$,2,"\\."))
if v<41 then suffix$="-nopie" else suffix$="-pie"

srce$ = datapath$+"/"+b$+suffix$
system.write ("cat "+srce$+" >./"+b$+" 2>&1")

    Note that the final binary copied to private storage need not have the pie/nopie suffix.

Ofcourse this means you have to compile two versions of your binary, main-pie and main-nopie.
Android.mk can build both versions at the same time.

Android.mk
LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)

LOCAL_MODULE := main-nopie

LOCAL_SRC_FILES:= main.c

include $(BUILD_EXECUTABLE)

#------------------------------
include $(CLEAR_VARS)

LOCAL_MODULE := main-pie

LOCAL_SRC_FILES:= main.c

APP_PLATFORM := android-16
LOCAL_CFLAGS += -fPIE
LOCAL_LDFLAGS += -fPIE -pie

include $(BUILD_EXECUTABLE)
Output

$ go.sh

[armeabi] Compile thumb  : main-nopie <= main.c
[armeabi] Executable     : main-nopie
[armeabi] Install        : main-nopie => libs/armeabi/main-nopie
[armeabi] Compile thumb  : main-pie <= main.c
[armeabi] Executable     : main-pie
[armeabi] Install        : main-pie => libs/armeabi/main-pie


note: since NDK 10d:
PIE is supposed to be anabled by default if APP_PLATFORM := android-16
but experience suggests this is not enough. To be sure that PIE is enabled, also set :
LOCAL_CFLAGS += -fPIE
LOCAL_LDFLAGS += -fPIE -pie



-End.


Sources and Acknowledgements:

https://en.wikipedia.org/wiki/Android_version_history
https://www.duosecurity.com/blog/exploit-mitigations-in-android-jelly-bean-4-1
http://stackoverflow.com/questions/24818902/running-a-native-library-on-android-l-error-only-position-independent-executab


Support my projects!
Donate





Leading Cloud Surveillance service

Leading Enterprise Cloud IT Service Since 2003

Powered by FirstCloudIT.com, a division of DriveHQ, the leading Cloud IT and Cloud Surveillance Service provider since 2003.