JavaScript Raw APIs

Low level APIs for controlling your android phone

getScreenSize()

Returns Object - {width: Integer, height: Integer}

var sizeObj = getScreenSize();
console.log(sizeObj.width, sizeObj.height);
// 1080 1920

getScreenshot()

Returns Integer - The image pointer

var img = getScreenshot();
console.log(img);
// 122344533 <- image pointer
releaseImage(img); // Don't forgot release a pointer

getScreenshotModify(cropX, cropY, cropWidth, cropHeight, resizeWidth, resizeHeight, qualitys)

Get screenshot, crop and resize. For speeding up screenshot.

  • cropX Integer

  • cropY Integer

  • cropWidth Integer

  • cropHeight Integer

  • resizeWidth Integer

  • resizeHeight Integer

  • quality Integer

Returns Integer - The image pointer

var image = getScreenshotModify(200, 200, 100, 100, 50, 50, 80);
console.log(image); // image width = 50, height = 50 
// 12333122
releaseImage(image);

execute(command)

Call exec command in android system. It's permission is same as adb shell

  • command String

Returns String - The result of the execution

var result = execute("ls -al /sdcard");
console.log(result);
// drwxr-xr-x   2 root  root    64B 12 14 23:44 Robotmon

tap(x, y, during)

Simulate a tap event

  • x Integer

  • y Integer

  • during Integer

tap(200, 200, 10);
// Will inject a tap down and a tap up event to system

tapDown(x, y, during)

  • x Integer

  • y Integer

  • during Integer

tapDown(200, 200, 40);
// Will inject a tapDown event to system

tapUp(x, y, during)

  • x Integer

  • y Integer

  • during Integer

tapUp(200, 200, 40);
// Will inject a tapUo event to system

moveTo(x, y, during)

moveTo should be betewwn tapDown and tapUp

  • x Integer

  • y Integer

  • during Integer

tapDown(500, 300, 40);
moveTo(500, 600, 40);
tapUp(500, 600, 40);
// Will inject a swipe down event

swipe(x1, y1, x2, y2, during)

Simulate a swipe event, using tapDown, moveTo and tapUp event. This function may not work in some game, you should implement yourself.

  • x1 Integer

  • y1 Integer

  • x2 Integer

  • y2 Integer

  • during Integer

swipe(500, 300, 40); // same as above example
// Will inject a swipe down event

keycode(label, during)

Send a key code event to system Like adb shell input keyevent command Android Keycode List

  • label String

  • during Integer

keycode('HOME', 40); // same as keycode('KEYCODE_HOME', 40);
// Will send a HOME event to system

typing(words, during)

Only allow English words

  • words String

  • during Integer

typing('Hello!', 100);
// Will type 'H' 'e' 'l' 'l' 'o' '!' 6 words

OpenCV

clone(sourceImg)

Duplicate an image to another.

  • sourceImg Integer

Returns Integer - The image pointer

var oriImage = getScreenshot();
for (var i = 0; i < 10; i++) {
  var cloneImage = clone(oriImage);
  // modify clone Image here
  smooth(cloneImage, 1, 5); // blur
  release(cloneImage);
}
release(oriImage);

smooth(sourceImg, smoothType, size)

Same as OpenCV smooth() function.

  • sourceImg Integer

  • smoothType Integer

  • size Integer

smoothType

description

0

CV_BLUR_NO_SCALE

1

CV_BLUR

2

CV_GAUSSIAN

3

CV_MEDIAN

4

CV_BILATERAL

var img = getScreenshot();
smooth(img, 2, 5); // Gaussian blur
saveImage(img, getStoragePath + '/smooth.png');
releaseImage(img);

convertColor(sourceImg, code)

Same as OpenCV cvtColor(). Not support different channels. If you want to convert to gray, please use bgrToGray. Note that getScreenshot and getScreenshotModify is BGR order;

  • sourceImg Integer

  • code Integer

code

description

40

CV_BGR2HSV

52

CV_BGR2HLS

See more: OpenCV Types

var img = getScreenshot();
// Convert BGR to HSV color
convertColor(img, 40);
releaseImage(img);

bgrToGray(sourceImg)

Convert form bgr (3 channels) to gray (1 channel).

  • sourceImg Integer

Returns Integer - The gray image pointer

var img = getScreenshot();
var gray = bgrToGray(img); // gray image
releaseImage(img);
releaseImage(gray);

absDiff(sourceImg, targetImg)

Same as OpenCV adbdiff().

  • sourceImg Integer

  • targetImg Integer

Returns Integer - The image pointer of the difference

var img1 = getScreenshot();
sleep(100);
var img2 = getScreenshot();
var diff = absDiff(img1, img2); // in gray order
releaseImage(img1);
releaseImage(img2);
releaseImage(diff);

threshold(sourceImg, thr, maxThr, code)

Same as OpenCV threshold().

  • sourceImg Integer

  • thr Float

  • maxThr Float

  • code Integer

code

description

0

CV_THRES_BINARY

See more: OpenCV Types

keycode('MENU');
sleep(1000);
var img1 = getScreenshot();
keycode('HOME');
sleep(1000);
var img2 = getScreenshot();
var diff = absDiff(img1, img2); // in gray order
threshold(diff, 100, 255); // set to 0 if <= 100, set to 255 if > 100
var value = getImageColor(diff, 500, 200); // value => {r":255,"g":0,"b":0","a":0}
console.log(value['r']); // current diff value is show on 'r'
// 255
releaseImage(img1);
releaseImage(img2);
releaseImage(diff);

eroid(sourceImg, width, height, x, y)

Same as OpenCV eroid.

width, height, x, y is getStructuringElement() parameters.

  • sourceImg Integer

  • width Integer

  • height Integer

  • x Integer

  • y Integer

var img = getScreenshot();
threshold(img, 100, 255);
eroid(img, 3, 3, 1, 1);
saveImage(img, getStoragePath() + '/test_eroid.png');
releaseImage(img);

dilate(sourceImg, width, height, x, y)

Same as OpenCV dilate.

width, height, x, y is getStructuringElement() parameters.

  • sourceImg Integer

  • width Integer

  • height Integer

  • x Integer

  • y Integer

var img = getScreenshot();
threshold(img, 100, 255);
dilate(img, 3, 3, 1, 1);
saveImage(img, getStoragePath() + '/test_dilate.png');
releaseImage(img);

inRange(sourceImg, minB, minG, minR, minA, maxB, maxG, maxR, maxA)

Same as OpenCV inRange + clone + mask. Filter with range color and clone to new image.

  • sourceImg Integer

  • minB Integer

  • minG Integer

  • minR Integer

  • minA Integer

  • maxB Integer

  • maxG Integer

  • maxR Integer

  • maxA Integer

Returns Integer - The filtered image pointer

var img = getScreenshot();
var filteredImg = inRange(img, 0, 255, 255, 255, 255, 255, 255, 255); // only keep blue color pixel
saveImage(filteredImg, getStoragePath() + '/test_filterd.png');
releaseImage(img);
releaseImage(filteredImg);

outRange(sourceImg, minB, minG, minR, minA, maxB, maxG, maxR, maxA)

Same as OpenCV inRange + clone + not + mask. Filter without range color and clone to new image.

  • sourceImg Integer

  • minB Integer

  • minG Integer

  • minR Integer

  • minA Integer

  • maxB Integer

  • maxG Integer

  • maxR Integer

  • maxA Integer

Returns Integer - The filtered image pointer

var img = getScreenshot();
var filteredImg = outRange(img, 0, 255, 255, 255, 255, 255, 255, 255); // keep all but blue color
saveImage(filteredImg, getStoragePath() + '/test_filterd.png');
releaseImage(img);
releaseImage(filteredImg);

cloneWithMask(sourceImg, mask)

Same as OpenCV copyTo. Clone image with mask (only support 1 channel)

  • sourceImg Integer

  • mask Integer

Returns Integer - new image pointer with mask

var img1 = getScreenshot();
sleep(100);
var img2 = getScreenshot();
var diff = absDiff(img1, img2);
sleep(100);
var img3 = cloneWithMask(img1, diff);
releaseImage(img1);
releaseImage(img2);
releaseImage(img3);
releaseImage(diff);

houghCircles(sourceImg, method, dp, minDist, p1, p2, minR, maxR)

Same as OpenCV houghCircles. For finding circles.

  • sourceImg Integer

  • method Integer (3 = CV_HOUGH_GRADIENT)

  • dp Float (1) (ratio between input image and input params.)

  • minDist Float (min distance between circles)

  • p1 Float (canny parameter)

  • p2 Float (canny parameter)

  • minR Integer (min radius)

  • maxR Integer (max radius)

Returns Object - Array of circles

var img = getScreenshot();
var points = houghCircles(img, 3, 1, 8, 4, 8, 6, 14);
console.log(points); // {"0": {"x": 102, "y": "233", "r": 9}}
releaseImage(img);

canny(sourceImg, t1, t2, apertureSize)

Same as OpenCV canny

  • sourceImg Integer

  • t1 Float

  • t2 Float

  • apertureSize Integer

Returns Integer - The canny image pointer

var img = getScreenshot();
threshold(img, 30, 255);
eroid(img, 5, 5, 1, 1);
var cannyImg = canny(img, 50, 150, 3);
saveImage(cannyImg, getStoragePath() + '/test_canny.png');
releaseImage(img);
releaseImage(cannyImg);

findContours(cannyImgPtr, minArea, maxArea)

Same as OpenCV findContours.

  • cannyImgPtr Integer (Canny image as input)

  • minArea Float

  • maxArea Float

Returns Object - {"0": {x: Integer, y: Integer}

var img = getScreenshot();
threshold(img, 30, 255);
eroid(img, 5, 5, 1, 1);
var cannyImg = canny(img, 50, 150, 3);
var results = findContours(cannyImg, 1000, 10000); // area > 100
console.log(JSON.stringify(results));
// {"0":{"x":537,"y":1850},"1":{"x":133,"y":601}}
releaseImage(img);
releaseImage(cannyImg);

drawCircle(sourceImg, x, y, radius, r, g, b, a)

Draw circle in an image.

  • sourceImg Integer

  • x Integer

  • y Integer

  • radius Integer

  • r Integer

  • g Integer

  • b Integer

  • a Integer

var img = getScreenshot();
drawCircle(img, 100, 100, 10, 0, 0, 255, 0); // draw a blue circle
saveImage(img, getStoragePath() + '/test_drawCircle.png');
releaseImage(img);

getIdentityScore(sourceImg, targetImg)

  • sourceImg Integer

  • targetImg Integer

Returns Float - The identity score

keycode('MENU');
sleep(1000);
var img1 = getScreenshot();
keycode('HOME');
sleep(1000);
var img2 = getScreenshot();
var score = getIdentityScore(img1, img2);
console.log(score); // 0.6004924774169922
releaseImage(img1);
releaseImage(img2);

cropImage(sourceImg, x, y, width, height)

Crop image.

  • x Integer

  • y Integer

  • width Integer

  • height Integer

Returns Integer - The image pointer

var img = getScreenshot();
var cropImg = cropImage(img, 350, 550, 150, 150);
saveImage(cropImg, getStoragePath() + '/test_crop.png');
releaseImage(img);
releaseImage(cropImg);

findImage(sourceImg, targetImg)

Using OpenCV Template Match to fing image.

  • sourceImg Integer

  • targetImg Integer

Returns Object - {x: Integer, y: Integer, score: Float}

var img = getScreenshot();
var cropImg = cropImage(img, 350, 550, 150, 150);
var result = findImage(img, cropImg);
console.log(JSON.stringify(result)); // {"score":0.9999997615814209,"x":350,"y":550}
releaseImage(img);
releaseImage(cropImg);

findImages(sourceImg, targetImg, scoreLimit, resultCountLimit, withoutOverlap)

Same as findImage(), but find mulitple times.

  • sourceImg Integer

  • targetImg Integer

  • scoreLimit Integer

  • resultCountLimit Integer

  • withoutOverlap Boolean

Returns String - {"0": {"x": Integer, "y": Integer, "score": Float}, "1": {"x": Integer, "y": Integer, "score": Float}}, Key is String!

var img = getScreenshot();
var cropImg = cropImage(img, 350, 550, 150, 150);
var result = findImages(img, cropImg, 0.95, 3, true);
console.log(JSON.stringify(result)); // {"0":{"score":0.9999997615814209,"x":350,"y":550}}
releaseImage(img);
releaseImage(cropImg);

resizeImage(sourceImg, width, height)

Resize image.

  • width Integer

  • height Integer

Returns Integer - The image pointer

var img = getScreenshot();
var resizeImg = resizeImage(img, 108, 192);
saveImage(resizeImg, getStoragePath() + '/test_resize.png');
releaseImage(img);
releaseImage(resizeImg);

releaseImage(imgPtr)

Very Important! You should call this function with all imgPtrs.

  • imgPtr Integer

var img = getScreenshot(); // keep in memory
releaseImage(img); // release from memory

getImageColor(sourceImg, x, y)

Get color of point from an image.

  • sourceImg Integer

  • x Integer

  • y Integer

Returns Object - {r: Integer, g: Integer, b: Integer, a: Integer}

var img = getScreenshot();
var color = getImageColor(img, 100, 100);
console.log(JSON.stringify(color)); // {"a":0,"b":21,"g":36,"r":198}
releaseImage(img);

getImageSize(imgPtr)

  • imgPtr Integer

Returns Object - {width: Integer, height: Integer}

var img = getScreenshot();
var size = getImageSize(img);
console.log(JSON.stringify(size)); // {"height":1920,"width":1080}
releaseImage(img);

saveImage(imgPtr, path)

Save image to disk.

  • imgPtr Integer

  • path String

var img = getScreenshot();
saveImage(img, getStoragePath + '/test_save.png');
releaseImage(img);

openImage(path)

Open image from disk.

  • path String

Returns Integer - The image pointer

var img = openImage(getStoragePath + '/test_save.png');
releaseImage(img);

sleep(milliseconds)

Like sleep function in C language, pause current process.

  • milliseconds Integer

console.log('Hello');
sleep(1000);
console.log('Andy');

getStoragePath()

Get Robotmon folder. Like /sdcard/Robotmon.

Returns String - The storage path

console.log(getStoragePath());

getImageFromURL(url)

Get image from an url.

  • url String

Returns Integer - The image pointer

getImageFromBase64(base64)

Get image from a base64 string.

  • base64 String

Returns Integer - The image pointer

getBase64FromImage(imgPtr)

Get base64 string from an image.

  • imgPtr Integer

Returns String - base64

readFile(path)

Read a file as string.

  • path String

Returns String - The text of the file

writeFile(path, text)

Write a string to a file.

  • path String

  • text String

encrypt(script)

Encrypted a string

  • script String

Returns String - The encrypted script

runEncryptedScript(script)

Run a encrypted javascript string.

  • script String - The script is encrypted by encrypt

runScript(script)

Run a javascript string.

  • script String

httpClient(method, url, body, headers)

Do a http request.

  • method String

  • url String

  • body String

  • headers Object

Returns String - The result

httpClient('GET', 'http://httpbin.org/get', '', {});
httpClient('POST', 'http://httpbin.org/post', 'body data', {});
httpClient('POST', 'http://httpbin.org/post', 'foo=bar&bar=foo', {'Content-Type': 'application/x-www-form-urlencoded'});

importJS(library)

Import an JS library.

  • library String

importJS('RBM-0.0.2') // import shared library in libs
importJS('js/customerJS') // import local library

getVirtualButtonHeight()

Returns Integer - The height of the virtual button

PreviousJavaScript InterfaceNextRBM library APIs

Last updated