JavaScript Raw APIs

Low level APIs for controlling your android phone

getScreenSize()

Returns Object - {width: Integer, height: Integer}

var sizeObj = getScreenSize();
console.log(sizeObj.width, sizeObj.height);
// 1080 1920

getScreenshot()

Returns Integer - The image pointer

var img = getScreenshot();
console.log(img);
// 122344533 <- image pointer
releaseImage(img); // Don't forgot release a pointer

getScreenshotModify(cropX, cropY, cropWidth, cropHeight, resizeWidth, resizeHeight, qualitys)

Get screenshot, crop and resize. For speeding up screenshot.

• cropX Integer

• cropY Integer

• cropWidth Integer

• cropHeight Integer

• resizeWidth Integer

• resizeHeight Integer

• quality Integer

Returns Integer - The image pointer

var image = getScreenshotModify(200, 200, 100, 100, 50, 50, 80);
console.log(image); // image width = 50, height = 50 
// 12333122
releaseImage(image);

execute(command)

Call exec command in android system. It's permission is same as adb shell

• command String

Returns String - The result of the execution

var result = execute("ls -al /sdcard");
console.log(result);
// drwxr-xr-x   2 root  root    64B 12 14 23:44 Robotmon

tap(x, y, during)

Simulate a tap event

• x Integer

• y Integer

• during Integer

tap(200, 200, 10);
// Will inject a tap down and a tap up event to system

tapDown(x, y, during)

• x Integer

• y Integer

• during Integer

tapDown(200, 200, 40);
// Will inject a tapDown event to system

tapUp(x, y, during)

• x Integer

• y Integer

• during Integer

tapUp(200, 200, 40);
// Will inject a tapUo event to system

moveTo(x, y, during)

moveTo should be betewwn tapDown and tapUp

• x Integer

• y Integer

• during Integer

tapDown(500, 300, 40);
moveTo(500, 600, 40);
tapUp(500, 600, 40);
// Will inject a swipe down event

swipe(x1, y1, x2, y2, during)

Simulate a swipe event, using tapDown, moveTo and tapUp event. This function may not work in some game, you should implement yourself.

• x1 Integer

• y1 Integer

• x2 Integer

• y2 Integer

• during Integer

swipe(500, 300, 40); // same as above example
// Will inject a swipe down event

keycode(label, during)

Send a key code event to system Like adb shell input keyevent command Android Keycode List

• label String

• during Integer

keycode('HOME', 40); // same as keycode('KEYCODE_HOME', 40);
// Will send a HOME event to system

typing(words, during)

Only allow English words

• words String

• during Integer

typing('Hello!', 100);
// Will type 'H' 'e' 'l' 'l' 'o' '!' 6 words

OpenCV

clone(sourceImg)

Duplicate an image to another.

• sourceImg Integer

Returns Integer - The image pointer

var oriImage = getScreenshot();
for (var i = 0; i < 10; i++) {
  var cloneImage = clone(oriImage);
  // modify clone Image here
  smooth(cloneImage, 1, 5); // blur
  release(cloneImage);
}
release(oriImage);

smooth(sourceImg, smoothType, size)

Same as OpenCV smooth() function.

• sourceImg Integer

• smoothType Integer

• size Integer

smoothType	description
0	CV_BLUR_NO_SCALE
1	CV_BLUR
2	CV_GAUSSIAN
3	CV_MEDIAN
4	CV_BILATERAL

var img = getScreenshot();
smooth(img, 2, 5); // Gaussian blur
saveImage(img, getStoragePath + '/smooth.png');
releaseImage(img);

convertColor(sourceImg, code)

Same as OpenCV cvtColor(). Not support different channels. If you want to convert to gray, please use bgrToGray. Note that getScreenshot and getScreenshotModify is BGR order;

• sourceImg Integer

• code Integer

code	description
40	CV_BGR2HSV
52	CV_BGR2HLS

See more: OpenCV Types

var img = getScreenshot();
// Convert BGR to HSV color
convertColor(img, 40);
releaseImage(img);

bgrToGray(sourceImg)

Convert form bgr (3 channels) to gray (1 channel).

• sourceImg Integer

Returns Integer - The gray image pointer

var img = getScreenshot();
var gray = bgrToGray(img); // gray image
releaseImage(img);
releaseImage(gray);

absDiff(sourceImg, targetImg)

Same as OpenCV adbdiff().

• sourceImg Integer

• targetImg Integer

Returns Integer - The image pointer of the difference

var img1 = getScreenshot();
sleep(100);
var img2 = getScreenshot();
var diff = absDiff(img1, img2); // in gray order
releaseImage(img1);
releaseImage(img2);
releaseImage(diff);

threshold(sourceImg, thr, maxThr, code)

Same as OpenCV threshold().

• sourceImg Integer

• thr Float

• maxThr Float

• code Integer

code	description
0	CV_THRES_BINARY

See more: OpenCV Types

keycode('MENU');
sleep(1000);
var img1 = getScreenshot();
keycode('HOME');
sleep(1000);
var img2 = getScreenshot();
var diff = absDiff(img1, img2); // in gray order
threshold(diff, 100, 255); // set to 0 if <= 100, set to 255 if > 100
var value = getImageColor(diff, 500, 200); // value => {r":255,"g":0,"b":0","a":0}
console.log(value['r']); // current diff value is show on 'r'
// 255
releaseImage(img1);
releaseImage(img2);
releaseImage(diff);

eroid(sourceImg, width, height, x, y)

Same as OpenCV eroid.

width, height, x, y is getStructuringElement() parameters.

• sourceImg Integer

• width Integer

• height Integer

• x Integer

• y Integer

var img = getScreenshot();
threshold(img, 100, 255);
eroid(img, 3, 3, 1, 1);
saveImage(img, getStoragePath() + '/test_eroid.png');
releaseImage(img);

dilate(sourceImg, width, height, x, y)

Same as OpenCV dilate.

width, height, x, y is getStructuringElement() parameters.

• sourceImg Integer

• width Integer

• height Integer

• x Integer

• y Integer

var img = getScreenshot();
threshold(img, 100, 255);
dilate(img, 3, 3, 1, 1);
saveImage(img, getStoragePath() + '/test_dilate.png');
releaseImage(img);

inRange(sourceImg, minB, minG, minR, minA, maxB, maxG, maxR, maxA)

Same as OpenCV inRange + clone + mask. Filter with range color and clone to new image.

• sourceImg Integer

• minB Integer

• minG Integer

• minR Integer

• minA Integer

• maxB Integer

• maxG Integer

• maxR Integer

• maxA Integer

Returns Integer - The filtered image pointer

var img = getScreenshot();
var filteredImg = inRange(img, 0, 255, 255, 255, 255, 255, 255, 255); // only keep blue color pixel
saveImage(filteredImg, getStoragePath() + '/test_filterd.png');
releaseImage(img);
releaseImage(filteredImg);

outRange(sourceImg, minB, minG, minR, minA, maxB, maxG, maxR, maxA)

Same as OpenCV inRange + clone + not + mask. Filter without range color and clone to new image.

• sourceImg Integer

• minB Integer

• minG Integer

• minR Integer

• minA Integer

• maxB Integer

• maxG Integer

• maxR Integer

• maxA Integer

Returns Integer - The filtered image pointer

var img = getScreenshot();
var filteredImg = outRange(img, 0, 255, 255, 255, 255, 255, 255, 255); // keep all but blue color
saveImage(filteredImg, getStoragePath() + '/test_filterd.png');
releaseImage(img);
releaseImage(filteredImg);

cloneWithMask(sourceImg, mask)

Same as OpenCV copyTo. Clone image with mask (only support 1 channel)

• sourceImg Integer

• mask Integer

Returns Integer - new image pointer with mask

var img1 = getScreenshot();
sleep(100);
var img2 = getScreenshot();
var diff = absDiff(img1, img2);
sleep(100);
var img3 = cloneWithMask(img1, diff);
releaseImage(img1);
releaseImage(img2);
releaseImage(img3);
releaseImage(diff);

houghCircles(sourceImg, method, dp, minDist, p1, p2, minR, maxR)

Same as OpenCV houghCircles. For finding circles.

• sourceImg Integer

• method Integer (3 = CV_HOUGH_GRADIENT)

• dp Float (1) (ratio between input image and input params.)

• minDist Float (min distance between circles)

• p1 Float (canny parameter)

• p2 Float (canny parameter)

• minR Integer (min radius)

• maxR Integer (max radius)

Returns Object - Array of circles

var img = getScreenshot();
var points = houghCircles(img, 3, 1, 8, 4, 8, 6, 14);
console.log(points); // {"0": {"x": 102, "y": "233", "r": 9}}
releaseImage(img);

canny(sourceImg, t1, t2, apertureSize)

Same as OpenCV canny

• sourceImg Integer

• t1 Float

• t2 Float

• apertureSize Integer

Returns Integer - The canny image pointer

var img = getScreenshot();
threshold(img, 30, 255);
eroid(img, 5, 5, 1, 1);
var cannyImg = canny(img, 50, 150, 3);
saveImage(cannyImg, getStoragePath() + '/test_canny.png');
releaseImage(img);
releaseImage(cannyImg);

findContours(cannyImgPtr, minArea, maxArea)

Same as OpenCV findContours.

• cannyImgPtr Integer (Canny image as input)

• minArea Float

• maxArea Float

Returns Object - {"0": {x: Integer, y: Integer}

var img = getScreenshot();
threshold(img, 30, 255);
eroid(img, 5, 5, 1, 1);
var cannyImg = canny(img, 50, 150, 3);
var results = findContours(cannyImg, 1000, 10000); // area > 100
console.log(JSON.stringify(results));
// {"0":{"x":537,"y":1850},"1":{"x":133,"y":601}}
releaseImage(img);
releaseImage(cannyImg);

drawCircle(sourceImg, x, y, radius, r, g, b, a)

Draw circle in an image.

• sourceImg Integer

• x Integer

• y Integer

• radius Integer

• r Integer

• g Integer

• b Integer

• a Integer

var img = getScreenshot();
drawCircle(img, 100, 100, 10, 0, 0, 255, 0); // draw a blue circle
saveImage(img, getStoragePath() + '/test_drawCircle.png');
releaseImage(img);

getIdentityScore(sourceImg, targetImg)

• sourceImg Integer

• targetImg Integer

Returns Float - The identity score

keycode('MENU');
sleep(1000);
var img1 = getScreenshot();
keycode('HOME');
sleep(1000);
var img2 = getScreenshot();
var score = getIdentityScore(img1, img2);
console.log(score); // 0.6004924774169922
releaseImage(img1);
releaseImage(img2);

cropImage(sourceImg, x, y, width, height)

Crop image.

• x Integer

• y Integer

• width Integer

• height Integer

Returns Integer - The image pointer

var img = getScreenshot();
var cropImg = cropImage(img, 350, 550, 150, 150);
saveImage(cropImg, getStoragePath() + '/test_crop.png');
releaseImage(img);
releaseImage(cropImg);

findImage(sourceImg, targetImg)

Using OpenCV Template Match to fing image.

• sourceImg Integer

• targetImg Integer

Returns Object - {x: Integer, y: Integer, score: Float}

var img = getScreenshot();
var cropImg = cropImage(img, 350, 550, 150, 150);
var result = findImage(img, cropImg);
console.log(JSON.stringify(result)); // {"score":0.9999997615814209,"x":350,"y":550}
releaseImage(img);
releaseImage(cropImg);

findImages(sourceImg, targetImg, scoreLimit, resultCountLimit, withoutOverlap)

Same as findImage(), but find mulitple times.

• sourceImg Integer

• targetImg Integer

• scoreLimit Integer

• resultCountLimit Integer

• withoutOverlap Boolean

Returns String - {"0": {"x": Integer, "y": Integer, "score": Float}, "1": {"x": Integer, "y": Integer, "score": Float}}, Key is String!

var img = getScreenshot();
var cropImg = cropImage(img, 350, 550, 150, 150);
var result = findImages(img, cropImg, 0.95, 3, true);
console.log(JSON.stringify(result)); // {"0":{"score":0.9999997615814209,"x":350,"y":550}}
releaseImage(img);
releaseImage(cropImg);

resizeImage(sourceImg, width, height)

Resize image.

• width Integer

• height Integer

Returns Integer - The image pointer

var img = getScreenshot();
var resizeImg = resizeImage(img, 108, 192);
saveImage(resizeImg, getStoragePath() + '/test_resize.png');
releaseImage(img);
releaseImage(resizeImg);

releaseImage(imgPtr)

Very Important! You should call this function with all imgPtrs.

• imgPtr Integer

var img = getScreenshot(); // keep in memory
releaseImage(img); // release from memory

getImageColor(sourceImg, x, y)

Get color of point from an image.

• sourceImg Integer

• x Integer

• y Integer

Returns Object - {r: Integer, g: Integer, b: Integer, a: Integer}

var img = getScreenshot();
var color = getImageColor(img, 100, 100);
console.log(JSON.stringify(color)); // {"a":0,"b":21,"g":36,"r":198}
releaseImage(img);

getImageSize(imgPtr)

• imgPtr Integer

Returns Object - {width: Integer, height: Integer}

var img = getScreenshot();
var size = getImageSize(img);
console.log(JSON.stringify(size)); // {"height":1920,"width":1080}
releaseImage(img);

saveImage(imgPtr, path)

Save image to disk.

• imgPtr Integer

• path String

var img = getScreenshot();
saveImage(img, getStoragePath + '/test_save.png');
releaseImage(img);

openImage(path)

Open image from disk.

• path String

Returns Integer - The image pointer

var img = openImage(getStoragePath + '/test_save.png');
releaseImage(img);

sleep(milliseconds)

Like sleep function in C language, pause current process.

• milliseconds Integer

console.log('Hello');
sleep(1000);
console.log('Andy');

getStoragePath()

Get Robotmon folder. Like /sdcard/Robotmon.

Returns String - The storage path

console.log(getStoragePath());

getImageFromURL(url)

Get image from an url.

• url String

Returns Integer - The image pointer

getImageFromBase64(base64)

Get image from a base64 string.

• base64 String

Returns Integer - The image pointer

getBase64FromImage(imgPtr)

Get base64 string from an image.

• imgPtr Integer

Returns String - base64

readFile(path)

Read a file as string.

• path String

Returns String - The text of the file

writeFile(path, text)

Write a string to a file.

• path String

• text String

encrypt(script)

Encrypted a string

• script String

Returns String - The encrypted script

runEncryptedScript(script)

Run a encrypted javascript string.

• script String - The script is encrypted by encrypt

runScript(script)

Run a javascript string.

• script String

httpClient(method, url, body, headers)

Do a http request.

• method String

• url String

• body String

• headers Object

Returns String - The result

httpClient('GET', 'http://httpbin.org/get', '', {});
httpClient('POST', 'http://httpbin.org/post', 'body data', {});
httpClient('POST', 'http://httpbin.org/post', 'foo=bar&bar=foo', {'Content-Type': 'application/x-www-form-urlencoded'});

importJS(library)

Import an JS library.

• library String

importJS('RBM-0.0.2') // import shared library in libs
importJS('js/customerJS') // import local library

getVirtualButtonHeight()

Returns Integer - The height of the virtual button