[ADB] Android device remote control using by UI Automator

Android/Android Debug Bridge

[ADB] Android device remote control using by UI Automator

Lowell Ahn 2022. 3. 15. 01:04

What's impossible?

I think there is no impossible thing. It is in ourself whether it makes possible.

The meaning is completly changed if we add '(single quotation mark) into the word of "Impossible".

"I'm possible with you". Let's find what we can.

Android OS is widely applied to the product which is not only for smartphone but also other devices such as wearable, Automotive, etc.
There are several ways to control the Android OS based devices and many developer for R&D, QA, etc. has been using Java, Python which well known as programming language for good compatibiliy with Android OS.

In this article, we would like to focus on the learning how we can control android device without the complex programming with the language on above.

As a practice,

I will explain what we have to do for push the icon which has following image on android device.

Basically, Android OS provides ADB (Android Debug Bridge) Interface and there are several packages compatible with ADB (In case of several related packages with ADB, we will check it after basic explanation about ADB interface).

Let's check the way to use ADB interface step by step.

1) Android SDK Platform-Tools package

(https://developer.android.com/studio/command-line/adb?gclid=Cj0KCQjwz7uRBhDRARIsAFqjulkzgXNcEpL01CGES8aVypbdZ0GShgesAYaja4R4e2ahbH2LG3AHmCMaAkTKEALw_wcB&gclsrc=aw.ds)

Required files will be installed in "<user defined directory>/platform-tools/"

2) System Variable setting

For the convenient usage without path definition, it is better to set system variable (PATH) for ADB path.

Windows - Find - "Edit the system environment variables"

a) Environment Variables

b) Edit Path in System Variables

c) New - input the path("<user defined directory>/platform-tools/")

path setting on System environment variables

3) Enable developer option

For activating adb interface on the device to be controlled, "USB debugging mode" has to be enabled in developer option and we have to makes it enabled on the device because developer option is hidden menu.

And also, we need to check the cursor position and the boundary of each control area.

Please do following steps.

Show Current position of cursor / boundary of controls

4) Check device connection via adb interface

If adb connection is normal, unique device ID value will be shown as response.

5) Execute adb command to get the output of uiautomator

adb command example for acquiring current layout structure

Following output XML can be obtained after the adb command execution.

<?xml version='1.0' encoding='UTF-8' standalone='yes' ?><hierarchy rotation="0"><node index="0" text="" resource-id="" c
lass="android.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false"
 clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="
false" selected="false" bounds="[0,0][840,2081]"><node index="0" text="" resource-id="" class="android.widget.LinearLayo
ut" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="tr
ue" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds
="[0,0][840,2081]"><node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.samsung.androi
d.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="
false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,82][840,2081]"><node index
="0" text="" resource-id="com.samsung.android.dialer:id/action_bar_root" class="android.widget.LinearLayout" package="co
m.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="
false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,82][840,20
81]"><node index="0" text="" resource-id="android:id/content" class="android.widget.FrameLayout" package="com.samsung.an
droid.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focus
ed="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,82][840,2081]"><node i
ndex="0" text="" resource-id="com.samsung.android.dialer:id/dialtacts_main_layout" class="android.view.ViewGroup" packag
e="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusa
ble="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,82][8
40,2081]"><node index="0" text="" resource-id="com.samsung.android.dialer:id/two_pane_container" class="android.widget.L
inearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" en
abled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="fals
e" bounds="[0,82][840,2005]"><node index="0" text="" resource-id="com.samsung.android.dialer:id/coordinator_layout" clas
s="android.view.ViewGroup" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clicka
ble="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false"
selected="false" bounds="[0,82][840,2005]"><node index="0" text="" resource-id="com.samsung.android.dialer:id/appbar_lay
out" class="android.widget.LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked=
"false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" pas
sword="false" selected="false" bounds="[0,82][840,292]"><node index="0" text="" resource-id="com.samsung.android.dialer:
id/collapsing_toolbar" class="android.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="" checkable
="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clic
kable="false" password="false" selected="false" bounds="[0,82][840,292]"><node index="0" text="" resource-id="com.samsun
g.android.dialer:id/collapsing_appbar_title_layout_parent" class="android.widget.LinearLayout" package="com.samsung.andr
oid.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused
="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,82][840,292]" /><node in
dex="1" text="" resource-id="com.samsung.android.dialer:id/dialtacts_toolbar" class="android.view.ViewGroup" package="co
m.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="
false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,82][840,29
2]"><node index="0" text="Phone" resource-id="" class="android.widget.TextView" package="com.samsung.android.dialer" con
tent-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scroll
able="false" long-clickable="false" password="false" selected="false" bounds="[63,178][179,237]" /><node index="1" text=
"" resource-id="" class="androidx.appcompat.widget.LinearLayoutCompat" package="com.samsung.android.dialer" content-desc
="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="fal
se" long-clickable="false" password="false" selected="false" bounds="[601,124][840,292]"><node index="0" text="" resourc
e-id="com.samsung.android.dialer:id/dialer_menu_search" class="android.widget.Button" package="com.samsung.android.diale
r" content-desc="Search" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="fal
se" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[601,124][716,292]" /><node index
="1" text="" resource-id="com.samsung.android.dialer:id/sesl_action_bar_overflow_button" class="android.widget.FrameLayo
ut" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="tr
ue" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds
="[716,124][840,292]"><node index="0" text="" resource-id="" class="android.widget.ImageView" package="com.samsung.andro
id.dialer" content-desc="More options" checkable="false" checked="false" clickable="true" enabled="true" focusable="true
" focused="false" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[716,124][840,292]"
 /></node></node></node></node></node><node index="1" text="" resource-id="com.samsung.android.dialer:id/dialtacts_left_
container" class="android.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" che
cked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false
" password="false" selected="false" bounds="[0,292][840,2005]"><node index="0" text="" resource-id="com.samsung.android.
dialer:id/keypad_fragment" class="android.widget.RelativeLayout" package="com.samsung.android.dialer" content-desc="" ch
eckable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" lo
ng-clickable="false" password="false" selected="false" bounds="[0,292][840,2005]"><node index="0" text="" resource-id="c
om.samsung.android.dialer:id/dialpad_spacer_view" class="android.view.View" package="com.samsung.android.dialer" content
-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable
="false" long-clickable="true" password="false" selected="false" bounds="[0,292][840,653]" /><node index="1" text="" res
ource-id="com.samsung.android.dialer:id/keypad_view" class="android.view.ViewGroup" package="com.samsung.android.dialer"
 content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" sc
rollable="false" long-clickable="false" password="false" selected="false" bounds="[0,653][840,2005]"><node index="0" tex
t="" resource-id="com.samsung.android.dialer:id/input_number_container" class="android.widget.FrameLayout" package="com.
samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="fa
lse" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,653][840,909
]"><node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.samsung.android.dialer" conten
t-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollabl
e="false" long-clickable="false" password="false" selected="false" bounds="[0,653][840,909]"><node index="0" text="" res
ource-id="" class="android.widget.LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" c
hecked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="fal
se" password="false" selected="false" bounds="[0,653][840,909]"><node NAF="true" index="0" text="" resource-id="com.sams
ung.android.dialer:id/dialpad_edit_text_container" class="android.widget.LinearLayout" package="com.samsung.android.dial
er" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" s
crollable="false" long-clickable="false" password="false" selected="false" bounds="[0,653][840,909]"><node NAF="true" in
dex="0" text="" resource-id="com.samsung.android.dialer:id/digits" class="android.widget.EditText" package="com.samsung.
android.dialer" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focus
ed="true" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[63,653][777,909]" /></node
></node></node></node><node index="1" text="" resource-id="com.samsung.android.dialer:id/dialpad_container" class="andro
id.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable=
"false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" sele
cted="false" bounds="[0,909][840,1745]"><node index="0" text="" resource-id="com.samsung.android.dialer:id/dialpad_keypa
d_with_side" class="android.widget.LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false"
checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="fa
lse" password="false" selected="false" bounds="[0,909][840,1745]"><node index="0" text="" resource-id="com.samsung.andro
id.dialer:id/dialpad_keypad_only" class="android.widget.LinearLayout" package="com.samsung.android.dialer" content-desc=
"" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="fals
e" long-clickable="false" password="false" selected="false" bounds="[0,909][840,1745]"><node index="0" text="" resource-
id="com.samsung.android.dialer:id/container_first_row" class="android.widget.LinearLayout" package="com.samsung.android.
dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="fa
lse" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[63,909][777,1118]"><node index
="0" text="" resource-id="com.samsung.android.dialer:id/one" class="android.widget.RelativeLayout" package="com.samsung.
android.dialer" content-desc="1&#10;" checkable="false" checked="false" clickable="true" enabled="true" focusable="true"
 focused="false" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[63,909][301,1118]">
<node index="0" text="1" resource-id="com.samsung.android.dialer:id/dialpad_key_number" class="android.widget.TextView"
package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true"
focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[1
59,946][204,1062]" /></node><node index="1" text="" resource-id="com.samsung.android.dialer:id/two" class="android.widge
t.RelativeLayout" package="com.samsung.android.dialer" content-desc="2&#10;,A,B,C" checkable="false" checked="false" cli
ckable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="true" password="false"
selected="false" bounds="[301,909][539,1118]"><node index="0" text="ABC" resource-id="com.samsung.android.dialer:id/dial
pad_key_letters" class="android.widget.TextView" package="com.samsung.android.dialer" content-desc=" " checkable="false"
 checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="f
alse" password="false" selected="false" bounds="[392,1041][447,1084]" /><node index="1" text="2" resource-id="com.samsun
g.android.dialer:id/dialpad_key_number" class="android.widget.TextView" package="com.samsung.android.dialer" content-des
c="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="fa
lse" long-clickable="false" password="false" selected="false" bounds="[397,943][442,1059]" /></node><node index="2" text
="" resource-id="com.samsung.android.dialer:id/three" class="android.widget.RelativeLayout" package="com.samsung.android
.dialer" content-desc="3&#10;,D,E,F" checkable="false" checked="false" clickable="true" enabled="true" focusable="true"
focused="false" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[539,909][777,1118]">
<node index="0" text="DEF" resource-id="com.samsung.android.dialer:id/dialpad_key_letters" class="android.widget.TextVie
w" package="com.samsung.android.dialer" content-desc=" " checkable="false" checked="false" clickable="false" enabled="tr
ue" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds
="[631,1041][684,1084]" /><node index="1" text="3" resource-id="com.samsung.android.dialer:id/dialpad_key_number" class=
"android.widget.TextView" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickab
le="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" s
elected="false" bounds="[635,943][680,1059]" /></node></node><node index="1" text="" resource-id="com.samsung.android.di
aler:id/container_second_row" class="android.widget.LinearLayout" package="com.samsung.android.dialer" content-desc="" c
heckable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" l
ong-clickable="false" password="false" selected="false" bounds="[63,1118][777,1327]"><node index="0" text="" resource-id
="com.samsung.android.dialer:id/four" class="android.widget.RelativeLayout" package="com.samsung.android.dialer" content
-desc="4&#10;,G,H,I" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false"
scrollable="false" long-clickable="true" password="false" selected="false" bounds="[63,1118][301,1327]"><node index="0"
text="GHI" resource-id="com.samsung.android.dialer:id/dialpad_key_letters" class="android.widget.TextView" package="com.
samsung.android.dialer" content-desc=" " checkable="false" checked="false" clickable="false" enabled="true" focusable="f
alse" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[155,1250][209
,1293]" /><node index="1" text="4" resource-id="com.samsung.android.dialer:id/dialpad_key_number" class="android.widget.
TextView" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabl
ed="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false"
bounds="[159,1152][204,1268]" /></node><node index="1" text="" resource-id="com.samsung.android.dialer:id/five" class="a
ndroid.widget.RelativeLayout" package="com.samsung.android.dialer" content-desc="5&#10;,J,K,L" checkable="false" checked
="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="true" passw
ord="false" selected="false" bounds="[301,1118][539,1327]"><node index="0" text="JKL" resource-id="com.samsung.android.d
ialer:id/dialpad_key_letters" class="android.widget.TextView" package="com.samsung.android.dialer" content-desc=" " chec
kable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long
-clickable="false" password="false" selected="false" bounds="[395,1250][445,1293]" /><node index="1" text="5" resource-i
d="com.samsung.android.dialer:id/dialpad_key_number" class="android.widget.TextView" package="com.samsung.android.dialer
" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" s
crollable="false" long-clickable="false" password="false" selected="false" bounds="[397,1152][442,1268]" /></node><node
index="2" text="" resource-id="com.samsung.android.dialer:id/six" class="android.widget.RelativeLayout" package="com.sam
sung.android.dialer" content-desc="6&#10;,M,N,O" checkable="false" checked="false" clickable="true" enabled="true" focus
able="true" focused="false" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[539,1118
][777,1327]"><node index="0" text="MNO" resource-id="com.samsung.android.dialer:id/dialpad_key_letters" class="android.w
idget.TextView" package="com.samsung.android.dialer" content-desc=" " checkable="false" checked="false" clickable="false
" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="
false" bounds="[628,1250][688,1293]" /><node index="1" text="6" resource-id="com.samsung.android.dialer:id/dialpad_key_n
umber" class="android.widget.TextView" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="f
alse" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" passw
ord="false" selected="false" bounds="[635,1152][680,1268]" /></node></node><node index="2" text="" resource-id="com.sams
ung.android.dialer:id/container_third_row" class="android.widget.LinearLayout" package="com.samsung.android.dialer" cont
ent-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrolla
ble="false" long-clickable="false" password="false" selected="false" bounds="[63,1327][777,1536]"><node index="0" text="
" resource-id="com.samsung.android.dialer:id/seven" class="android.widget.RelativeLayout" package="com.samsung.android.d
ialer" content-desc="7&#10;,P,Q,R,S" checkable="false" checked="false" clickable="true" enabled="true" focusable="true"
focused="false" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[63,1327][301,1536]">
<node index="0" text="PQRS" resource-id="com.samsung.android.dialer:id/dialpad_key_letters" class="android.widget.TextVi
ew" package="com.samsung.android.dialer" content-desc=" " checkable="false" checked="false" clickable="false" enabled="t
rue" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bound
s="[144,1459][219,1502]" /><node index="1" text="7" resource-id="com.samsung.android.dialer:id/dialpad_key_number" class
="android.widget.TextView" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clicka
ble="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false"
selected="false" bounds="[159,1361][204,1477]" /></node><node index="1" text="" resource-id="com.samsung.android.dialer:
id/eight" class="android.widget.RelativeLayout" package="com.samsung.android.dialer" content-desc="8&#10;,T,U,V" checkab
le="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clic
kable="true" password="false" selected="false" bounds="[301,1327][539,1536]"><node index="0" text="TUV" resource-id="com
.samsung.android.dialer:id/dialpad_key_letters" class="android.widget.TextView" package="com.samsung.android.dialer" con
tent-desc=" " checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrol
lable="false" long-clickable="false" password="false" selected="false" bounds="[392,1459][447,1502]" /><node index="1" t
ext="8" resource-id="com.samsung.android.dialer:id/dialpad_key_number" class="android.widget.TextView" package="com.sams
ung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false"
 focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[397,1361][442,1477
]" /></node><node index="2" text="" resource-id="com.samsung.android.dialer:id/nine" class="android.widget.RelativeLayou
t" package="com.samsung.android.dialer" content-desc="9&#10;,W,X,Y,Z" checkable="false" checked="false" clickable="true"
 enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="true" password="false" selected="fal
se" bounds="[539,1327][777,1536]"><node index="0" text="WXYZ" resource-id="com.samsung.android.dialer:id/dialpad_key_let
ters" class="android.widget.TextView" package="com.samsung.android.dialer" content-desc=" " checkable="false" checked="f
alse" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" passw
ord="false" selected="false" bounds="[619,1459][696,1502]" /><node index="1" text="9" resource-id="com.samsung.android.d
ialer:id/dialpad_key_number" class="android.widget.TextView" package="com.samsung.android.dialer" content-desc="" checka
ble="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-c
lickable="false" password="false" selected="false" bounds="[635,1361][680,1477]" /></node></node><node index="3" text=""
 resource-id="com.samsung.android.dialer:id/container_forth_row" class="android.widget.LinearLayout" package="com.samsun
g.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" f
ocused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[63,1536][777,1745]">
<node index="0" text="" resource-id="com.samsung.android.dialer:id/star" class="android.widget.RelativeLayout" package="
com.samsung.android.dialer" content-desc="Asterisk" checkable="false" checked="false" clickable="true" enabled="true" fo
cusable="true" focused="false" scrollable="false" long-clickable="true" password="false" selected="false" bounds="[63,15
36][301,1745]"><node index="0" text="∗" resource-id="com.samsung.android.dialer:id/dialpad_key_number" class="android.wi
dget.TextView" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false"
enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="fa
lse" bounds="[154,1579][210,1691]" /></node><node index="1" text="" resource-id="com.samsung.android.dialer:id/zero" cla
ss="android.widget.RelativeLayout" package="com.samsung.android.dialer" content-desc="0&#10;,+" checkable="false" checke
d="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="true" pass
word="false" selected="false" bounds="[301,1536][539,1745]"><node index="0" text="0" resource-id="com.samsung.android.di
aler:id/dialpad_key_number" class="android.widget.TextView" package="com.samsung.android.dialer" content-desc="" checkab
le="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-cl
ickable="false" password="false" selected="false" bounds="[397,1582][442,1698]" /></node><node index="2" text="" resourc
e-id="com.samsung.android.dialer:id/pound" class="android.widget.RelativeLayout" package="com.samsung.android.dialer" co
ntent-desc="Pound" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" sc
rollable="false" long-clickable="true" password="false" selected="false" bounds="[539,1536][777,1745]"><node index="0" t
ext="#" resource-id="com.samsung.android.dialer:id/dialpad_key_number" class="android.widget.TextView" package="com.sams
ung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false"
 focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[630,1579][685,1691
]" /></node></node></node></node></node><node index="2" text="" resource-id="com.samsung.android.dialer:id/dial_button_c
ontainer" class="android.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" chec
ked="false" clickable="true" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false"
password="false" selected="false" bounds="[0,1745][840,2005]"><node index="0" text="" resource-id="" class="android.widg
et.LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false
" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="
false" bounds="[0,1745][840,2005]"><node index="0" text="" resource-id="com.samsung.android.dialer:id/callButtonContaine
r" class="android.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="fa
lse" clickable="true" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" passwor
d="false" selected="false" bounds="[301,1745][539,2005]"><node index="0" text="" resource-id="com.samsung.android.dialer
:id/dialButton" class="android.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="Call button" check
able="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-cl
ickable="true" password="false" selected="false" bounds="[334,1789][505,1960]"><node index="0" text="" resource-id="com.
samsung.android.dialer:id/dialButtonImage" class="android.widget.ImageView" package="com.samsung.android.dialer" content
-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable
="false" long-clickable="false" password="false" selected="false" bounds="[367,1822][472,1927]" /></node></node></node><
/node></node></node></node></node></node><node index="1" text="" resource-id="com.samsung.android.dialer:id/dialtacts_bo
ttom_outer_layout" class="android.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="" checkable="fa
lse" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickabl
e="false" password="false" selected="false" bounds="[0,2005][840,2081]"><node index="0" text="" resource-id="com.samsung
.android.dialer:id/dialtacts_tab_layout" class="android.widget.HorizontalScrollView" package="com.samsung.android.dialer
" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" s
crollable="false" long-clickable="false" password="false" selected="false" bounds="[0,2005][840,2081]"><node index="0" t
ext="" resource-id="com.samsung.android.dialer:id/dialpad_tab_content_view" class="android.widget.LinearLayout" package=
"com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusabl
e="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[42,2005][
798,2081]"><node index="0" text="" resource-id="com.samsung.android.dialer:id/dialpad_tab_button" class="android.widget.
LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="true" en
abled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false
" bounds="[42,2005][231,2081]"><node index="0" text="" resource-id="com.samsung.android.dialer:id/tab_text_container" cl
ass="android.widget.LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false"
 clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="
false" selected="false" bounds="[78,2051][195,2081]"><node index="0" text="Keypad" resource-id="com.samsung.android.dial
er:id/tab_text_view" class="android.widget.TextView" package="com.samsung.android.dialer" content-desc="" checkable="fal
se" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable
="false" password="false" selected="true" bounds="[78,2064][195,2081]" /><node index="1" text="" resource-id="com.samsun
g.android.dialer:id/tab_strip" class="android.widget.ImageView" package="com.samsung.android.dialer" content-desc="" che
ckable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" lon
g-clickable="false" password="false" selected="false" bounds="[0,0][0,0]" /></node></node><node index="1" text="" resour
ce-id="com.samsung.android.dialer:id/calllog_tab_button" class="android.widget.LinearLayout" package="com.samsung.androi
d.dialer" content-desc="" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="fa
lse" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[231,2005][420,2081]"><node ind
ex="0" text="" resource-id="com.samsung.android.dialer:id/tab_text_container" class="android.widget.LinearLayout" packag
e="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusa
ble="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[259,205
1][392,2081]"><node index="0" text="Recents" resource-id="com.samsung.android.dialer:id/tab_text_view" class="android.wi
dget.TextView" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false"
enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="fa
lse" bounds="[259,2064][392,2081]" /></node></node><node index="2" text="" resource-id="com.samsung.android.dialer:id/co
ntactlist_tab_button" class="android.widget.LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable
="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clicka
ble="false" password="false" selected="false" bounds="[420,2005][609,2081]"><node index="0" text="" resource-id="com.sam
sung.android.dialer:id/tab_text_container" class="android.widget.LinearLayout" package="com.samsung.android.dialer" cont
ent-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrolla
ble="false" long-clickable="false" password="false" selected="false" bounds="[441,2051][587,2081]"><node index="0" text=
"Contacts" resource-id="com.samsung.android.dialer:id/tab_text_view" class="android.widget.TextView" package="com.samsun
g.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" f
ocused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[441,2064][587,2081]"
 /></node></node><node index="3" text="" resource-id="com.samsung.android.dialer:id/places_tab_button" class="android.wi
dget.LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="tru
e" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="
false" bounds="[609,2005][798,2081]"><node index="0" text="" resource-id="com.samsung.android.dialer:id/tab_text_contain
er" class="android.widget.LinearLayout" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="
false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" pass
word="false" selected="false" bounds="[650,2051][757,2081]"><node index="0" text="Places" resource-id="com.samsung.andro
id.dialer:id/tab_text_view" class="android.widget.TextView" package="com.samsung.android.dialer" content-desc="" checkab
le="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-cl
ickable="false" password="false" selected="false" bounds="[650,2064][757,2081]" /></node></node></node></node></node></n
ode></node></node></node></node><node index="1" text="" resource-id="android:id/statusBarBackground" class="android.view
.View" package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled=
"true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bou
nds="[0,0][840,82]" /><node index="2" text="" resource-id="android:id/navigationBarBackground" class="android.view.View"
 package="com.samsung.android.dialer" content-desc="" checkable="false" checked="false" clickable="false" enabled="true"
 focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[
0,0][0,0]" /></node></hierarchy>

Actually, each controls on Android Layout has specific "resource-id" value.

In case of the contents described on above whole XML data,

There is "Call button" as content-desc attribute and clickable is true in additional attribute of Node.

I think we can make customized algorithm to trigger other attribute's value change in accordance with manual operation of Android device in real-time.

<node index="0" text="" resource-id="com.samsung.android.dialer
:id/dialButton" class="android.widget.FrameLayout" package="com.samsung.android.dialer" content-desc="Call button" check
able="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-cl
ickable="true" password="false" selected="false" bounds="[334,1789][505,1960]">

6) Get actual XY coordinates of target control

Anyway, we can get following resource-id as suitable for target resource to be remotely controlled.

"com.samsung.android.dialer:id/dialButton" resource-id is included in node attributes.

In the node, bounds attribute values indicates the position and size information of resource.

bounds="[334,1789][505,1960]"

[334 (X.Left) , 1789 (Y.Top)]

[505 (X.Right), 1960 (Y.Bottom)]

For getting center position of the control,

we can calculate XY coordinates with following equation.

X = 334 + round{(505-334 + 1) / 2} => 420

Y = 1789 + round{(1960 – 1789 + 1) / 2} => 1875

7) adb command for executing target control

As you can see, target control is tapped with the calculated XY coordinates and it works well.

This is just one of examples to control android devices.

I think there are any other efficient ways has more compatibility and flexibility than this approach.

I hope you can understand that the purpose of this activities and the insight, it is for working more smart without the barrier of thinking.