README.md 14.6 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1
2
ioppytest framework:
--------------------
Federico Sismondi's avatar
Federico Sismondi committed
3

Federico Sismondi's avatar
Federico Sismondi committed
4
ioppytest is a framework for running interoperability tests.
Federico Sismondi's avatar
Federico Sismondi committed
5

6
7
This initial version tackles technical interoperability testing (CoAP,
 LwM2M, 6LoWPAN and OneM2M interop tests).
Federico Sismondi's avatar
Federico Sismondi committed
8
9
10
11
12
13

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

This can be run as standalone software and also integrated to f-interop
14
platform (go.f-interop.eu)
Federico Sismondi's avatar
Federico Sismondi committed
15
16


Federico Sismondi's avatar
Federico Sismondi committed
17
18
Implemented test suites in the ioppytest framework:
---------------------------------------------------
Federico Sismondi's avatar
Federico Sismondi committed
19

Federico Sismondi's avatar
Federico Sismondi committed
20
21
The test suites implemented which are currently supported by the
framework are:
Federico Sismondi's avatar
Federico Sismondi committed
22

Federico Sismondi's avatar
Federico Sismondi committed
23
24
25
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
26
27
- LwM2M Test Suite (between two users' IUT) (WIP)
- oneM2M Test Suite (between two users' IUT) (WIP)
Federico Sismondi's avatar
Federico Sismondi committed
28

29
30


Federico Sismondi's avatar
Federico Sismondi committed
31
32
Test setup:
-----------
Federico Sismondi's avatar
Federico Sismondi committed
33

Federico Sismondi's avatar
Federico Sismondi committed
34
35
36
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
37

Federico Sismondi's avatar
Federico Sismondi committed
38
39
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
40
41

```
Federico Sismondi's avatar
Federico Sismondi committed
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
                    +----------------------------+             +----------------------------+             +----------------------------+
                    |                            |             |                            |             |                            |
                    |    ioppytest Test Tool     |             |     User Interface         |             |     User Interface         |
                    |(CoAP, 6LoWPAN, OneM2M, etc)|             |         (user 1)           |             |         (user 2)           |
                    |                            |             |                            |             |                            |
                    |                            |             |                            |             |                            |
                    +----------------------------+             +----------------------------+             +----------------------------+

                             ^    +                                     ^    +                                     ^    +
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_x.tun.packet |    | toAgent.agent_y.tun.packet          |    |  ui.user1.step_verify.reply         |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_y.tun.packet |    | toAgent.agent_x.tun.packet          |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    | ui.user1.step_verify.request        |    |                                     |    |
                             +    v                                     +    v                                     +    v

        +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                         AMQP Event Bus
        <-------------------------------------------------------------------------------------------------------------------------------------------------+

                                                       +     ^                                        +     ^
                                                       |     | toAgent.agent_x.tun.packetet           |     |  fromAgent.agent_y.tun.packet
                              data.tun.toAgent.agent_x |     |                                        |     |
                                                       |     |              toAgent.agent_y.tun.packet|     |
                                                       v     |                                        v     |
                                PC        +------------+-----+-------------+              +-----------+-----+--------------+
                                user 1    |                                |              |                                |
                                          |      Agent (agent_x)           |              |      Agent (agent_y)           |
                                          |        (tun mode)              |              |        (tun mode)              |
                                          |                                |              |                                |
                                          |                                |              |                                |
                                          | +-----+tun interface+-------+  |              | +-----+tun interface+-------+  |
                                          |                                |              |                                |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          | |         IPv6+based         | |              | |         IPv6+based         | |
                                          | |        communicating       | |              | |        communicating       | |
                                          | |      piece of software     | |              | |      piece of software     | |
                                          | |      (e.g. coap client)    | |              | |      (e.g. coap server)    | |
                                          | |                            | |              | |                            | |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          |                                |              |                                |
                                          +--------------------------------+              +--------------------------------+
87
88
```

Federico Sismondi's avatar
Federico Sismondi committed
89
90
91
92
93
94
95
Event Bus API:
--------------

All the calls between the components are documented here:

[interop tests API doc](http://doc.f-interop.eu/interop/)

96

Federico Sismondi's avatar
Federico Sismondi committed
97
98
Running a test suite:
---------------------
Federico Sismondi's avatar
Federico Sismondi committed
99

Federico Sismondi's avatar
Federico Sismondi committed
100
user needs :
Federico Sismondi's avatar
Federico Sismondi committed
101

Federico Sismondi's avatar
Federico Sismondi committed
102
- an implementation under test (IUT) of a standard supported/protocol
103
by ioppytest framework e.g. a coap client implementation
104
-  run
Federico Sismondi's avatar
Federico Sismondi committed
105
[the agent component](http://doc.f-interop.eu/interop/#agent)
106
107
108
which plays the role of a vpn-client , and which will route all the
packets sent from the IUT (on a certain ipv6 network) to the
backend -which is later on routed to second IUT (and viceversa)-.
Federico Sismondi's avatar
Federico Sismondi committed
109
110
111
- a user interface to help coordinating the tests
(either GUI or CLI component)

112
For simplifying the access to CLI, agent and other components, ioppytest
113
includes a Makefile, with it you can use `make <cmd>`,
114
115
for more information execute `make help`

116
### make commands
Federico Sismondi's avatar
Federico Sismondi committed
117

118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
```
➜  ioppytest git:(master) ✗ make help
help:                          Help dialog.
build-all-coap-images:         Build all testing tool in docker images, and other docker image resources too
build-tools:                   builds all testing tool docker images (only testing tool)
build-automated-iuts:          Build all automated-iut docker images
build-all:                     Build all testing tool in docker images, and other docker image resources too
clean:                         clean data directory
sniff-bus:                     Listen and echo all messages in the event bus
run-cli:                       Run interactive shell
run-6lowpan-testing-tool:      Run 6LoWPAN testing tool in docker container
run-coap-testing-tool:         Run CoAP testing tool in docker container
run-lwm2m-testing-tool:        Run lwm2m testing tool in docker container
run-onem2m-testing-tool:       Run oneM2M testing tool in docker container
run-comi-testing-tool:         Run CoMI testing tool in docker container
stop-all:                      stops testing tools and IUTs running as docker containers
validate-test-description-syntax: validate (yaml) test description file syntax
run-tests:                     runs all unittests
get-logs:                      echoes logs from the running containers
install-python-dependencies:   installs all py2 and py3 pip dependencies
```
Federico Sismondi's avatar
Federico Sismondi committed
139

140
# Test session setups:
Federico Sismondi's avatar
Federico Sismondi committed
141

142
The supported setups are:
Federico Sismondi's avatar
Federico Sismondi committed
143

Federico Sismondi's avatar
Federico Sismondi committed
144
- user controls one IUT and wants to run tests against one of the
Federico Sismondi's avatar
Federico Sismondi committed
145
automated-IUTs the framework supports
Federico Sismondi's avatar
Federico Sismondi committed
146
- user controls one IUT and is in direct contact with a second user
Federico Sismondi's avatar
Federico Sismondi committed
147
controlling a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
148
- user controls both implementations (IUTs) taking part in the interop
Federico Sismondi's avatar
Federico Sismondi committed
149
150
session

151
# (opt 1) Running a test suite using F-Interop platform
152
153
154

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

155
156
157
Recommended option (more user friendly).

# (opt 2) Running a test suite standalone
Federico Sismondi's avatar
Federico Sismondi committed
158

159
160
161
for this, you will use ioppytest_cli  as CLI for
interfacing with testing tool (comms go over AMQP event bus).

162
Recommended option only for testing tool contributors.
163

164
## (opt 2.1) Set up up the message broker
165
166
167

The interop testing tool use RabbitMQ (RMQ) message broker for sending
messages between its components, and the remote ones (like the agent).
Federico Sismondi's avatar
Federico Sismondi committed
168
169

RMQ broker is a component which is **external** to the testing tool
170
and which establish the session and infrastructure so components can
Federico Sismondi's avatar
Federico Sismondi committed
171
communicate with each other during the test session.
Federico Sismondi's avatar
Federico Sismondi committed
172

173
174
175
176
177
178
If using [go.f-interop.eu](go.f-interop.eu) then this is automatically
set-up for you.

When running a standalone setup the user first needs to have a RMQ
broker running..

179
180
181
182
183
184
185
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
186
- Request a remote vhost and credentials (user,pass) to
187
federico<dot>sismondi<at>inria<dot>fr (recommended)
Federico Sismondi's avatar
Federico Sismondi committed
188
189
190

don't hesitate to contact me, this is a very simple procedure and it's
free :D
191

192
## (opt 2.2) Export AMQP environment variables
193

194
195
196
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
Federico Sismondi's avatar
Federico Sismondi committed
197

Federico Sismondi's avatar
Federico Sismondi committed
198
199
200
201
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
202

203
## (opt 2.3) Download the source code (see `Clonning the project` for more info)
Federico Sismondi's avatar
Federico Sismondi committed
204
```
205
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
206
207
cd ioppytest
```
208
209


210
## (opt 2.4) Build the testing tools
211

Federico Sismondi's avatar
Federico Sismondi committed
212
213
(docker, py2 and py3 needs to be installed in the machine)
```
214
make build-all
Federico Sismondi's avatar
Federico Sismondi committed
215
```
216

217
## (opt 2.5) Run testing tool (CoAP testing tool example)
Federico Sismondi's avatar
Federico Sismondi committed
218
219
220
```
make run-coap-testing-tool
```
221

222
## (opt 2.6) Connect to the interop session using the CLI
Federico Sismondi's avatar
Federico Sismondi committed
223
224
225
```
make run-cli
```
226

227
## (opt 2.7) Connect the agent to the backend
228

Federico Sismondi's avatar
Federico Sismondi committed
229
if user's IUT is a CoAP client:
230

Federico Sismondi's avatar
Federico Sismondi committed
231
232
233
```
make run-agent-coap-client
```
234

Federico Sismondi's avatar
Federico Sismondi committed
235
if user's IUT is a CoAP server:
236

237
```
Federico Sismondi's avatar
Federico Sismondi committed
238
make run-agent-coap-server
239
240
```

241
## (opt 2.8) Running a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
242

243
### (opt 2.8.1) User to user session, second user with his/her own IUT
Federico Sismondi's avatar
Federico Sismondi committed
244
245
246
247
248
249
250
251

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

252
### (opt 2.8.2) Single user session, against an automated-IUT
Federico Sismondi's avatar
Federico Sismondi committed
253
254
255

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
256

257
```
Federico Sismondi's avatar
Federico Sismondi committed
258
make run-coap-server
259
260
```

Federico Sismondi's avatar
Federico Sismondi committed
261
262
or for a coap client automated implementation:

263
```
Federico Sismondi's avatar
Federico Sismondi committed
264
make run-coap-client
265
266
```

Federico Sismondi's avatar
Federico Sismondi committed
267

268
## (opt 2.9) Running the interop session
Federico Sismondi's avatar
Federico Sismondi committed
269
270
271
272
273
274
275
276
277
278
279
280

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
281
```
282
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
283
cd ioppytest
284
285
```

Federico Sismondi's avatar
Federico Sismondi committed
286
287
288
289
290
291
292
293
294
295
## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
296

297
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
298

299
```
Federico Sismondi's avatar
Federico Sismondi committed
300
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
301
302
303
304
305
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
306
307
```

308
309

alternatively, if you are curious and you want to know
310
311
what's under the hood, you can see which processes are being run -in
the docker container- by the testing tool:
Federico Sismondi's avatar
Federico Sismondi committed
312

313
```
Federico Sismondi's avatar
Federico Sismondi committed
314
315
316
317
318
319
320
321
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

322
323
root@bab3b2220510:/ioppytest# supervisord -c supervisor.conf
root@bab3b2220510:/ioppytest# supervisorctl -c supervisor.conf
324
325
326
327
328
329
330
331
agent                            RUNNING   pid 28, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
332
333
or you can also run directly the processes without docker:
(supervisord needed)
334

Federico Sismondi's avatar
Federico Sismondi committed
335
336
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
337
sudo -E supervisorctl -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
338
```
339

Federico Sismondi's avatar
Federico Sismondi committed
340
341
342
you can use for example envs/coap_testing_tool/supervisor.conf.ini
for using the coap_testing_tool

Federico Sismondi's avatar
Federico Sismondi committed
343
note: use -E when launching supervisor process, it preserves the
344
env vars (like an exported AMQP_URL)
345
346


Federico Sismondi's avatar
Federico Sismondi committed
347
348
FAQ
---
349

Federico Sismondi's avatar
Federico Sismondi committed
350
- How can I install docker on my machine?
351

Federico Sismondi's avatar
Federico Sismondi committed
352
    For this just follow this instructions: https://docs.docker.com/engine/installation/
353

Federico Sismondi's avatar
Federico Sismondi committed
354

Federico Sismondi's avatar
Federico Sismondi committed
355
356
357
358
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
Federico Sismondi's avatar
Federico Sismondi committed
359

Federico Sismondi's avatar
Federico Sismondi committed
360
361
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
Federico Sismondi's avatar
Federico Sismondi committed
362

363
    please contact federico<dot>sismondi<at>inria<dot>fr
364
365


Federico Sismondi's avatar
Federico Sismondi committed
366
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
367

Federico Sismondi's avatar
Federico Sismondi committed
368
    try using ```--no-cache``` for the docker build
369

Federico Sismondi's avatar
Federico Sismondi committed
370
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a