README.md 16.1 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1 2
ioppytest framework:
--------------------
3

4
ioppytest is a framework for building interoperability testing tools.
Federico Sismondi's avatar
Federico Sismondi committed
5 6 7 8 9

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
Handles requirements coming from:

- interop testing best practises for network protocols
- distributed execution of the tests (IUTs may be in remote locations)
- test coordination between users (user-assisted test)
- driving automated IUTs, in a generic, program-language-agnostic way
- tunneling mechanism for enabling remote interop test over
distant locations and bypass middle-box.

Some of the features include:
tunneling, sniffing, test coordination, dissection, traces analysis,
test spec online server, etc. Each of these are independent python
modules, they communicate using json messages over AMQP.

For more about this check out the AMQP API defined in the `messages`
package which is installed with
(ioppytest-utils)*[https://pypi.org/project/ioppytest-utils/]:


Test suites
-----------

currently ioppytest includes interop tests for:

- coap
- 6lowpan
- onem2m
- lwm2m
- wot
39

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
see http://doc.f-interop.eu/testsuites/

ioppytest is parametrized for running the different test suites using
the files in env/<test_suite_folder> and ioppytest/test_descriptions

Either if you are a testing tool developers or test suite users,
you can opt to disable some features. You can do this just by modifying
supervisor.conf.ini in env/<test_suite_folder> directory.

How can I use it?
-----------------

You can either go to (F-Interop Platform)*[https://go.f-interop.eu]
which builds and deploys the tool automatically for you, this provides
also a nice looking web-based GUI (recommended).

Run in a stanalone less user friendly way. Some docker images builds,
and setting up of a RMQ server is needed. User will then use the CLI
for interfacing with the testin tool.

for more info about the standalone deployment, continue reading..
61

Federico Sismondi's avatar
Federico Sismondi committed
62 63
Implemented test suites in the ioppytest framework:
---------------------------------------------------
64

Federico Sismondi's avatar
Federico Sismondi committed
65 66
The test suites implemented which are currently supported by the
framework are:
67

Federico Sismondi's avatar
Federico Sismondi committed
68 69 70
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
71 72
- LwM2M Test Suite (between two users' IUT) (WIP)
- oneM2M Test Suite (between two users' IUT) (WIP)
73

Federico Sismondi's avatar
Federico Sismondi committed
74 75
Test setup:
-----------
76

Federico Sismondi's avatar
Federico Sismondi committed
77 78 79
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
80

Federico Sismondi's avatar
Federico Sismondi committed
81 82
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
83 84

```
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130

                                                                        +----------------------------+
                                                                        |                            |
                                                                        |    ioppytest Test Tool     |
                                                                        |(CoAP, 6LoWPAN, OneM2M, etc)|
                                                                        |                            |
                                                                        |                            |
                                                                        +----------------------------+

                                                                                    ^    +
                                                                                    |    |
                                                                                    |    |
                                                                                    |    |
                                                            packet.fromAgent.agent_x|    |  packet.toAgent_agent_y
                                                                                    |    |
                                                            packet.fromAgent.agent_x|    |  packet.toAgent_agent_y
                                                                                    |    |
                                                            ui.user_1.reply         |    |  ui.user_1.request
                                                                                    +    v

                +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                                 AMQP Event Bus
                <-------------------------------------------------------------------------------------------------------------------------------------------------+
                      |   ^                              ^  |                                           |   ^                              ^  |
                      |   |     packet.fromAgent.agent_x |  | packet.toAgent_agent_x                    |   |     packet.fromAgent.agent_y |  | packet.toAgent_agent_y
             ui.user_1|   |                              |  |                                           |   |                              |  |
              .request|   | ui.user_1.reply              |  |                          ui.user_2.request|   | ui.user_2.reply              |  |
                      |   |                              |  |                                           |   |                              |  |
                      |   |                              |  v                                           |   |                              |  v
              +-------v-------------+      +--------------------------------+                   +-------v-------------+      +--------------------------------+
              |                     |      | +---------------------------+  |                   |                     |      | +---------------------------+  |
              |   User Interface    |      | |      Agent (agent_x)      |  |                   |   User Interface    |      | |      Agent (agent_y)      |  |
              |       (user 1)      |      | |        (tun mode)         |  |                   |       (user 2)      |      | |        (tun mode)         |  |
              |                     |      | +---------------------------+  |                   |                     |      | +---------------------------+  |
              +---------------------+      |                                |                   +---------------------+      |                                |
                                           | +-----+tun interface+-------+  |                                                | +-----+tun interface+-------+  |
                                           |                                |                                                |                                |
                                           | +----------------------------+ |                                                | +----------------------------+ |
                                           | | Implementation under test  | |                                                | | Implementation under test  | |
                                           | |          using IP          | |                                                | |          using IP          | |
                                           | |      (e.g. coapoclient)    | |                                                | |      (e.g. coaposerver)    | |
                                           | +----------------------------+ |                                                | +----------------------------+ |
                                           +--------------------------------+                                                +--------------------------------+



131 132
```

Federico Sismondi's avatar
Federico Sismondi committed
133 134 135 136 137 138 139
Event Bus API:
--------------

All the calls between the components are documented here:

[interop tests API doc](http://doc.f-interop.eu/interop/)

140

Federico Sismondi's avatar
Federico Sismondi committed
141 142
Running a test suite:
---------------------
143

Federico Sismondi's avatar
Federico Sismondi committed
144
user needs :
145

Federico Sismondi's avatar
Federico Sismondi committed
146
- an implementation under test (IUT) of a standard supported/protocol
147
by ioppytest framework e.g. a coap client implementation
148
-  run
Federico Sismondi's avatar
Federico Sismondi committed
149
[the agent component](http://doc.f-interop.eu/interop/#agent)
150 151 152
which plays the role of a vpn-client , and which will route all the
packets sent from the IUT (on a certain ipv6 network) to the
backend -which is later on routed to second IUT (and viceversa)-.
Federico Sismondi's avatar
Federico Sismondi committed
153 154 155
- a user interface to help coordinating the tests
(either GUI or CLI component)

156
For simplifying the access to CLI, agent and other components, ioppytest
157
includes a Makefile, with it you can use `make <cmd>`,
158 159
for more information execute `make help`

160
### make commands
Federico Sismondi's avatar
Federico Sismondi committed
161

162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182
```
➜  ioppytest git:(master) ✗ make help
help:                          Help dialog.
build-all-coap-images:         Build all testing tool in docker images, and other docker image resources too
build-tools:                   builds all testing tool docker images (only testing tool)
build-automated-iuts:          Build all automated-iut docker images
build-all:                     Build all testing tool in docker images, and other docker image resources too
clean:                         clean data directory
sniff-bus:                     Listen and echo all messages in the event bus
run-cli:                       Run interactive shell
run-6lowpan-testing-tool:      Run 6LoWPAN testing tool in docker container
run-coap-testing-tool:         Run CoAP testing tool in docker container
run-lwm2m-testing-tool:        Run lwm2m testing tool in docker container
run-onem2m-testing-tool:       Run oneM2M testing tool in docker container
run-comi-testing-tool:         Run CoMI testing tool in docker container
stop-all:                      stops testing tools and IUTs running as docker containers
validate-test-description-syntax: validate (yaml) test description file syntax
run-tests:                     runs all unittests
get-logs:                      echoes logs from the running containers
install-python-dependencies:   installs all py2 and py3 pip dependencies
```
Federico Sismondi's avatar
Federico Sismondi committed
183

184
# Test session setups:
Federico Sismondi's avatar
Federico Sismondi committed
185

186
The supported setups are:
Federico Sismondi's avatar
Federico Sismondi committed
187

188
- user drives one IUT and wants to run tests against one of the
Federico Sismondi's avatar
Federico Sismondi committed
189
automated-IUTs the framework supports
190
- user drives both implementations (IUTs) taking part in the interop
Federico Sismondi's avatar
Federico Sismondi committed
191
session
192 193
- user1 drives an IUT, user2 drives an IUT, users are either both
in-situ, or remotely located.
Federico Sismondi's avatar
Federico Sismondi committed
194

195
# (opt 1) Running a test suite using F-Interop platform
196 197 198

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

199 200 201
Recommended option (more user friendly).

# (opt 2) Running a test suite standalone
Federico Sismondi's avatar
Federico Sismondi committed
202

203 204 205
for this, you will use ioppytest_cli  as CLI for
interfacing with testing tool (comms go over AMQP event bus).

206
Recommended option only for testing tool contributors.
207

208
## (opt 2.1) Set up up the message broker
209 210 211

The interop testing tool use RabbitMQ (RMQ) message broker for sending
messages between its components, and the remote ones (like the agent).
Federico Sismondi's avatar
Federico Sismondi committed
212 213

RMQ broker is a component which is **external** to the testing tool
214
and which establish the session and infrastructure so components can
Federico Sismondi's avatar
Federico Sismondi committed
215
communicate with each other during the test session.
216

217 218 219 220 221 222
If using [go.f-interop.eu](go.f-interop.eu) then this is automatically
set-up for you.

When running a standalone setup the user first needs to have a RMQ
broker running..

223 224 225 226 227 228 229
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
230
- Request a remote vhost and credentials (user,pass) to
231
federico<dot>sismondi<at>inria<dot>fr (recommended)
Federico Sismondi's avatar
Federico Sismondi committed
232 233 234

don't hesitate to contact me, this is a very simple procedure and it's
free :D
235

236
## (opt 2.2) Export AMQP environment variables
237

238 239 240
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
241

Federico Sismondi's avatar
Federico Sismondi committed
242 243 244 245
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
246

247
## (opt 2.3) Download the source code (see `Clonning the project` for more info)
Federico Sismondi's avatar
Federico Sismondi committed
248
```
249
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
250 251
cd ioppytest
```
252 253


254
## (opt 2.4) Build the testing tools
255

Federico Sismondi's avatar
Federico Sismondi committed
256 257
(docker, py2 and py3 needs to be installed in the machine)
```
258
make build-all
Federico Sismondi's avatar
Federico Sismondi committed
259
```
260

261
## (opt 2.5) Run testing tool (CoAP testing tool example)
Federico Sismondi's avatar
Federico Sismondi committed
262 263 264
```
make run-coap-testing-tool
```
265

266
## (opt 2.6) Connect to the interop session using the CLI
Federico Sismondi's avatar
Federico Sismondi committed
267 268 269
```
make run-cli
```
270

271
## (opt 2.7) Connect the agent to the backend
272

Federico Sismondi's avatar
Federico Sismondi committed
273
if user's IUT is a CoAP client:
274

Federico Sismondi's avatar
Federico Sismondi committed
275 276 277
```
make run-agent-coap-client
```
278

Federico Sismondi's avatar
Federico Sismondi committed
279
if user's IUT is a CoAP server:
280

281
```
Federico Sismondi's avatar
Federico Sismondi committed
282
make run-agent-coap-server
283 284
```

285
## (opt 2.8) Running a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
286

287
### (opt 2.8.1) User to user session, second user with his/her own IUT
Federico Sismondi's avatar
Federico Sismondi committed
288 289 290 291 292 293 294 295

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

296
### (opt 2.8.2) Single user session, against an automated-IUT
Federico Sismondi's avatar
Federico Sismondi committed
297 298 299

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
300

301
```
Federico Sismondi's avatar
Federico Sismondi committed
302
make run-coap-server
303 304
```

Federico Sismondi's avatar
Federico Sismondi committed
305 306
or for a coap client automated implementation:

307
```
Federico Sismondi's avatar
Federico Sismondi committed
308
make run-coap-client
309 310
```

Federico Sismondi's avatar
Federico Sismondi committed
311

312
## (opt 2.9) Running the interop session
Federico Sismondi's avatar
Federico Sismondi committed
313 314 315 316 317 318 319 320 321 322 323 324

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
325
```
326
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
327
cd ioppytest
328 329
```

Federico Sismondi's avatar
Federico Sismondi committed
330 331 332 333 334 335 336 337 338 339
## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
340

341
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
342

343
```
Federico Sismondi's avatar
Federico Sismondi committed
344
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
345 346 347 348 349
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
350 351
```

352 353

alternatively, if you are curious and you want to know
354 355
what's under the hood, you can see which processes are being run -in
the docker container- by the testing tool:
Federico Sismondi's avatar
Federico Sismondi committed
356

357
```
Federico Sismondi's avatar
Federico Sismondi committed
358 359 360 361 362 363 364 365
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

366 367
root@bab3b2220510:/ioppytest# supervisord -c supervisor.conf
root@bab3b2220510:/ioppytest# supervisorctl -c supervisor.conf
368 369 370 371 372 373 374 375
agent                            RUNNING   pid 28, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
376 377
or you can also run directly the processes without docker:
(supervisord needed)
378

Federico Sismondi's avatar
Federico Sismondi committed
379 380
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
381
sudo -E supervisorctl -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
382
```
383

Federico Sismondi's avatar
Federico Sismondi committed
384 385 386
you can use for example envs/coap_testing_tool/supervisor.conf.ini
for using the coap_testing_tool

Federico Sismondi's avatar
Federico Sismondi committed
387
note: use -E when launching supervisor process, it preserves the
388
env vars (like an exported AMQP_URL)
389 390


Federico Sismondi's avatar
Federico Sismondi committed
391 392
FAQ
---
393

Federico Sismondi's avatar
Federico Sismondi committed
394
- How can I install docker on my machine?
395

Federico Sismondi's avatar
Federico Sismondi committed
396
    For this just follow this instructions: https://docs.docker.com/engine/installation/
397

398

Federico Sismondi's avatar
Federico Sismondi committed
399 400 401 402
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
403

Federico Sismondi's avatar
Federico Sismondi committed
404 405
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
406

407
    please contact federico<dot>sismondi<at>inria<dot>fr
408 409


Federico Sismondi's avatar
Federico Sismondi committed
410
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
411

Federico Sismondi's avatar
Federico Sismondi committed
412
    try using ```--no-cache``` for the docker build
413

Federico Sismondi's avatar
Federico Sismondi committed
414
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a