README.md 14.6 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1 2
ioppytest framework:
--------------------
3

Federico Sismondi's avatar
Federico Sismondi committed
4
ioppytest is a framework for running interoperability tests.
5

6 7
This initial version tackles technical interoperability testing (CoAP,
 LwM2M, 6LoWPAN and OneM2M interop tests).
Federico Sismondi's avatar
Federico Sismondi committed
8 9 10 11 12 13

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

This can be run as standalone software and also integrated to f-interop
14
platform (go.f-interop.eu)
15 16


Federico Sismondi's avatar
Federico Sismondi committed
17 18
Implemented test suites in the ioppytest framework:
---------------------------------------------------
19

Federico Sismondi's avatar
Federico Sismondi committed
20 21
The test suites implemented which are currently supported by the
framework are:
22

Federico Sismondi's avatar
Federico Sismondi committed
23 24 25
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
26 27
- LwM2M Test Suite (between two users' IUT) (WIP)
- oneM2M Test Suite (between two users' IUT) (WIP)
28

29 30


Federico Sismondi's avatar
Federico Sismondi committed
31 32
Test setup:
-----------
33

Federico Sismondi's avatar
Federico Sismondi committed
34 35 36
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
37

Federico Sismondi's avatar
Federico Sismondi committed
38 39
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
40 41

```
Federico Sismondi's avatar
Federico Sismondi committed
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
                    +----------------------------+             +----------------------------+             +----------------------------+
                    |                            |             |                            |             |                            |
                    |    ioppytest Test Tool     |             |     User Interface         |             |     User Interface         |
                    |(CoAP, 6LoWPAN, OneM2M, etc)|             |         (user 1)           |             |         (user 2)           |
                    |                            |             |                            |             |                            |
                    |                            |             |                            |             |                            |
                    +----------------------------+             +----------------------------+             +----------------------------+

                             ^    +                                     ^    +                                     ^    +
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_x.tun.packet |    | toAgent.agent_y.tun.packet          |    |  ui.user1.step_verify.reply         |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_y.tun.packet |    | toAgent.agent_x.tun.packet          |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    | ui.user1.step_verify.request        |    |                                     |    |
                             +    v                                     +    v                                     +    v

        +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                         AMQP Event Bus
        <-------------------------------------------------------------------------------------------------------------------------------------------------+

                                                       +     ^                                        +     ^
                                                       |     | toAgent.agent_x.tun.packetet           |     |  fromAgent.agent_y.tun.packet
                              data.tun.toAgent.agent_x |     |                                        |     |
                                                       |     |              toAgent.agent_y.tun.packet|     |
                                                       v     |                                        v     |
                                PC        +------------+-----+-------------+              +-----------+-----+--------------+
                                user 1    |                                |              |                                |
                                          |      Agent (agent_x)           |              |      Agent (agent_y)           |
                                          |        (tun mode)              |              |        (tun mode)              |
                                          |                                |              |                                |
                                          |                                |              |                                |
                                          | +-----+tun interface+-------+  |              | +-----+tun interface+-------+  |
                                          |                                |              |                                |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          | |         IPv6+based         | |              | |         IPv6+based         | |
                                          | |        communicating       | |              | |        communicating       | |
                                          | |      piece of software     | |              | |      piece of software     | |
                                          | |      (e.g. coap client)    | |              | |      (e.g. coap server)    | |
                                          | |                            | |              | |                            | |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          |                                |              |                                |
                                          +--------------------------------+              +--------------------------------+
87 88
```

Federico Sismondi's avatar
Federico Sismondi committed
89 90 91 92 93 94 95
Event Bus API:
--------------

All the calls between the components are documented here:

[interop tests API doc](http://doc.f-interop.eu/interop/)

96

Federico Sismondi's avatar
Federico Sismondi committed
97 98
Running a test suite:
---------------------
99

Federico Sismondi's avatar
Federico Sismondi committed
100
user needs :
101

Federico Sismondi's avatar
Federico Sismondi committed
102
- an implementation under test (IUT) of a standard supported/protocol
103
by ioppytest framework e.g. a coap client implementation
104
-  run
Federico Sismondi's avatar
Federico Sismondi committed
105
[the agent component](http://doc.f-interop.eu/interop/#agent)
106 107 108
which plays the role of a vpn-client , and which will route all the
packets sent from the IUT (on a certain ipv6 network) to the
backend -which is later on routed to second IUT (and viceversa)-.
Federico Sismondi's avatar
Federico Sismondi committed
109 110 111
- a user interface to help coordinating the tests
(either GUI or CLI component)

112
For simplifying the access to CLI, agent and other components, ioppytest
113
includes a Makefile, with it you can use `make <cmd>`,
114 115
for more information execute `make help`

116
### make commands
Federico Sismondi's avatar
Federico Sismondi committed
117

118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138
```
➜  ioppytest git:(master) ✗ make help
help:                          Help dialog.
build-all-coap-images:         Build all testing tool in docker images, and other docker image resources too
build-tools:                   builds all testing tool docker images (only testing tool)
build-automated-iuts:          Build all automated-iut docker images
build-all:                     Build all testing tool in docker images, and other docker image resources too
clean:                         clean data directory
sniff-bus:                     Listen and echo all messages in the event bus
run-cli:                       Run interactive shell
run-6lowpan-testing-tool:      Run 6LoWPAN testing tool in docker container
run-coap-testing-tool:         Run CoAP testing tool in docker container
run-lwm2m-testing-tool:        Run lwm2m testing tool in docker container
run-onem2m-testing-tool:       Run oneM2M testing tool in docker container
run-comi-testing-tool:         Run CoMI testing tool in docker container
stop-all:                      stops testing tools and IUTs running as docker containers
validate-test-description-syntax: validate (yaml) test description file syntax
run-tests:                     runs all unittests
get-logs:                      echoes logs from the running containers
install-python-dependencies:   installs all py2 and py3 pip dependencies
```
Federico Sismondi's avatar
Federico Sismondi committed
139

140
# Test session setups:
Federico Sismondi's avatar
Federico Sismondi committed
141

142
The supported setups are:
Federico Sismondi's avatar
Federico Sismondi committed
143

Federico Sismondi's avatar
Federico Sismondi committed
144
- user controls one IUT and wants to run tests against one of the
Federico Sismondi's avatar
Federico Sismondi committed
145
automated-IUTs the framework supports
Federico Sismondi's avatar
Federico Sismondi committed
146
- user controls one IUT and is in direct contact with a second user
Federico Sismondi's avatar
Federico Sismondi committed
147
controlling a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
148
- user controls both implementations (IUTs) taking part in the interop
Federico Sismondi's avatar
Federico Sismondi committed
149 150
session

151
# (opt 1) Running a test suite using F-Interop platform
152 153 154

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

155 156 157
Recommended option (more user friendly).

# (opt 2) Running a test suite standalone
Federico Sismondi's avatar
Federico Sismondi committed
158

159 160 161
for this, you will use ioppytest_cli  as CLI for
interfacing with testing tool (comms go over AMQP event bus).

162
Recommended option only for testing tool contributors.
163

164
## (opt 2.1) Set up up the message broker
165 166 167

The interop testing tool use RabbitMQ (RMQ) message broker for sending
messages between its components, and the remote ones (like the agent).
Federico Sismondi's avatar
Federico Sismondi committed
168 169

RMQ broker is a component which is **external** to the testing tool
170
and which establish the session and infrastructure so components can
Federico Sismondi's avatar
Federico Sismondi committed
171
communicate with each other during the test session.
172

173 174 175 176 177 178
If using [go.f-interop.eu](go.f-interop.eu) then this is automatically
set-up for you.

When running a standalone setup the user first needs to have a RMQ
broker running..

179 180 181 182 183 184 185
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
186
- Request a remote vhost and credentials (user,pass) to
187
federico<dot>sismondi<at>inria<dot>fr (recommended)
Federico Sismondi's avatar
Federico Sismondi committed
188 189 190

don't hesitate to contact me, this is a very simple procedure and it's
free :D
191

192
## (opt 2.2) Export AMQP environment variables
193

194 195 196
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
197

Federico Sismondi's avatar
Federico Sismondi committed
198 199 200 201
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
202

203
## (opt 2.3) Download the source code (see `Clonning the project` for more info)
Federico Sismondi's avatar
Federico Sismondi committed
204
```
205
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
206 207
cd ioppytest
```
208 209


210
## (opt 2.4) Build the testing tools
211

Federico Sismondi's avatar
Federico Sismondi committed
212 213
(docker, py2 and py3 needs to be installed in the machine)
```
214
make build-all
Federico Sismondi's avatar
Federico Sismondi committed
215
```
216

217
## (opt 2.5) Run testing tool (CoAP testing tool example)
Federico Sismondi's avatar
Federico Sismondi committed
218 219 220
```
make run-coap-testing-tool
```
221

222
## (opt 2.6) Connect to the interop session using the CLI
Federico Sismondi's avatar
Federico Sismondi committed
223 224 225
```
make run-cli
```
226

227
## (opt 2.7) Connect the agent to the backend
228

Federico Sismondi's avatar
Federico Sismondi committed
229
if user's IUT is a CoAP client:
230

Federico Sismondi's avatar
Federico Sismondi committed
231 232 233
```
make run-agent-coap-client
```
234

Federico Sismondi's avatar
Federico Sismondi committed
235
if user's IUT is a CoAP server:
236

237
```
Federico Sismondi's avatar
Federico Sismondi committed
238
make run-agent-coap-server
239 240
```

241
## (opt 2.8) Running a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
242

243
### (opt 2.8.1) User to user session, second user with his/her own IUT
Federico Sismondi's avatar
Federico Sismondi committed
244 245 246 247 248 249 250 251

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

252
### (opt 2.8.2) Single user session, against an automated-IUT
Federico Sismondi's avatar
Federico Sismondi committed
253 254 255

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
256

257
```
Federico Sismondi's avatar
Federico Sismondi committed
258
make run-coap-server
259 260
```

Federico Sismondi's avatar
Federico Sismondi committed
261 262
or for a coap client automated implementation:

263
```
Federico Sismondi's avatar
Federico Sismondi committed
264
make run-coap-client
265 266
```

Federico Sismondi's avatar
Federico Sismondi committed
267

268
## (opt 2.9) Running the interop session
Federico Sismondi's avatar
Federico Sismondi committed
269 270 271 272 273 274 275 276 277 278 279 280

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
281
```
282
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
283
cd ioppytest
284 285
```

Federico Sismondi's avatar
Federico Sismondi committed
286 287 288 289 290 291 292 293 294 295
## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
296

297
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
298

299
```
Federico Sismondi's avatar
Federico Sismondi committed
300
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
301 302 303 304 305
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
306 307
```

308 309

alternatively, if you are curious and you want to know
310 311
what's under the hood, you can see which processes are being run -in
the docker container- by the testing tool:
Federico Sismondi's avatar
Federico Sismondi committed
312

313
```
Federico Sismondi's avatar
Federico Sismondi committed
314 315 316 317 318 319 320 321
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

322 323
root@bab3b2220510:/ioppytest# supervisord -c supervisor.conf
root@bab3b2220510:/ioppytest# supervisorctl -c supervisor.conf
324 325 326 327 328 329 330 331
agent                            RUNNING   pid 28, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
332 333
or you can also run directly the processes without docker:
(supervisord needed)
334

Federico Sismondi's avatar
Federico Sismondi committed
335 336
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
337
sudo -E supervisorctl -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
338
```
339

Federico Sismondi's avatar
Federico Sismondi committed
340 341 342
you can use for example envs/coap_testing_tool/supervisor.conf.ini
for using the coap_testing_tool

Federico Sismondi's avatar
Federico Sismondi committed
343
note: use -E when launching supervisor process, it preserves the
344
env vars (like an exported AMQP_URL)
345 346


Federico Sismondi's avatar
Federico Sismondi committed
347 348
FAQ
---
349

Federico Sismondi's avatar
Federico Sismondi committed
350
- How can I install docker on my machine?
351

Federico Sismondi's avatar
Federico Sismondi committed
352
    For this just follow this instructions: https://docs.docker.com/engine/installation/
353

354

Federico Sismondi's avatar
Federico Sismondi committed
355 356 357 358
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
359

Federico Sismondi's avatar
Federico Sismondi committed
360 361
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
362

363
    please contact federico<dot>sismondi<at>inria<dot>fr
364 365


Federico Sismondi's avatar
Federico Sismondi committed
366
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
367

Federico Sismondi's avatar
Federico Sismondi committed
368
    try using ```--no-cache``` for the docker build
369

Federico Sismondi's avatar
Federico Sismondi committed
370
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a