README.md 14.4 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1 2
ioppytest framework:
--------------------
3

Federico Sismondi's avatar
Federico Sismondi committed
4
ioppytest is a framework for running interoperability tests.
5

6 7
This initial version tackles technical interoperability testing (CoAP,
 LwM2M, 6LoWPAN and OneM2M interop tests).
Federico Sismondi's avatar
Federico Sismondi committed
8 9 10 11 12 13

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

This can be run as standalone software and also integrated to f-interop
14
platform (go.f-interop.eu)
15 16


Federico Sismondi's avatar
Federico Sismondi committed
17 18
Implemented test suites in the ioppytest framework:
---------------------------------------------------
19

Federico Sismondi's avatar
Federico Sismondi committed
20 21
The test suites implemented which are currently supported by the
framework are:
22

Federico Sismondi's avatar
Federico Sismondi committed
23 24 25
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
26 27
- LwM2M Test Suite (between two users' IUT) (WIP)
- oneM2M Test Suite (between two users' IUT) (WIP)
28

29 30


Federico Sismondi's avatar
Federico Sismondi committed
31 32
Test setup:
-----------
33

Federico Sismondi's avatar
Federico Sismondi committed
34 35 36
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
37

Federico Sismondi's avatar
Federico Sismondi committed
38 39
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
40 41

```
Federico Sismondi's avatar
Federico Sismondi committed
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
                    +----------------------------+             +----------------------------+             +----------------------------+
                    |                            |             |                            |             |                            |
                    |    ioppytest Test Tool     |             |     User Interface         |             |     User Interface         |
                    |(CoAP, 6LoWPAN, OneM2M, etc)|             |         (user 1)           |             |         (user 2)           |
                    |                            |             |                            |             |                            |
                    |                            |             |                            |             |                            |
                    +----------------------------+             +----------------------------+             +----------------------------+

                             ^    +                                     ^    +                                     ^    +
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_x.tun.packet |    | toAgent.agent_y.tun.packet          |    |  ui.user1.step_verify.reply         |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_y.tun.packet |    | toAgent.agent_x.tun.packet          |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    | ui.user1.step_verify.request        |    |                                     |    |
                             +    v                                     +    v                                     +    v

        +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                         AMQP Event Bus
        <-------------------------------------------------------------------------------------------------------------------------------------------------+

                                                       +     ^                                        +     ^
                                                       |     | toAgent.agent_x.tun.packetet           |     |  fromAgent.agent_y.tun.packet
                              data.tun.toAgent.agent_x |     |                                        |     |
                                                       |     |              toAgent.agent_y.tun.packet|     |
                                                       v     |                                        v     |
                                PC        +------------+-----+-------------+              +-----------+-----+--------------+
                                user 1    |                                |              |                                |
                                          |      Agent (agent_x)           |              |      Agent (agent_y)           |
                                          |        (tun mode)              |              |        (tun mode)              |
                                          |                                |              |                                |
                                          |                                |              |                                |
                                          | +-----+tun interface+-------+  |              | +-----+tun interface+-------+  |
                                          |                                |              |                                |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          | |         IPv6+based         | |              | |         IPv6+based         | |
                                          | |        communicating       | |              | |        communicating       | |
                                          | |      piece of software     | |              | |      piece of software     | |
                                          | |      (e.g. coap client)    | |              | |      (e.g. coap server)    | |
                                          | |                            | |              | |                            | |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          |                                |              |                                |
                                          +--------------------------------+              +--------------------------------+
87 88
```

Federico Sismondi's avatar
Federico Sismondi committed
89 90 91 92 93 94 95
Event Bus API:
--------------

All the calls between the components are documented here:

[interop tests API doc](http://doc.f-interop.eu/interop/)

96

Federico Sismondi's avatar
Federico Sismondi committed
97 98
Running a test suite:
---------------------
99

Federico Sismondi's avatar
Federico Sismondi committed
100
user needs :
101

Federico Sismondi's avatar
Federico Sismondi committed
102
- an implementation under test (IUT) of a standard supported/protocol
103
by ioppytest framework e.g. a coap client implementation
104
-  run
Federico Sismondi's avatar
Federico Sismondi committed
105
[the agent component](http://doc.f-interop.eu/interop/#agent)
106 107 108
which plays the role of a vpn-client , and which will route all the
packets sent from the IUT (on a certain ipv6 network) to the
backend -which is later on routed to second IUT (and viceversa)-.
Federico Sismondi's avatar
Federico Sismondi committed
109 110 111
- a user interface to help coordinating the tests
(either GUI or CLI component)

112
For simplifying the access to CLI, agent and other components, ioppytest
113
includes a Makefile, with it you can use `make <cmd>`,
114 115
for more information execute `make help`

116
### make commands
Federico Sismondi's avatar
Federico Sismondi committed
117

118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138
```
➜  ioppytest git:(master) ✗ make help
help:                          Help dialog.
build-all-coap-images:         Build all testing tool in docker images, and other docker image resources too
build-tools:                   builds all testing tool docker images (only testing tool)
build-automated-iuts:          Build all automated-iut docker images
build-all:                     Build all testing tool in docker images, and other docker image resources too
clean:                         clean data directory
sniff-bus:                     Listen and echo all messages in the event bus
run-cli:                       Run interactive shell
run-6lowpan-testing-tool:      Run 6LoWPAN testing tool in docker container
run-coap-testing-tool:         Run CoAP testing tool in docker container
run-lwm2m-testing-tool:        Run lwm2m testing tool in docker container
run-onem2m-testing-tool:       Run oneM2M testing tool in docker container
run-comi-testing-tool:         Run CoMI testing tool in docker container
stop-all:                      stops testing tools and IUTs running as docker containers
validate-test-description-syntax: validate (yaml) test description file syntax
run-tests:                     runs all unittests
get-logs:                      echoes logs from the running containers
install-python-dependencies:   installs all py2 and py3 pip dependencies
```
Federico Sismondi's avatar
Federico Sismondi committed
139

140
# Test session setups:
Federico Sismondi's avatar
Federico Sismondi committed
141

142
The supported setups are:
Federico Sismondi's avatar
Federico Sismondi committed
143

Federico Sismondi's avatar
Federico Sismondi committed
144
- user controls one IUT and wants to run tests against one of the
Federico Sismondi's avatar
Federico Sismondi committed
145
automated-IUTs the framework supports
Federico Sismondi's avatar
Federico Sismondi committed
146
- user controls one IUT and is in direct contact with a second user
Federico Sismondi's avatar
Federico Sismondi committed
147
controlling a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
148
- user controls both implementations (IUTs) taking part in the interop
Federico Sismondi's avatar
Federico Sismondi committed
149 150
session

151 152 153 154 155
## Running a test suite using F-Interop platform

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

## Running a test suite standalone
Federico Sismondi's avatar
Federico Sismondi committed
156

157 158 159 160 161 162 163 164
for this, you will use ioppytest_cli  as CLI for
interfacing with testing tool (comms go over AMQP event bus).


### Set up up the message broker

The interop testing tool use RabbitMQ (RMQ) message broker for sending
messages between its components, and the remote ones (like the agent).
Federico Sismondi's avatar
Federico Sismondi committed
165 166

RMQ broker is a component which is **external** to the testing tool
167
and which establish the session and infrastructure so components can
Federico Sismondi's avatar
Federico Sismondi committed
168
communicate with each other during the test session.
169

170 171 172 173 174 175
If using [go.f-interop.eu](go.f-interop.eu) then this is automatically
set-up for you.

When running a standalone setup the user first needs to have a RMQ
broker running..

176 177 178 179 180 181 182
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
183
- Request a remote vhost and credentials (user,pass) to
184
federico<dot>sismondi<at>inria<dot>fr (recommended)
Federico Sismondi's avatar
Federico Sismondi committed
185 186 187

don't hesitate to contact me, this is a very simple procedure and it's
free :D
188

189
### Export AMQP environment variables
190

191 192 193
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
194

Federico Sismondi's avatar
Federico Sismondi committed
195 196 197 198
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
199

200
### Download the source code (see `Clonning the project` for more info)
Federico Sismondi's avatar
Federico Sismondi committed
201
```
202
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
203 204
cd ioppytest
```
205 206


207
### Build the testing tools
208

Federico Sismondi's avatar
Federico Sismondi committed
209 210
(docker, py2 and py3 needs to be installed in the machine)
```
211
make build-all
Federico Sismondi's avatar
Federico Sismondi committed
212
```
213

214
### Run testing tool (CoAP testing tool example)
Federico Sismondi's avatar
Federico Sismondi committed
215 216 217
```
make run-coap-testing-tool
```
218

219
### Connect to the interop session using the CLI
Federico Sismondi's avatar
Federico Sismondi committed
220 221 222
```
make run-cli
```
223

224
### Connect the agent to the backend
225

Federico Sismondi's avatar
Federico Sismondi committed
226
if user's IUT is a CoAP client:
227

Federico Sismondi's avatar
Federico Sismondi committed
228 229 230
```
make run-agent-coap-client
```
231

Federico Sismondi's avatar
Federico Sismondi committed
232
if user's IUT is a CoAP server:
233

234
```
Federico Sismondi's avatar
Federico Sismondi committed
235
make run-agent-coap-server
236 237
```

238
### Running a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
239

240
#### User to user session, second user with his/her own IUT
Federico Sismondi's avatar
Federico Sismondi committed
241 242 243 244 245 246 247 248

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

249
#### Single user session, against an automated-IUT
Federico Sismondi's avatar
Federico Sismondi committed
250 251 252

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
253

254
```
Federico Sismondi's avatar
Federico Sismondi committed
255
make run-coap-server
256 257
```

Federico Sismondi's avatar
Federico Sismondi committed
258 259
or for a coap client automated implementation:

260
```
Federico Sismondi's avatar
Federico Sismondi committed
261
make run-coap-client
262 263
```

Federico Sismondi's avatar
Federico Sismondi committed
264 265 266 267 268 269 270 271 272 273 274 275 276 277

## Running the interop session

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
278
```
279
git clone https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
Federico Sismondi's avatar
Federico Sismondi committed
280
cd ioppytest
281 282
```

Federico Sismondi's avatar
Federico Sismondi committed
283 284 285 286 287 288 289 290 291 292
## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
293

294
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
295

296
```
Federico Sismondi's avatar
Federico Sismondi committed
297
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
298 299 300 301 302
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
303 304
```

305 306

alternatively, if you are curious and you want to know
307 308
what's under the hood, you can see which processes are being run -in
the docker container- by the testing tool:
Federico Sismondi's avatar
Federico Sismondi committed
309

310
```
Federico Sismondi's avatar
Federico Sismondi committed
311 312 313 314 315 316 317 318
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

319 320
root@bab3b2220510:/ioppytest# supervisord -c supervisor.conf
root@bab3b2220510:/ioppytest# supervisorctl -c supervisor.conf
321 322 323 324 325 326 327 328
agent                            RUNNING   pid 28, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
329 330
or you can also run directly the processes without docker:
(supervisord needed)
331

Federico Sismondi's avatar
Federico Sismondi committed
332 333
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
334
sudo -E supervisorctl -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
335
```
336

Federico Sismondi's avatar
Federico Sismondi committed
337 338 339
you can use for example envs/coap_testing_tool/supervisor.conf.ini
for using the coap_testing_tool

Federico Sismondi's avatar
Federico Sismondi committed
340
note: use -E when launching supervisor process, it preserves the
341
env vars (like an exported AMQP_URL)
342 343


Federico Sismondi's avatar
Federico Sismondi committed
344 345
FAQ
---
346

Federico Sismondi's avatar
Federico Sismondi committed
347
- How can I install docker on my machine?
348

Federico Sismondi's avatar
Federico Sismondi committed
349
    For this just follow this instructions: https://docs.docker.com/engine/installation/
350

351

Federico Sismondi's avatar
Federico Sismondi committed
352 353 354 355
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
356

Federico Sismondi's avatar
Federico Sismondi committed
357 358
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
359

360
    please contact federico<dot>sismondi<at>inria<dot>fr
361 362


Federico Sismondi's avatar
Federico Sismondi committed
363
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
364

Federico Sismondi's avatar
Federico Sismondi committed
365
    try using ```--no-cache``` for the docker build
366

Federico Sismondi's avatar
Federico Sismondi committed
367
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a